Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update hardware-failure.md #1221

Merged
merged 2 commits into from
Mar 21, 2024
Merged

Update hardware-failure.md #1221

merged 2 commits into from
Mar 21, 2024

Conversation

jklgrasso
Copy link
Contributor

Changed the gpu-burn example to use tc instead of d to create more of a workload when testing.

Changed the gpu-burn example to use tc instead of d to create more of a workload when testing.
@@ -156,7 +156,7 @@ make
And now we can run it like so (this example will run it for 60 minutes/1 hour):

```bash
./gpu_burn -d 3600
./gpu_burn -tc 3600
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How does this behave on old GPUs without tensor cores?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will crash on cards older than RTX 20X0

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's keep both commands then, and explain that -d must be used for older GPUs.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I removed the double switch for GTX GPUs and added another prompt for RTX GPUs with the -tc switch.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Screenshot from 2024-03-21 17-19-32
Screenshot from 2024-03-21 17-21-19
Screenshot from 2024-03-21 17-23-01
Screenshot from 2024-03-21 17-25-00

Just to add my 2¢
It is probably best to go with a multifaceted approach and do more variations for less time.
-d uses less power than using no argument other than time. While this may stress parts of the GPU more, it doesn't stress the power delivery components as much which could be where the failure is.

./gpu_burn 1800
./gpu_burn -d 1800
./gpu_burn -tc 1800
./gpu_burn -d -tc 1800

Adjusted for RTX GPUs to use -tc and removed the -d switch for GTX GPUS.
Copy link
Contributor

@thomas-zimmerman thomas-zimmerman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks good

@thomas-zimmerman thomas-zimmerman merged commit 5380712 into master Mar 21, 2024
3 checks passed
@thomas-zimmerman thomas-zimmerman deleted the gpu-burn-d-to-tc branch March 21, 2024 20:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants