CUDA apps time out & fail after several seconds – how to work around this?

by

I’m not a CUDA expert, — I’ve been developing with the AMD Stream SDK, which AFAIK is roughly comparable.

You can disable the Windows watchdog timer, but that is highly not recommended, for reasons that should be obvious.
To disable it, you need to regedit HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Watchdog\Display\DisableBugCheck, create a REG_DWORD and set it to 1.
You may also need to do something in the NVidia control panel. Look for some reference to “VPU Recovery” in the CUDA docs.

Ideally, you should be able to break your kernel operations up into multiple passes over your data to break it up into operations that run in the time limit.

Alternatively, you can divide the problem domain up so that it’s computing fewer output pixels per command. I.e., instead of computing 1,000,000 output pixels in one fell swoop, issue 10 commands to the gpu to compute 100,000 each.

The basic unit that has to fit within the time slice is not your entire application, but the execution of a single command buffer. In the AMD Stream SDK, a long sequence of operations can be broken up into multiple time slices by explicitly flushing the command queue with a CtxFlush() call. Perhaps CUDA has something similar?

You should not have to read all of your data back and forth across the PCIX bus on every time slice; you can leave your textures, etc. in gpu local memory; you just have some command buffers complete occasionally, to prove to the OS that you’re not stuck in an infinite loop.

Finally, GPUs are fast, so if your application is not able to do useful work in that 5 or 10 seconds, I’d take that as a sign that something is wrong.

[EDIT Mar 2010 to update:] (outdated again, see the updates below for the most recent information) The registry key above is out-of-date. I think that was the key for Windows XP 64-bit. There are new registry keys for Vista and Windows 7. You can find them here: http://www.microsoft.com/whdc/device/display/wddm_timeout.mspx
or here: http://msdn.microsoft.com/en-us/library/ee817001.aspx

[EDIT Apr 2015 to update:] This is getting really out of date. The easiest way to disable TDR for Cuda programming, assuming you have the NVIDIA Nsight tools installed, is to open the Nsight Monitor, click on “Nsight Monitor options”, and under “General” set “WDDM TDR enabled” to false. This will change the registry setting for you. Close and reboot. Any change to the TDR registry setting won’t take effect until you reboot.

[EDIT August 2018 to update:]
Although the NVIDIA tools allow disabling the TDR now, the same question is relevant for AMD/OpenCL developers. For those: The current link that documents the TDR settings is at https://learn.microsoft.com/en-us/windows-hardware/drivers/display/tdr-registry-keys

More Related Contents:

  1. Modifying registry to increase GPU timeout, windows 7
  2. How to measure the inner kernel time in NVIDIA CUDA?
  3. Passing Host Function as a function pointer in __global__ OR __device__ function in CUDA
  4. nvidia-smi Volatile GPU-Utilization explanation?
  5. How do I use Nvidia Multi-process Service (MPS) to run multiple non-MPI CUDA applications?
  6. How does CUDA assign device IDs to GPUs?
  7. GPU Emulator for CUDA programming without the hardware [closed]
  8. How is CUDA memory managed?
  9. sending 3d array to CUDA kernel
  10. How do CUDA blocks/warps/threads map onto CUDA cores?
  11. How are 2D / 3D CUDA blocks divided into warps?
  12. Any particular function to initialize GPU other than the first cudaMalloc call?
  13. CUDA: How many concurrent threads in total?
  14. multi-GPU basic usage
  15. CUDA compute capability requirements
  16. Cuda atomics change flag
  17. How can I implement a custom atomic function involving several variables?
  18. How to verify CuDNN installation?
  19. What is the purpose of using multiple “arch” flags in Nvidia’s NVCC compiler?
  20. Using CUDA with Visual Studio 2017
  21. Copy an object to device?
  22. Should I unify two similar kernels with an ‘if’ statement, risking performance loss?
  23. CUDA: How to use -arch and -code and SM vs COMPUTE
  24. How do I select which GPU to run a job on?
  25. device function pointers
  26. How can I make tensorflow run on a GPU with capability 2.x?
  27. cudaMemset() – does it set bytes or integers?
  28. CUDA determining threads per block, blocks per grid
  29. CUDA function pointers
  30. Using an array of device function pointers

Leave a Comment