Why does my CUDA kernel crash (unspecified launch failure) with a different dataset size?

by

Solved the problem. Turns out the WDDM TDR (timeout detecion recovery) was enabled and the delay was set to 2 seconds. This means that if the kernel execution time exceeds 2s, the driver will crash and recover. This is applicable to graphics and rendering (for general purpose uses of the GPU). In this case however, the TDR must either me disabled or the delay increased. By increasing the delay to 10s, the crash error “unspecified launch failure” ceased to appear and kernel execution continued as before.

The TDR delay (as well as enabling/disabling) can be done through Nsight options in the Nsight Monitor or through the Registry (HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\GraphicsDrivers) – DWORDS Tdrdelay and Tdrlevel.

More Related Contents:

  1. CUDA compute capability requirements
  2. Cuda atomics change flag
  3. How is CUDA memory managed?
  4. Modifying registry to increase GPU timeout, windows 7
  5. How can I implement a custom atomic function involving several variables?
  6. Passing Host Function as a function pointer in __global__ OR __device__ function in CUDA
  7. How to verify CuDNN installation?
  8. What is the purpose of using multiple “arch” flags in Nvidia’s NVCC compiler?
  9. How are 2D / 3D CUDA blocks divided into warps?
  10. What can I do against ‘CUDA driver version is insufficient for CUDA runtime version’?
  11. What is a bank conflict? (Doing Cuda/OpenCL programming)
  12. nvidia-smi Volatile GPU-Utilization explanation?
  13. How do I use Nvidia Multi-process Service (MPS) to run multiple non-MPI CUDA applications?
  14. Any particular function to initialize GPU other than the first cudaMalloc call?
  15. CUDA and nvcc: using the preprocessor to choose between float or double
  16. Copy an object to device?
  17. Polymorphism and derived classes in CUDA / CUDA Thrust
  18. CUDA: How to use -arch and -code and SM vs COMPUTE
  19. How do I select which GPU to run a job on?
  20. multi-GPU basic usage
  21. How can I check the progress of matrix multiplication?
  22. device function pointers
  23. How can I make tensorflow run on a GPU with capability 2.x?
  24. cudaMemset() – does it set bytes or integers?
  25. How does CUDA assign device IDs to GPUs?
  26. Inconsistency of IDs between ‘nvidia-smi -L’ and cuDeviceGetName()
  27. CUDA allocation alignment is 256 bytes – seriously?
  28. CUDA function pointers
  29. Which Compute Capability is supported by which CUDA versions?
  30. Using an array of device function pointers

Leave a Comment