Debugging core files generated on a Customer’s box

by

What happens when a core file is generated from a Linux distro other than the one we are running in Dev? Is the stack trace even meaningful?

It the executable is dynamically linked, as yours is, the stack GDB produces will (most likely) not be meaningful.

The reason: GDB knows that your executable crashed by calling something in libc.so.6 at address 0x00454ff1, but it doesn’t know what code was at that address. So it looks into your copy of libc.so.6 and discovers that this is in select, so it prints that.

But the chances that 0x00454ff1 is also in select in your customers copy of libc.so.6 are quite small. Most likely the customer had some other procedure at that address, perhaps abort.

You can use disas select, and observe that 0x00454ff1 is either in the middle of instruction, or that the previous instruction is not a CALL. If either of these holds, your stack trace is meaningless.

You can however help yourself: you just need to get a copy of all libraries that are listed in (gdb) info shared from the customer system. Have the customer tar them up with e.g.

cd /
tar cvzf to-you.tar.gz lib/libc.so.6 lib/ld-linux.so.2 ...

Then, on your system:

mkdir /tmp/from-customer
tar xzf to-you.tar.gz -C /tmp/from-customer
gdb /path/to/binary
(gdb) set solib-absolute-prefix /tmp/from-customer
(gdb) core core  # Note: very important to set solib-... before loading core
(gdb) where      # Get meaningful stack trace!

We then advice the Customer to run a -g binary so it becomes easier to debug.

A much better approach is:

  • build with -g -O2 -o myexe.dbg
  • strip -g myexe.dbg -o myexe
  • distribute myexe to customers
  • when a customer gets a core, use myexe.dbg to debug it

You’ll have full symbolic info (file/line, local variables), without having to ship a special binary to the customer, and without revealing too many details about your sources.

More Related Contents:

  1. How to pretty-print STL containers in GDB?
  2. What’s the difference between 'gcc -g' and 'gcc -g3'?
  3. Can I set a breakpoint on ‘memory access’ in GDB?
  4. How do I print the elements of a C++ vector in GDB?
  5. Run an Application in GDB Until an Exception Occurs
  6. How do I print the full value of a long string in gdb?
  7. How to solve “ptrace operation not permitted” when trying to attach GDB to a process?
  8. Cannot insert breakpoints. Addresses with low values
  9. How to set breakpoints on future shared libraries with a command flag
  10. How to get a “codesigned” gdb on OSX?
  11. How to set a breakpoint in GDB where the function returns?
  12. How do I pass a command line argument while starting up GDB in Linux? [duplicate]
  13. How do I use the MinGW gdb debugger to debug a C++ program in Windows?
  14. How to pass arguments and redirect stdin from a file to program run in gdb?
  15. How to avoid entering library’s source files while debugging in Qt Creator with gdb?
  16. How to allocate memory dynamically for class using new in C++?
  17. How to display a dynamically allocated array in the Visual Studio debugger?
  18. Using kbhit() and getch() on Linux
  19. 64 bit ntohl() in C++?
  20. How to increase thread priority in pthreads?
  21. How to check what shared libraries are loaded at run time for a given process?
  22. Implementing a no-op std::ostream
  23. How to print every executed line in GDB automatically until a given breakpoint is reached?
  24. C++: Safe to use longjmp and setjmp?
  25. Child process receives parent’s SIGINT
  26. linux perf: how to interpret and find hotspots
  27. Linking problems due to symbols with abi::cxx11?
  28. How to name a thread in Linux? [duplicate]
  29. How do I use GDB in Eclipse for C/C++ Debugging?
  30. On OS X, simple C++ program gives incorrect results (which are a result of command-line options ‘c++03’ vs ‘c++11’)

Leave a Comment