I just discovered strace

strace is a nifty linux program that will give you extra insight into what a program is doing. This is particularly helpful when you're troubleshooting 3rd party software and you don't have any source code to look at. You run it like this:

strace -o /tmp/strace.log command [arg ...]

This will run the command as if you'd typed

command [arg ...]

It launches your program in such a way that it is able to monitor all the system calls your program makes to the kernel. Each of these system calls gets logged to the file you specify. For each system call you can see the arguments passed in and the return value. This may not immediately sound useful but it can be surprisingly helpful.

Here's how I used it. We had some third party software that was exiting with a cryptic error message. By running the program under strace, I was able to determine that the program exited while reading a certain input file from the following line:

open("/usr/local/inputfile.gz", O_RDONLY) = 4

which was followed by multiple lines like this:

read(4, "\313\232b\26.\243\363\374\277|\256\330\303`\255\374b\320\310\207\340\363l\336\305+A#c\367\306K"..., 8192) = 8192
read(4, "\304\377F\344L\336\17\21\351J\374\17\"\357\f?B$s\377\27\221\314\275\4\351\32_D|\231\354"..., 8192) = 8192
....

The program seems to be reading this file 8 K at a time. By counting the number of lines like this, and multiplying by 8K, I was able to determine an approximate offset into the file of where it was having trouble. Once I had that offset, I had to see what was in the file at that location.

head -c 1818624 /usr/local/inputfile.gz > /tmp/first.gz

Then I unzipped the file with

gzip -dc /tmp/first.gz > /tmp/first

Looking near the end of the unzipped file I was able to find the input that was causing the problem. Conclusion: strace is easy to use, surprisingly helpful, and is a great tool to have in your troubleshooting arsenal.