It’s common to use C extensions in Python applications: in order to access pre-existing libraries, or for performance reasons. But unlike Python, the lack of memory safety in C and C++ can lead to crashes—and you’ll need to figure out what caused the crash. Show
This is extra fun when you get a silent crash half-way through a test run on your CI system:
In production, you’ll also often lack a core file, especially if you’re using Docker where the filesystem is often ephemeral. In this article I’ll cover some ways you can prepare for crashes in advance, so when they do occur you can quickly figure out which part of the codebase caused them:
The problemIf you have buggy Python code, you’ll get a traceback when you run it:
That’s helpful in figuring which code was responsible for the problem. But some Python programs crash due to bugs in C code, and then you don’t get a traceback. Let’s create a file called 1:
If we run it:
There’s no traceback. And if you can’t get access to the core file, you’ll have a very hard time figuring out what caused the problem. 1. Tracebacks on segfaults with faulthandlerThe Python standard library has a handy module called faulthandler that can print a traceback when a segfault occurs—that is, when a C extension crashes (the documentation has ). All you need to do set the environment variable 3 before running your code:
Notice how now we get a traceback, which means it’s much easier to figure out which code caused the problem. The only caveat is that if the problem involved sufficiently bad memory corruption you won’t be able to get any useful output. If you’re using 4 to run your tests you can alternatively just install the 5 package: it will enable faulthandler automatically when you use 4 to run tests.2. Enable detailed reporting of which test is runningMany test runners don’t print which tests are being run by default: you just get a list of dots:
The problem is that if you crash, and the only thing you have access to is that output, you won’t know exactly where the crash happened: you’ll know the test module, but what if your module has 100 tests, or these are integration tests that can call lots of different codepaths? So on CI at least make sure you run tests with more detailed reporting, so you know which tests exactly ran. E.g. add the 8 flag for 4:
If you crash you will then be able to see which test caused the problem—the last one printed, typically. 3. Dump the installed packages at the start of the CI runIf the crash is in a library, sometimes you’ll start getting crashes because of a minor change in the library version. If your local development machine has different package versions you won’t be able to reproduce the problem. So unless you’re explicitly pinning specific packages builds (with hashes for 0, or via 1 for Conda), make sure to print out the packages you’ve installed at the start of each CI run.That is, before you run your tests, run either 2 (or 1 if you use Conda) to make sure you know exactly which packages were used in the CI run.4. Use catchsegv on Linux
(Thanks to Glyph Lefkowitz for the suggestion.) 5. Using faulthandler in DockerC crashes are painful not just in tests, but in production too. In your Docker images, you can enable 8:
Failure is inevitableSooner or later something will go wrong—and with just a smidgen of bad luck it will happen in a way that makes it very hard to figure out what exactly crashed. So don’t wait for crashes to occur before adding this debug output—do it today, and your future self (or coworkers) will thank you. You might also enjoy:» Stuck with slow tests? Speed up your feedback loop» Why Pylint is both useful and unusable, and how you can use itThe concise and action-oriented guide to Docker packaging for productionDocker packaging for production is complicated, with as many as 70+ best practices to get right. And you want small images, fast builds, and your Python application running securely. Take the fast path to learning best practices, by using the Python on Docker Production Handbook. Learn practical Python software engineering skills you can use at your jobSign up for my newsletter, and join over 6700 Python developers and data scientists learning practical tools and techniques, from Python performance to Docker packaging, with a free new article in your inbox every week. |