Getting cProfile to handle scripts that import __main__.

Previously, I’ve written about the challenges of handling __main__ while tracing/monitoring execution of scripts, and that cProfile (the built-in python profiling tool) doesn’t handle this well. Long story short, I was able to contribute a fix that makes this possible! The cpython maintainers were super responsive and very helpful, which made it feel super approachable. This is probably the largest project I’ve ever contributed to, and it’s surreal to think that a small bit of code I wrote will eventually be distributed to so many machines around the world.

The fix for cProfile is quite similar to what I do in chkpt, but the difference is that instead of executing the code directly through the module loader, I instead pass a reference to a newly created module named __main__’s __dict__ as the globals to exec.

Something that was pretty funny to me was that pdb implements this correctly. Had I tried pdb instead of cProfile, I would have never found this bug, and also probably figured out how to implement something similar for chkpt, but I would have never gained a deeper understanding. I think the current implementation for chkpt (and now cProfile), where a new module is created and then set to be __main__ for the user supplied code is much cleaner and less hacky than the pdb implementation. I wonder if there’s a better, more reusable way to express this pattern in a way that can easily be used by other monitoring/profiling tools.

Working on this has given me a lot more understanding and comfort in understanding how modules are implemented in python. It’s made me rethink my previous hacks I’ve done to get multiple versions of the same module to be importable within the same project. I think it’s possible to do this in a cleaner way by constructing the module objects directly, and giving each module a different view of sys.path. Anyway, this contribution a fun little spin-off from my ongoing effort in building chkpt. Glad to have contributed back to the community!