Sunday, November 9, 2008

Multithreading and Python

Came across this fantastic review of various Python VMs out there. Of these, Stackless Python caught my eye particularly.

Native concurrency support. No GIL issues. Point to point message passing for free.

Not bad. Not bad at all.

I have written before on my desire of writing multithreaded code. Due to other demands on my time, that project is still stuck in limbo. But I realized that it will take a lot of work (aka lots of infrastructure code) to get it done. So, naturally, I turned to Python, because

The best code you write is the code that you don't write at all.

Turns out that while python has full support for threads, the CPython interpreter will interpret Python bytecode only one thread at a time, enforcing de-facto serialization of your parallel code.

Not good.

But then you can call multiple extensions from the main thread and have them release the GIL. And after this, extending the interpreter doesn't appear so difficult. A quick google search later I landed here, which shows just how easy it is for all your extensions to release the GIL before launching and re-acquiring it before returning to Python land.

Will take a shot at this way if time permits. I have a few ideas on what to do. But I am not abandoning the classical interpreter for Stackless python just yet. Need to figure out how nicely Stackless Python plays with C extensions before I do that. And of course, there needs to be a fedora package. There is simply no question of compiling the whole thing myself.

No comments: