Sunday, February 7, 2010

Semi-rant against python multiprocessing module

Go here. And now look at issue 13 in that bug track. Or just have a look at this.

This is bad news. The unpicklability of lambdas has forced me to make some ugly changes to the parallel code I was using to drive my python extensions. I really-really wish it was doable.

For a while, I have been thinking that use of python's multiprocessing module to parallelize your C extensions akin to solving the problem at the wrong abstraction level. This nails it.

OTOH, opencl has a wonderful parallelization and synchronization model. In future, I am looking to write OCL code only for compute intensive tasks. Implementations from AMD and nvidia are out of beta, but they are immature atm. Intel is missing in action.

I expect it will take them about a year more to mature.

No comments: