Sunday, November 1, 2009

Mmap-ing temporary files

There is a cool way to allocate memory without actually doing malloc. It uses temporary files.

Here's how to do it,

1) Make a temporary file, and then use fileno() to get it's file descriptor.

2) Seek to the the end of the region which you want mmap-ed and the write a dummy data (just a "") to make the file of that size

3) Actually mmap this file with this size itself.

Why bother with all this, you ask. It can be useful to inter-operate between C land and numpy while sharing memory while avoiding memory leaks. This can be achieved by using numpy.memmap class and then passing in the numpy array to obtain the pointer.

This will in general need some amount of reworking in your library's memory allocation routine, but avoids the troubles associated with making sure that the C object is deallocated after the associated numpy object is freed.