Feature request: /run/shm instead of /tmp for .mat file passing about oct2py HOT 6 CLOSED

That is a good idea. I could use posix_ipc as an optional dependency. I'll try it out.

from oct2py.

Do you know of a way to open an mmap file in Octave?

from oct2py.

No, I found the 2012 message that you may have seen from Jordi saying that mmap was not in Octave yet as it is in Matlab.

I am not sure if mmap can help this situation:
I pass in a variable ~500MByte in size along with several smaller variables and parameters that are manipulated with an .m script. The computation takes about 100 milliseconds, but the /tmp file I/O takes far longer than that in the oct2py .mat file passing.

Following your very sensible recommendations in the documentation, I should modify my .m file to not mindlessly pass the 500MByte variable repeatedly, but the .m file is a decade-old well-respected package that for data fidelity reasons people wouldn't want to modify.

A trivial example of what causes the file I/O issue is:

import numpy as np
from oct2py import octave

A = np.ones((2000,20000),dtype=np.float64) #320Mbyte

for i in np.arange(10):
    y = octave.prod(A,i)

What I did to solve this was to port the actual function to Python. I have done this a few times already for various functions, I try them out on my data with oct2py, but due to file I/O hampering speed, I port interesting functions from Matlab to Python.

Implicit in this is that EVERY value of large matrix A is used every time, that's why I wasn't immediately sure mmap would help -- though you may be thinking of some novel trick that I haven't thought of.

from oct2py.

Okay, I added temp_dir as an optional argument to Oct2Py, which can be /run/shm if desired. That puts the burden on the user to set up the shared memory outside of oct2py. Sound good?

from oct2py.

Yes I noticed over 4 times speedup with the silly test code below. I should try it with the way I actually use the function, but 4 times speedup is pretty good (and I'm testing with an SSD, so HDD users should see substantially more improvement).

Now Windows users can use free RAM drive software while Mac/Linux can use built-in RAM drive.
Thank you so much! I should have made the pull request myself.

import numpy as np
import oct2py
from time import time

octmp = oct2py.Oct2Py()
ocram = oct2py.Oct2Py(temp_dir='/run/shm')

A = np.ones((2000,20000),dtype=np.float64) #320Mbyte

tic = time()
for i in np.arange(10):
    y = A*i
print(str(time()-tic) + ' seconds using Numpy')

tic = time()
for i in np.arange(10):
    y = octmp.prod(A,i)
print(str(time()-tic) + ' seconds using /tmp')

tic = time()
for i in np.arange(10):
    y = ocram.prod(A,i)
print(str(time()-tic) + ' seconds using /run/shm RAM')

0.702534914017 seconds using Numpy
62.1404719353 seconds using /tmp
14.0810651779 seconds using /run/shm RAM

from oct2py.

Excellent, thank you for the great suggestions.

from oct2py.