Slow performance about salometoopenfoam HOT 17 OPEN

A profile of a (roughly) 2.5 minute export - As can be seen, major time is currently spent on determining group baffles (isGroupBaffle, over a minute), and obtaining face nodes (GetElemFaceNodes) via the CORBA platform (_omnipy.invoke).

from salometoopenfoam.

From the above we can see that the hashing and sorting is somewhat inefficient, but is second to the determining baffles and obtaining face nodes. Call counts to _omnipy.invoke are almost exclusive to obtaining face nodes (2049220 out of 2208669), some way of obtaining the data in fewer calls would be ideal.

from salometoopenfoam.

from salometoopenfoam.

No need to drop the functionality! The problem was with the use of data structure - Lists:
If I had a list of faces [1, 2, 3, 4, 5] and I wanted to check the id 5 was in there, I would have to iterate linearly - 1, 2, 3, 4, 5 - and then I've found this id - so it's a group baffle. That was happening with the isGroupBaffle function. By converting it to the data type set - we have the same speed as a dictionary looking for a unique item - it's hashed in like a key, but without a returning value - so we know immediately whether it's in the set or not. For more info : https://wiki.python.org/moin/TimeComplexity

I've sent a pull request with this implementation - I've compared the changes in times and it seems to drop the baffle check from being a time issue - I've checked the exports myself but if you have time a second check would be appreciated :)

You made a good suggestion about comparing the length - but I think this change means we shouldn't have to do that.

from salometoopenfoam.

from salometoopenfoam.

It's merged. Again really nice work!

Welcome as a contributor! I've should have added you in your last pull request. Sorry about that.

I'm still having trouble with installing salome on the new os. But then again I manage about 5 minutes i front of the machine per day so it will take time =).

/Nicolas

from salometoopenfoam.

From some further profiling it seems that the remaining bottleneck of the exporter lies within the MeshBuffer class constructor - specifically making calls to obtain the faces.

I've tried a couple of methods to try and reduce the CORBA overhead:

• Parallel loops (running 2 or more threads, running GetElemFaceNodes across these threads) but this ended up slowing the script down with each additional core. So it seems that this CORBA implementation is not parallel, or there is some sort of settings that must be invoked to enable the calls to be parallel - it may be that I post a query on the forums.
• Attempting to call the method omnipy.invoke directly - but I couldn't get this to work, and it would probably introduce a version dependency that defeats the point of using Python :)

It may be that we make some sort of c++ binding option that can be compiled to obtain the SMESH pointer to get the faces in a single call - then provide the user with an option to run the native built binding if better performance is necessary.

Any ideas that could assist?

Happy holidays!
Sam.

from salometoopenfoam.

from salometoopenfoam.

Just so you know. I haven't forgot about this! I keep taking a look from time to time to see if anything sticks out. I've also posted on the salome forum so with any luck i'll get a response.

The speed has become more convenient for my use, it's taken hour long macro meshes exports down to about 10 minutes. Especially ones with many boundary conditions. But even so that's quite a lot so I'm interested to bring the speed down more!

from salometoopenfoam.

from salometoopenfoam.

It's not something I've had a problem with because I'm using Fedora and Windows to test my code. Perhaps you need to update (apt-get) the global protobuf package? It seems to be a compiler related package.

from salometoopenfoam.

from salometoopenfoam.

Ah! That's a shame! I think it is quite hard to compile it all by source, that's something that really needs to be improved upon because it's not well explained. It makes writing compiled extensions quite tough going too.

I think the way I'm going to try tackle this is by getting back to parallel calls using multithreaded corba. The plan is to give the option of the user choosing the amount of threads they want to use, and then splitting the face integer collection task across these. From the linked diagram it seems that we need to start multiple client sockets to the corba 'naming service', once we have the IOR of the mesh I think we might be able to do some light parallelisation to save time. I'm not sure if the plan is feasible yet but it's worth looking into.

--update-- using omniorb we can get our path to the configuration file

import os
os.environ['OMNIORB_CONFIG']

we can see that there are not many properties set if the user is using an automatically generated configuration file. In the omniORB configuration documentation we can see the following information about threads per CORBA connection:

threadPerConnectionPolicy default = 1

If true (the default), the ORB dedicates one server thread to each incoming connection. Setting it false means the server should use a thread pool.

So, it's currently set to one server thread per connection by default. I think I might try to configure this to be a greater number, and see if that will bring the time down. However, this means having to modify a users config file, so it may be a better option to make multiple connections to get the object reference.

from salometoopenfoam.

from salometoopenfoam.

from salometoopenfoam.

Glad I can be of assistance!

I have uploaded a parallel script on the following branch source. Based on the thread count specified the array of volumes is partitioned by the thread count and handled across the threads, then rejoined once all work is complete.

If you try it out, you'll notice as you increase the number of threads - the process seems to take longer than just a single thread, call latency is introduced, so clearly the parallel calls are being blocked... I can't seem to find any CORBA configuration settings that will allow the server to multithread (I probably need more searching to crack it) - it may be that this script is adapted to make several client connections to the naming server across the threads to remain compliant with the 'one thread per connection' policy. This would mean setting up a client connection across each thread and getting the object reference using the IOR.

Unfortunately, it seems to be a lot of complexity to make the thing faster...

from salometoopenfoam.

from salometoopenfoam.