Hadoop state non-functional about crosscat HOT 9 OPEN

@gregory-marton question for you :)

from crosscat.

Sounds like you've done some investigation already. Can you share the details? In what way did it not work, which settings did you find, etc.?

from crosscat.

Oh, I thought it was intentional.

There appears to not be a crosscat.settings since v0.1.25
The change that has broken seems sensible and straightforward but I would ask if the implementation, if fixed, would still be functional in the current version.

from crosscat.

I'm actually not sure of the history. I came onboard around v0.1.40, and this was not on my radar.
@riastradh-probcomp if you have time, can you give more context?

@kadwanev, pull requests are always welcome!

from crosscat.

Nobody has touched the Hadoop code in years, and it apparently requires various moving parts that were customized for one developer's setup years ago, with some private network layout and Amazon S3 account and local Hadoop installation &c.

I expect it would be easier to start from scratch than to try to revive what's there.

from crosscat.

Given that context, I expect the appropriate "fix" would be to remove HadoopEngine. @kadwanev, if you want to take this on instead, we would absolutely welcome it. If interested, let me know a time frame to check back with you?

from crosscat.

Understood. Thanks for the responses.

Just want to ask:
Is the distribution technique still sound?
Did it ever work?

I ask this because the only reducer reference I see is /bin/cat, which leads me to question if it collected results back into a single response.

I want to know if the current implementation is a good starting point or not.

from crosscat.

There is no 'reduce' step because Crosscat's job is just to apply a transition operator to each of a number of independent states -- it's all 'map', and it is embarrassingly parallelizable, so any parallelism you throw at it should stick, no matter how trivial.

The MultiprocessingEngine is just LocalEngine with Python map replaced by multiprocessing.pool().map to transition the states in separate processes. Doing the same on different computers will certainly work just fine.

from crosscat.

Thanks. That definitely answers my question. I'll be looking to contribute some code as soon as possible.

from crosscat.