Grid Tree data adapters are very limited about pygrid-deprecated---see-pysyft- HOT 10 CLOSED

Seems like something Adapters should handle? If not in the near-term, somewhere down the line.

from pygrid-deprecated---see-pysyft-.

@jvmancuso agreed - the next step for this project is probably just to require that the input data be in a matrix format (which the adapters spec is responsible for transforming it into). This would then let the Grid Tree specify arbitrary required numbers of input dimensions which should be quite flexible for many use cases.

from pygrid-deprecated---see-pysyft-.

(except for recurrent ones and RL ones)

from pygrid-deprecated---see-pysyft-.

I am a little wary adapters but maybe I have a wrong idea.

One problem is that we can't conceive of every way to map data so it's impossible for us to define an adapter for every case. This leaves it up for either the task owner or the architecture owner to define the adapter, but that starts to open up more attack vectors to use grid in a malicious way.

In the case of images we can easily define what shape the image needs to be in to go into the model and cover all cases for that, but that will likely not work for other types of data -- maybe that's not true?

Requiring the nodes to format their own data makes running a grid node way more work so I don't like that idea either.

from pygrid-deprecated---see-pysyft-.

This leaves it up for either the task owner or the architecture owner to define the adapter

semi related -- right now it is unclear who's responsibility this should be. Should the task owner define sample data and it is the nodes/arch owner's responsibility to match this data. This seems like it will be required if the task owner is going to ultimately own validation. However, it makes it harder for non-experts to define a problem.

from pygrid-deprecated---see-pysyft-.

I was thinking it'd be up to the task owner to provide the adapters and they would need to be posted on github or something and I'm not sure we need to worry about security risks in those cases because the code would be auditable. I'm not sure we need to worry about non-experts proposing tasks because can a non-expert really propose a task in a meaningful way? For example, I'm sure everyone on Kaggle providing competitions have experts on their side. I feel like the grid workers should have to do nothing though besides point the adapters to their data.

from pygrid-deprecated---see-pysyft-.

I think the security issues on this one can be solved by documentation and making sure people running grid workers understand the security risks of running just any old adapter.

from pygrid-deprecated---see-pysyft-.

I don't think I agree with what I've just said above. I think I made it sound way easier than this problem actually is.

from pygrid-deprecated---see-pysyft-.

I'm not sure we need to worry about non-experts proposing tasks because can a non-expert really propose a task in a meaningful way?

It does make sense to focus on data scientists who can define tasks in meaningful ways at first so i'm fine with that .. but devs who want a model but don't know data science well probably could define tasks ... "I want to be able to figure out what type of shoe is in a picture" and then they get some code they can plug into their iOS would be pretty powerful.

from pygrid-deprecated---see-pysyft-.

Yeah maybe once the network has grown there will be adapters non-experts can pick up and use. Like generic ones for just images.

from pygrid-deprecated---see-pysyft-.