python model server about kubeflow HOT 12 CLOSED

Thanks so much for letting us know, @cliveseldon! I think the overlap here is really strong, and would love to collaborate/include Seldon-core.

Do you have any thoughts on whether or not it would be monolithic, or break it down to component parts? It does a lot of different things, and was just curious about your philosophy.

from kubeflow.

I spent some time looking at it this morning. The Seldon prediction graph and CRD looks super cool and should be pretty easy to incorporate.

I think we would do something very similar to what we do for the TfJob operator.

We can create a ksonnet package to deploy the SeldonDeployment CRD. At this point I believe users should just be able to create SeldonDeployment's and they would work. We could also deploy the Grafana and Prometheus stuff but it might make sense to make that a separate ksonnet package or include it in the core ksonnet package.

/cc @cliveseldon @aronchick

from kubeflow.

For building user images, we (at Project Jupyter) have built http://github.com/jupyter/repo2docker. https://github.com/jupyterhub/binderhub (deployed at mybinder.org) is a JupyterHub service that lets you dynamically build & run images based on a git repo (heroku style!). Perhaps that'll be cool to have?

from kubeflow.

@BrentDorsey We definitely need containers suitable for serving a variety of models. I think a lot of companies have already developed this piece. For example seldon has defined a prediction API with implementations for various model types.

So hopefully we can figure out how to leverage one or more of the existing solutions rather than developing a new one.

@yuvipanda I'd love to understand binderhub more. One of the problems we have right now is if you're developing on cluster (e.g. via Juptyerlab thank you for that) and writing a program (e.g. TF) we need to build a container from within the cluster. Right now I don't have a good solution for building docker images from within the cluster. It sounds like binderhub might solve this problem.

/cc @cliveseldon

from kubeflow.

@jlewi - I like Pipeline.AI's prediction API, of course I'm biased as I'm a Pipeline.AI committer.

@yuvipanda - I'd also love to understand more about binderhub.

from kubeflow.

@BrentDorsey Do you have a pointer to AI's prediction server? I had trouble finding it in the repo.

from kubeflow.

from kubeflow.

Hi,

I work for Seldon, referenced above by @jlewi

We have pushed our seldon-core product to github https://github.com/SeldonIO/seldon-core

It's goals are to allow generic machine learning models to be deployed on Kubernetes. It would be great to get some feedback and how this contributes to this issue and more generally for the deployment side of KubeFlow.

from kubeflow.

Thanks @aronchick We're focused just on serving, not training. The core of what we provide is allow runtime graphs to be specified via a CRD. Those graphs may contain containers that are microservices which expose ML predictions but also routers (e.g., AB routing), combiners (e.g., ensembling), transformers (e.g., feature transforms) which may also be microservices inside containers.

At present this functionality is handled by 3 components that run in kubernetes, an operator to handle the CRD, an engine to handle the request/response service mesh described by your graph and an API front end that exposes these to the outside world as gRPC and REST. All this is quite tightly coupled to the CRD. So I suppose these parts would be similar to the TensorFlow serving part of the present KubeFlow except aiming to allow more complex graphs and generic ML runtime Tools (thus the discussion in this issue).

There is also Prometheus, Grafana (for metrics and a dashboard) and Kafka (and thus zookeeper) in the current helm chart but its not core and could be replaced/ignored.

from kubeflow.

Oh for sure but, unless I misunderstand, Seldon-core also handles packaging of the model for serving? If it's limited to serving, then I totally agree with you. When I was referring to components, it was more the different stages of a pipeline.

from kubeflow.

Yes, we provide a generic API for prediction models to handle request/responses. At a high level we want to make it easy to add runtime models so either the data scientist can do it themselves (dockerize their model and expose a REST/gRPC endpoint that respects the API) and/or we plan to provide some automated wrappers to make this easy.

So to be clear, packaging of the model is separate so could be done outside the project.

from kubeflow.

Seldon is now available as part of Kubeflow. So I'm closing this issue.

from kubeflow.