qilingu / mlrun Goto Github PK

MLRun enables production pipeline design using a modular strategy, where the different parts contribute to a continuous, automated, and far simpler path from research and development to scalable production pipelines, without refactoring code, adding glue logic, or spending significant efforts on data and ML engineering.

MLRun uses Serverless Function technology: write the code once, using your preferred development environment and simple “local” semantics, and then run it as-is on different platforms and at scale. MLRun automates the build process, execution, data movement, scaling, versioning, parameterization, outputs tracking, CI/CD integration, deployment to production, monitoring, and more.

Those easily developed data or ML “functions” can then be published or loaded from a marketplace and used later to form offline or real-time production pipelines with minimal engineering efforts.

Data preparation, model development, model and application delivery, and end to end monitoring are tightly connected: they cannot be managed in silos. This is where MLRun MLOps orchestration comes in. ML, data, and DevOps/MLOps teams collaborate using the same set of tools, practices, APIs, metadata, and version control.

MLRun simplifies & accelerates the time to production.

MLRun is composed of the following layers:

• Feature Store — collects, prepares, catalogs, and serves data features for development (offline) and real-time (online) usage for real-time and batch data. See also Feature store: data ingestion and Feature store: data retrieval, as well as the Feature Store tutorials.
• ML CI/CD pipeline — automatically trains, tests, optimizes, and deploys or updates models using a snapshot of the production data (generated by the feature store) and code from the source control (Git).
• Real-Time Serving Pipeline — Rapid deployment of scalable data and ML pipelines using real-time serverless technology, including the API handling, data preparation/enrichment, model serving, ensembles, driving and measuring actions, etc.
• Real-Time monitoring and retraining — monitors data, models, and production components and provides a feedback loop for exploring production data, identifying drift, alerting on anomalies or data quality issues, triggering re-training jobs, measuring business impact, etc.

It's easy to start using MLRun:

1. Install the MLRun service locally using Docker or over Kubernetes Cluster. Alternatively, you can use Iguazio's managed MLRun service
2. Set up your client environment to work with the service.
3. Follow the Quick Start tutorial and Additional Tutorials and Examples to learn how to use MLRun to develop and deploy machine learning applications to production.
   

For hands-on learning, try the MLRun Katakoda Scenarios. And you can watch the Tutorial on Youtube to see the flow in action.

Read more in the MLRun documentation, including:

• MLRun basics
  • What is MLRun?
  • Quick-Start Guide
  • Tutorials and examples
  • Installation and setup guide
• Concepts
  • Projects
  • MLRun serverless functions
  • Data stores and data items
  • Feature store
  • Runs, functions, and workflows
  • Artifacts and models
  • Deployment and monitoring
• Working with data
  • Data ingestion
  • Data retrieval
  • Tutorials
• Develop Functions and models
  • Creating and using functions
  • Run, track, and compare jobs
• Deploy ML applications
  • Real-time serving pipelines (graphs)
  • Model serving pipelines
  • CI/CD, rolling upgrades, git
• References
  • API
  • CLI
  • Glossary