foundation-model-stack / fms-lm-eval-service Goto Github PK
View Code? Open in Web Editor NEWLicense: Apache License 2.0
License: Apache License 2.0
Currently, the Flask app runs the lm-eval job by spawning subprocesses within the same Pod. Since the CRD and controller is ready, we need to update the implementation to:
submit_job
API, create an LMEvalJob
CRpoll_job
API, check the corresponding CR's statusjob_results
API, retrieve the results from the corresponding CRlist_jobs
, get a list of LMEvalJob
cancel_job
, update the LMEvalJOb
's status field to initiate the concealing flowUsers can pass their API keys or specify a secret ref for the lm-eval job.
Provide the capability to customize the datasets and use them in the tasks.
Use the Message
field in the status to populate the progress information
In the REST API, users can pass label/annotation information for the LMEvalJob. That information will be passed down to the pod when the controller creates the pod to run the evaluation job.
Users can choose whether to run the controller on cluster or namespace scope
Provide the capability to define custom tasks and submit a job to run custom tasks
Use the design doc as the base to discuss further CRD or design changes.
The test cases generated from the kubebuilder
are using Ginkgo and need a cluster to run them.
Need to add go unit tests and also some e2e tests
In the REST APIs, users can specify the resources that the pod would use when running the lm-eval job
Provide the options for users to specify where to store the logs or results of a lm-eval job
Currently, the driver needs specific RBAC to update the state of CRs. Need a mechanism to avoid the RBACs when running the pod for the lm-eval jobs
Set up GH actions to run go unit tests
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.