foundation-model-stack / fms-lm-eval-service Goto Github PK

Currently, the Flask app runs the lm-eval job by spawning subprocesses within the same Pod. Since the CRD and controller is ready, we need to update the implementation to:

• submit_job API, create an LMEvalJob CR
• poll_job API, check the corresponding CR's status
• job_results API, retrieve the results from the corresponding CR
• list_jobs, get a list of LMEvalJob
• cancel_job, update the LMEvalJOb's status field to initiate the concealing flow

Users can pass their API keys or specify a secret ref for the lm-eval job.

Provide the capability to customize the datasets and use them in the tasks.

Use the Message field in the status to populate the progress information

In the REST API, users can pass label/annotation information for the LMEvalJob. That information will be passed down to the pod when the controller creates the pod to run the evaluation job.

Users can choose whether to run the controller on cluster or namespace scope

Provide the capability to define custom tasks and submit a job to run custom tasks

Use the design doc as the base to discuss further CRD or design changes.

The test cases generated from the kubebuilder are using Ginkgo and need a cluster to run them.
Need to add go unit tests and also some e2e tests

In the REST APIs, users can specify the resources that the pod would use when running the lm-eval job

Provide the options for users to specify where to store the logs or results of a lm-eval job

Currently, the driver needs specific RBAC to update the state of CRs. Need a mechanism to avoid the RBACs when running the pod for the lm-eval jobs

Set up GH actions to run go unit tests