opendatahub-io-contrib / genai-llm-rag-pattern Goto Github PK

Have reference code for deploying an instance and credential configuration.

Motivation:

1. To allow use of the openAI api format
2. To be able to use older GPUs e.g. v100s

DoD:

• Configured as code in github repo
• Model server tested on reference model
• Model server works with llama + V100 GPUs
• Documented how to use / reconfigure

GPU utilisation is important for admins:

• Should I mig slice my gpus
• Are people actually using GPUs or just sitting on them?

Dashboard should include:

1. Overall utilisation
2. Utilization per workbench and/or app wrapper job vs allocated.

Primary utilisation metric is GPU memory
Secondary is FLOPs / GPU processor.

Clusters which are lifecycle managed via OCM such as ROSA, ARO and OCD uses machine pools and autoscaling managed from the OCM console.

instascale does allow the use of a pool managed from OCM.

This requires a workflow to setup the pool. (This may be partially done by hand initially; potentially in the future via an ansible job or similar).

As an AI developer, I want a simple process to pull model X from huggingface and convert it to Caikit format.

Most likely by a data pipeline.

• Secrets management required for HF
• Parameterize with model on HF so it is reusable.

Definition of Done:

• UaT process shows model can be served from Caikit using fastapi
• Documentation in repo for website.

As a model developer, i want to be able to finetune separate specialised AI Agents for specific tasks, and orchestrate them.

Investigate https://flowiseai.com tooling or similar to support orchestration of open source LLMs

• https://github.com/RHEcosystemAppEng/genAI-for-document-generation
• https://github.com/validatedpatterns-sandbox/rag-llm-gitops

Today many of our demos use MCAD - but are blocking e.g. a user needs to keep their workbench running, and needs to progress the steps to shut down the ray cluster / MCAD job.

Provide a workflow where the MCAD batch job (e.g. fine tuning) is done asynchronously.

Document how to use the artifacts (e.g. jupyter notebooks) both inline and in docs website.

In a production context, it is important to have an evaluation framework included as part of the CI/CD pipeline to allow continuous evaluation.

Evaluate this or similar: https://github.com/explodinggradients/ragas

• AV to provide which EC2 to provide:

  • G5 instances

• Bryon deploy ROSA instance

  • central Europe EU Central and EU West

OpenLLMetry is a set of extensions built on top of OpenTelemetry that gives complete observability over your LLM application.

We should integrate our stack with OpenTelemetry Collector, instrumenting the LLM Providers / VectorDB.

Update as appropriate to ubi8 or ubi9 based node20 e.g.:

https://catalog.redhat.com/software/containers/ubi8/nodejs-20/647466671b1440a9c7cd4704

My understanding is traceloop is a SaaS platform. Today the default deployment is not configured and I believe should be a no-op. TO BE TESTED.

Either:

1. Configure for in cluster.
2. Deploy something else.

Presumes 'replacement' based batch ingestion.

@tnscorcoran to populate

Create reference recipe for deploying an instance and credential configuration.

DoD:

• Code is rehosted
• Version management / tagging strategy is in place to allow container builds to function correctly (semver pipeline? or just manual)

Historical repo was here:

https://github.com/tnscorcoran/rhoai-llm-demos-gitops

migrate content for infrastructure layer across

Today the PR review stage is not mandatory. Require the PR pipeline to pass before merging.

Config manages usecase defined by backend code (app/engines/loaders)

Open question as to whether S2I or Tekton can be used to build images

• Ensure 'bootstrapped' cluster will trigger at least manual build (to populate images)
• Use internal registry

Overall footprint:

• V100 minimum spec for non-TGIS

• Flash attention minimum e.g. required for MoE Models

• Implications of flash attention not being present.

Use drawio please

Potentially with a hardware partner who has access to a Lab who would be able to use it.

Currently minio is (mostly) deploying.

Automate the configuration of Minio including a default set of access credentials (configured from the validated patterns secrets file).

DoD

• have demonstrated the ability to deploy content into a minio tenant.

Today build is done inside of the container... do upstream and copy in required artifacts.

Preconfigure a data science project or projects* including:

1. Creation of a minio bucket
2. Configuration of data connection for minio
3. Setup a pytorch workbench
4. Setup workbench with secrets for huggingface credential management.
5. Setup a model server
6. Pre-load jupyter content into the workbench, if possible

Open questions:

1. Can you point to a specific sub-directory in a repo for jupyter content (e.g. allowing for mono-repo setups)

The repository root README.md file has had a number of violations of both the superlinter and mdformat.

This is a problem as all "README.md" files had to be exlcluded in order for running superlinter to be consistent.

• One for models
• One for raw data (chat history)
• One for transformed datasets.