What would you like to be added:

Speculative Decoding helps to accelerate the prediction of large language models. which is supported by vllm by default.

Why is this needed:

Improve the inference throughput.

Completion requirements:

This enhancement requires the following artifacts:

• Design doc
• API change
• Docs update

The artifacts should be linked in subsequent comments.

What would you like to be added:

It would be super great to support benchmarking the LLM throughputs or latencies with different backends.

Why is this needed:

Provide proofs for users.

Completion requirements:

This enhancement requires the following artifacts:

• Design doc
• API change
• Docs update

The artifacts should be linked in subsequent comments.

Add the support for inference services.

Why is this needed:

Backends and datasources support heavily depends on the e2e tests to verify that it works as expected, however, we're lack of GPU machines.

What would you like to be cleaned:

For example, people want to deploy the model with different scheduling primitives, colocated or exclusive?

Why is this needed:

Expressing deploy primitives.

What would you like to be added:

Different backends support a large range of popular models, but not all models are supported, so we should have a failover policy for them, at least this policy should be supported in Playground.

Why is this needed:

Completion requirements:

This enhancement requires the following artifacts:

• Design doc
• API change
• Docs update

The artifacts should be linked in subsequent comments.

We use lws as underlying workload to support multi-host inference, however, we only support one pod one model right now. The general idea is once model flavor requires like nvidia.com/gpu: 32, we'll split into 4 hosts each requires 8GPU.

At the first glance, because the Models are published by the admins, it maybe ok because the data source is under supervising.

Or is that user need?

Support splitwise which split the prefill and decode apart, belongs to different Pods.

As the serve.Spec describes, we have minReplicas and maxReplicas, what we hope to do is adjust the number based on the traffic, aka. servreless. We can use ray or keda/knative as alternatives, but here we hope we can have a simple implementation, then no need to depend on other libraries.

Hope we can do that.

This requires the support of lws community.

What would you like to be added:

Right now, we only have one model version for common deployment, however, if we take a higher level view of the model lifecycle, version management is necessary.

Why is this needed:

Support the full lifecycle of models.

Completion requirements:

This enhancement requires the following artifacts:

• Design doc
• API change
• Docs update

The artifacts should be linked in subsequent comments.

xref: #20

This can help to optimize the startup time of the Pod, however, usually this is limited by the bandwidth, which means it will slow down the weight downloading. If the image has been downloaded, there's no different since it's cached.

But if your registry is deployed in the intranet, it will still benefit your startup time.

We should have the baseline for the project, generally three kinds of tests:

• unit test -> make test
• integration tests -> make test-intergration
• e2e tests -> make test-e2e

We support lws as the default workload, however, most of the cases mutli-hosts is not needed, even with Llama3.1 405B. So maybe this is a better choice.

What happened:

Once the Playground containers the dot in the name, for example qwen2-0.5b, failed to create Pods. we should add validations to the name field.

What you expected to happen:

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:

Environment:

• Kubernetes version (use kubectl version):
• LWS version:
• llmaz version (use git describe --tags --dirty --always):
• Cloud provider or hardware configuration:
• OS (e.g: cat /etc/os-release):
• Kernel (e.g. uname -a):
• Install tools:
• Others:

What would you like to be added:

TGI is also a popular inference backend that we should support. More importantly, we should set this as an example once people would like to support another backend in the future.

Why is this needed:

Adopte more backends and make this a reference.

Completion requirements:

This enhancement requires the following artifacts:

• Design doc
• API change
• Docs update

The artifacts should be linked in subsequent comments.

What would you like to be added:

ollama provides sdk for integrations, we can easily integrate with it, one of the benefits I can think of is ollama maintains a bunch of quantized models, we can leverage.

Why is this needed:

Ecosystem integration.

Completion requirements:

This enhancement requires the following artifacts:

• Design doc
• API change
• Docs update

The artifacts should be linked in subsequent comments.

What would you like to be added:

Models can be loaded with different accelerators, for example, llama2-70b can be located with 2 A100 80GB or 4 A100 40 GB, we should support this. And usually high-end GPUs will be stockout frequently, this can help improve the SLO of services.

Why is this needed:

Cost saving and SLO consideration.

Completion requirements:

This enhancement requires the following artifacts:

• Design doc
• API change
• Docs update

The artifacts should be linked in subsequent comments.

Because of the huge size of model weights and the GIL lock of python, let's move to rust instead for performance save.

Right now, we support inference engines like vllm for inference, what if people want calling OpenAPIs like chatGPT, it's easy to integrate.

What would you like to be added:

Kueue is a great project which focus on job queueing and resource management, it can also support inference service by managing Pods, it's efficient because we have the overview of the cluster and we know much whether the GPU kinds are insufficient or not, comparing to runtime failover.

What's more, if kueue is already part of your component, it would be really great!

Why is this needed:

Fungibility capacity.

Completion requirements:

This enhancement requires the following artifacts:

• Design doc
• API change
• Docs update

The artifacts should be linked in subsequent comments.

Right now, when I add a label to the Model, the most important one, llmaz.io/model-family-name, I hope it can be passed through to the Pods used this model, however, lws can't do that right now.

What would you like to be added:

Right now, model management is a tricky problem in the cluster, it's big, so we need to cache them in the node just like images, however, kubelet will take over the image lifecycle management but files, so that's a problem, and will not be tacked in the near future, so maybe we need to manage the models manually and make it aware by the scheduler to make pod placement decisions.

Why is this needed:

Efficient pod scheduling with models

Completion requirements:

This enhancement requires the following artifacts:

• Design doc
• API change
• Docs update

The artifacts should be linked in subsequent comments.

Right now model is a cluster scope obj, should it be namespaced? Maybe we should add a namespacedModel, or a better choice is the model obj is visible to selected namespaces, but this is not following the design of kubernetes.

when patching the inferenceService, it reports metadata.creationTimestamp: field not declared in schema". Log the creationTimestamp, it has a null value, "metadata":{"creationTimestamp":null,.

See related issue: kubernetes/kubernetes#109427

See https://github.com/kubernetes-sigs/lws

What would you like to be added:

Currently, the model-runner is about 56MB, however, the model-loader is about 466MB, we should try to smaller the size.

Why is this needed:

Fast startup.

Completion requirements:

This enhancement requires the following artifacts:

• Design doc
• API change
• Docs update

The artifacts should be linked in subsequent comments.

We support env & vars right now in Playground, however, sometimes, some parameters should be protected and now show directly in Pod yaml, just like the huggingface_token or something like that, we should support secret and plumbed into pod spec.

What would you like to be added:

llama.cpp supports running inference on cpus this is useful for uses has no GPU accelerators, in fact, also helpful to llmaz as we have no GPU servers right now.

What's more, llama.cpp also supports multi-host inference, see https://github.com/ggerganov/llama.cpp/tree/master/examples/rpc

Completion requirements:

This enhancement requires the following artifacts:

• Design doc
• API change
• Docs update

The artifacts should be linked in subsequent comments.

We only have one replica for controller right now.

What happened:

Let's image there's no Model in the cluster, I created a Playground, because no Model exists, inference Pods will not be created. However, if I created the corresponding Model, the Playground should be triggered to create Services and then Pods. Right now, this is not true.

The solution is quite simple, make Playground watch for model creation events.

What you expected to happen:

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:

Environment:

• Kubernetes version (use kubectl version):
• LWS version:
• llmaz version (use git describe --tags --dirty --always):
• Cloud provider or hardware configuration:
• OS (e.g: cat /etc/os-release):
• Kernel (e.g. uname -a):
• Install tools:
• Others:

The ability to serve model from docker image(contains the model weight), whether we can integrate with ollama?

What would you like to be added:

Right now we don't support multi-host in playground, but this is supported in Service, we should provide an example.

Why is this needed:

Completion requirements:

This enhancement requires the following artifacts:

• Design doc
• API change
• Docs update

The artifacts should be linked in subsequent comments.

The general idea is quite similar to deployment, we'll set the pods number equal to the replicas.
What we need here:

• Reconcile logics
• Test framework
• Examples

We can use nginx:1.14.2 as the default image at first.

What happened:

Because of InftyAI/omnistore#12, we'll always download the model weights even we cached them in the host machine.

What you expected to happen:

When model weights downloaded, we should not reload again.

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:

Environment:

• Kubernetes version (use kubectl version):
• LWS version:
• llmaz version (use git describe --tags --dirty --always):
• Cloud provider or hardware configuration:
• OS (e.g: cat /etc/os-release):
• Kernel (e.g. uname -a):
• Install tools:
• Others:

What would you like to be added:

For inference scenarios, prompts management is an important part of it.

Why is this needed:

Easy to use for inference users.

Completion requirements:

This enhancement requires the following artifacts:

• Design doc
• API change
• Docs update

The artifacts should be linked in subsequent comments.

The first minor release should includes all the stuff in https://github.com/InftyAI/llmaz/milestone/1

What would you like to be added:

This is how it should look like:

    volumeMounts:
    - mountPath: /dev/shm
      name: dshm

But this memory size is unknown.

Why is this needed:

Accelerate model loading and model inference.

Completion requirements:

This enhancement requires the following artifacts:

• Design doc
• API change
• Docs update

The artifacts should be linked in subsequent comments.

SGLang see https://github.com/sgl-project/sglang

We have several integration tests for webhooks, however, they're the very simple ones, we need more, like covering the update cases.

What would you like to be added:

Similar to kserve https://kserve.github.io/website/latest/modelserving/v1beta1/custom/custom_model/#parallel-model-inference

Why is this needed:

Completion requirements:

This enhancement requires the following artifacts:

• Design doc
• API change
• Docs update

The artifacts should be linked in subsequent comments.

What would you like to be added:

We may have s3, gcs, oss as model storage, we should load them during runtime.

Why is this needed:

Support more model sources.

Completion requirements:

This enhancement requires the following artifacts:

• Design doc
• API change
• Docs update

The artifacts should be linked in subsequent comments.

This is how the code tree looks like right now:

├── api
│   ├── core
│   │   └── v1alpha1
│   │       ├── groupversion_info.go
│   │       ├── model_types.go
│   │       ├── types.go
│   │       └── zz_generated.deepcopy.go
│   └── inference
│       └── v1alpha1
│           ├── config_types.go
│           ├── groupversion_info.go
│           ├── playground_types.go
│           ├── service_types.go
│           ├── types.go
│           └── zz_generated.deepcopy.go

However we do not need the core folder, this is just because a bug of code-generator, which is fixed in kubernetes/kubernetes#125162, we should remove the folder once kubernetes v1.31 is released.