Comments (5)
yuzisun
commented on September 24, 2024
@robertgshaw2-neuralmagic were you able to manage to load the 70b model using oci storage ?
from kserve.
robertgshaw2-neuralmagic
commented on September 24, 2024
@yuzisun i managed to get it working with a 7b model!
Going to try it again in a multi node setup with this model scale then move onto the 70b
from kserve.
robertgshaw2-neuralmagic
commented on September 24, 2024
@yuzisun is oci is the recommended pathway moving forward?
from kserve.
yuzisun
commented on September 24, 2024
@yuzisun is oci is the recommended pathway moving forward?
Not sure if it is best practice to store 130g model in an OCI image, the other alternative solution is to store the model on local NVMe volumes and load the model with PVC.
from kserve.
robertgshaw2-neuralmagic
commented on September 24, 2024
That makes sense - I will try both
Just curious: is there a specific model size you recommend for the OCI images?
The docs suggest that this OCI pathway was built specifically for the LLM serving use case
from kserve.
Related Issues (20)
- Support Multi-Node Inference and Serving.
- Release 0.14 Tracking HOT 1
- Storage initializer missing HOT 5
- when use huggingface backend get `trust_remote_code=True` error
- Can kserve support eks pod download s3 model from another aws account? HOT 3
- An attempt to run microservice on Windows fails with NotImplementedException
- integrating the kserve hosted model with a third party application
- Consistent Error Responses between Huggingface and vLLM Backends in huggingfaceserver
- SDLC is broken when KServe is installed with helm
- Deploying GGUF models HOT 1
- CRDS not installing first when using kserve-crd chart as a dependency
- RBAC permission for secret is too broad HOT 2
- MLflow model cannot be served if storageUri doesn't end in "/"
- Revision create failed - Duplicate value: "kserve-provision-location"
- aiffairness image dependency issue HOT 4
- Knative- queue proxy resource setting is not correct. HOT 1
- Kserve Storage-initializer Optimization HOT 3
- Support OCI image volume mounts HOT 3
- Kserve agent exec probe failed HOT 3
- What are the best practices for using `ClusterServingRuntime` and `ServingRuntime`? HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from kserve.