Comments (4)
HamidShojanazeri
commented on August 16, 2024
cc @nskool
from sagemaker-pytorch-inference-toolkit.
HamidShojanazeri
commented on August 16, 2024
I believe exposing few knobs for some of the settings including storage for the host instances would be helpful. Thanks @lxning for the offline discussions, it would be great if could add this as a feature to Sagemaker SDK.
from sagemaker-pytorch-inference-toolkit.
lxning
commented on August 16, 2024
According to SM hosting team, currently SM SDK does not support storage size configuration. The only available solution is to change instance type. Pls refer host-instance-storage-volumes-table
from sagemaker-pytorch-inference-toolkit.
HamidShojanazeri
commented on August 16, 2024
@lxning this is a limiting factor, as it is easy to hit the limit mostly on gpu instance 30GB, some of Nvidia dockers similar in this case can go up to 21 GB and heavier workloads that chain multiple models can end up having a large model_artifact size that goes beyond the limit.
from sagemaker-pytorch-inference-toolkit.
Related Issues (20)
- How do I access Custom Attributes from the model during inference? HOT 1
- Need for a minimum reproducible example in readme.md
- No model logs from PyTorch 1.10 SageMaker endpoint HOT 2
- Launch TorchServe without repackaging model contents HOT 5
- Batch Inference does not work when using the default handler
- add environment variable "OMP_NUM_THREADS"
- Document how to locally run the container HOT 2
- using cuda enabled pytorch image
- how to use gpu in sagemaker instance HOT 1
- Is this Dockerfile compatible with sagemaker elastic inference
- MMS mode in inference does not support in GPU instance
- [Question] Using model.mar with built-in handler script
- Specify batch size for MME
- Prepend `code_dir` to `sys.path` rather than `append`
- Incorrect reporting of memory utilisation
- Documentation for inference.py `transform_fn`
- Reuse the requirements.txt installation logic from sagemaker-inference-toolkit
- ModuleNotFoundError: Sagemaker only copies entry_point file to /opt/ml/code/ instead of the holy-cloned source code
- Improve debuggability during model load and inference failures
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from sagemaker-pytorch-inference-toolkit.