opendatahub-io / modelmesh Goto Github PK

This project forked from kserve/modelmesh

Distributed Model Serving Framework

License: Apache License 2.0

Shell 1.35% Python 0.20% Java 97.54% Thrift 0.48% Dockerfile 0.43%

modelmesh's Introduction

ModelMesh

The ModelMesh framework is a mature, general-purpose model serving management/routing layer designed for high-scale, high-density and frequently-changing model use cases. It works with existing or custom-built model servers and acts as a distributed LRU cache for serving runtime models.

For full Kubernetes-based deployment and management of ModelMesh clusters and models, see the ModelMesh Serving repo. This includes a separate controller and provides K8s custom resource based management of ServingRuntimes and InferenceServices along with common, abstracted handling of model repository storage and ready-to-use integrations with some existing OSS model servers.

For more information on supported features and design details, see these charts.

Get Started

To learn more about and get started with the ModelMesh framework, check out the documentation.

Developer guide

Use the developer guide to learn about development practices for the project.

modelmesh's People

Contributors

Stargazers

Watchers

Forkers

anishasthana red-hat-data-services robgeada jason4849 jooho zdtsw-forking vaibhavjainwiz israel-hdez spolti odh-rhtap test-opendatahub-io heyselbi rpancham odh-on-pz ruivieira-forking

modelmesh's Issues

Fix CI configuration

Currently, CI is not working properly because tests are not running properly.

This repository would need a similar fix to the one implemented here: openshift/release#40548.
Once a similar fix is done, it is going to be possible to remove no longer used files, like:

Backport fix manifest incompatibility in ModelMesh labels

Create a sample notebook to do model serving calls

Currently there are no real hints on how to use model serving service. User needs to know how I can take my Python in a notebook using this model, to a request. What does that look like? Develop a sample notebook to do the calls internal/external.

Move from odh-manifests to modelmesh source repo

/kind feature

Describe the solution you'd like
As part of the opendatahub-io/opendatahub-operator#338, to ensure all manifests required by modelmesh in https://github.com/opendatahub-io/odh-manifests/tree/master/model-mesh can be mapped to one of the source code repo

Test issue - please ignore

Testing GH projects - please ignore the issue

PayloadProcessor log entries not useful

The log entries emitted by LoggingPayloadProcessor are NOT very useful because the actual payload data is not displayed correctly. It only logs the number of bytes instead of the actual payload.

Example:

The value of the data field is data=86B.

{"instant":{"epochSecond":1694276873,"nanoOfSecond":586464093},"thread":"pool-4-thread-1","level":"INFO","loggerName":"com.ibm.watson.modelmesh.payload.LoggingPayloadProcessor","message":"Payload: Payload{id='69-1', modelId='with-logging', method='inference.GRPCInferenceService/ModelInfer', status=request, metadata=Metadata(content-type=application/grpc,user-agent=grpc-go/1.51.0,grpcgateway-user-agent=python-requests/2.31.0,authorization=Basic xxxx,grpcgateway-authorization=Basic xxxx,grpcgateway-accept=*/*,grpcgateway-content-type=application/json,x-forwarded-host=with-logging-wines.apps.fmflask2.faisallabs.net,x-forwarded-for=3.105.194.26, 10.131.2.48, ::1), data=63B}","endOfBatch":false,"loggerFqcn":"org.apache.logging.slf4j.Log4jLogger","contextMap":{},"threadId":45,"threadPriority":5}
{"instant":{"epochSecond":1694276873,"nanoOfSecond":586949553},"thread":"pool-4-thread-1","level":"INFO","loggerName":"com.ibm.watson.modelmesh.payload.LoggingPayloadProcessor","message":"Payload: Payload{id='69-1', modelId='with-logging', method='inference.GRPCInferenceService/ModelInfer', status=Status{code=OK, description=null, cause=null}, metadata=Metadata(grpc-encoding=identity,grpc-accept-encoding=gzip), data=86B}","endOfBatch":false,"loggerFqcn":"org.apache.logging.slf4j.Log4jLogger","contextMap":{},"threadId":45,"threadPriority":5}

Wouldn't it be useful if we log the actual payload data instead?

At line 108 of the Payload.java file, i think it will be more useful if we use ByteBuf.toString(charset) instead of .readallbytes().

Build issue - modelmesh need buildId to build a new image

ModelMesh needs buildId to work properly but openshift-ci can not set it dynamically. Without the buildId, modelmesh runtime always failed.

To build from a Dockerfile alone, this buildID must be dynamically generated and used in the Dockerfile. You can use a script to do this. The following command is for building modelmesh image so you can refer this.

GIT_COMMIT=$(git rev-parse HEAD)
BUILD_ID=$(date '+%Y%m%d')-$(git rev-parse HEAD | cut -c -5)
IMAGE_TAG_VERSION=0.11.0-alpha
IMAGE_TAG=${IMAGE_TAG_VERSION}-$(git branch --show-current)_${BUILD_ID}

docker build -t quay.io/opendatahub/modelmesh:${IMAGE_TAG_VERSION} \
    --build-arg imageVersion=${IMAGE_TAG} \
    --build-arg buildId=${BUILD_ID} \
    --build-arg commitSha=${GIT_COMMIT} .