question: Possibility of simplifying running AI model-specific wasm Workloads with Embedded Configuration about wasmedge HOT 7 OPEN

Hi @sohankunkerkar

We have been trying very hard to get WasmEdge GGML plugin working with crun-wasm. It is high on our list.

A sticking point is that crun does not detect the correct CUDA version on the host machine. We are not sure why. @hydai perhaps can shed more light on this?

from wasmedge.

@juntao Thanks for the reply. Let me know if you need some traction on the crun-specific issue.

from wasmedge.

Hi @CaptainVincent
We would like to have an issue tracking the crun+WasmEdge+ggml plugin. Could you please raise a new issue and talk about the current status of the run-related integration?

from wasmedge.

#3217 has been added.

from wasmedge.

@sohankunkerkar You mention "simplify" running llama in containerized environment, could you please guide me to some resource with steps to run llama2 with crun wasm environment which works currently .

from wasmedge.

https://wasmedge.org/docs/start/build-and-run/docker_wasm_gpu
https://wasmedge.org/docs/start/build-and-run/podman_wasm_gpu
I have two documents that you might find useful.

from wasmedge.

@shiveshcodes, I believe the links provided by @CaptainVincent would be helpful for your needs. If you're interested in the cri-o + crun workflow, you can also check out this link: https://github.com/sohankunkerkar/wasm-kubecon-demos/tree/main.

from wasmedge.