Error installing and running Falcon Models about openllm HOT 5 CLOSED

I got the same issue trying to use falcon with openllm==0.1.17

from openllm.

seems like your machine doesn't have enough resource, hence they are offloading it to disk. I will need more bandwidth to investigate how to run falcon on smaller machine

from openllm.

I have the same issue with a massive resource server:

CalledProcessError Traceback (most recent call last)
in
3 import os
4
----> 5 llm = OpenLLM(model_name='falcon', model_id='tiiuae/falcon-40b-instruct', temperature=0.0)
6
7 llm("What is the difference between a duck and a goose? And why there are so many Goose in Canada?")

~/.local/lib/python3.8/site-packages/langchain/llms/openllm.py in init(self, model_name, model_id, server_url, server_type, embedded, **llm_kwargs)
168 # in-process. Wrt to BentoML users, setting embedded=False is the expected
169 # behaviour to invoke the runners remotely
--> 170 runner = openllm.Runner(
171 model_name=model_name,
172 model_id=model_id,

~/.local/lib/python3.8/site-packages/openllm/_llm.py in Runner(model_name, ensure_available, init_local, implementation, **attrs)
1404 behaviour
1405 """
-> 1406 runner = t.cast(
1407 "_BaseAutoLLMClass",
1408 openllm[implementation if implementation is not None else EnvVarMixin(model_name)["framework_value"]], # type: ignore (internal API)

~/.local/lib/python3.8/site-packages/openllm/models/auto/factory.py in create_runner(cls, model_name, model_id, **attrs)
155 A LLM instance.
156 """
--> 157 llm, runner_attrs = cls.for_model(model_name, model_id, return_runner_kwargs=True, **attrs)
158 return llm.to_runner(**runner_attrs)
159

~/.local/lib/python3.8/site-packages/openllm/models/auto/factory.py in for_model(cls, model_name, model_id, return_runner_kwargs, llm_config, ensure_available, **attrs)
133 llm.model_id,
134 )
--> 135 llm.ensure_model_id_exists()
136 if not return_runner_kwargs:
137 return llm

~/.local/lib/python3.8/site-packages/openllm/_llm.py in ensure_model_id_exists(self)
898 Auto LLM initialisation.
899 """
--> 900 output = subprocess.check_output(
901 [
902 sys.executable,

/usr/lib/python3.8/subprocess.py in check_output(timeout, *popenargs, **kwargs)
413 kwargs['input'] = empty
414
--> 415 return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
416 **kwargs).stdout
417

/usr/lib/python3.8/subprocess.py in run(input, capture_output, timeout, check, *popenargs, **kwargs)
514 retcode = process.poll()
515 if check and retcode:
--> 516 raise CalledProcessError(retcode, process.args,
517 output=stdout, stderr=stderr)
518 return CompletedProcess(process.args, retcode, stdout, stderr)

CalledProcessError: Command '['/usr/bin/python3', '-m', 'openllm', 'download', 'falcon', '--model-id', 'tiiuae/falcon-40b-instruct', '--machine', '--implementation', 'pt']' returned non-zero exit status 1.

from openllm.

Same issue as the one on ticket: #121

from openllm.

Please reopen if you still see this error on 0.3.0

from openllm.