guider's People
guider's Issues
not an issue - fyi - https://github.com/fullstackwebdev/localLLM_guidance
I found your repo digging through github
guidance-ai/guidance#328
I'll have a play with this repo soon - but thought I'd share the above localllm with guidance was great repo to showcase things.
It seems to need some of the code your working on.
UPDATE
I started playing with your code -
I like the streaming code.
I'm really wanting to plug in llama2 model to use with guidance.
I attempt to use this - but no joy.
it's looking for safetensors.
I also have TheBloke_Llama-2-13B-chat-GGML (116gb) - in ./models/
but that doesn't work either.
def run(server_class=HTTPServer, handler_class=MyHandler):
global MODEL_EXECUTOR
# if len(sys.argv) < 3:
# raise Exception(
# "Expected to be invoked with two arguments: model_name and executor"
# )
model_name = "remyxai/ffmperative-7b"# sys.argv[1]
MODEL_EXECUTOR = 'autogptq' #sys.argv[2]
setup_models(model_name)
server_address = ("0.0.0.0", 8000)
httpd = server_class(server_address, handler_class)
print("Starting httpd...\n")
httpd.serve_forever()
if __name__ == "__main__":
run()
looking at directory structure - can I use pytorch_model-00002-of-00002.bin instead of .safetensors I wonder.
-rw-rw-r-- 1 oem oem 630 Aug 9 09:36 config.json
-rw-rw-r-- 1 oem oem 197 Aug 9 09:36 generation_config.json
-rw-rw-r-- 1 oem oem 1.5K Aug 9 09:36 .gitattributes
lrwxrwxrwx 1 oem oem 149 Aug 9 09:36 pytorch_model-00001-of-00002.bin -> ../../../../../home/oem/.cache/huggingface/hub/models--remyxai--ffmperative-7b/blobs/9510ce1eadd1b50ab810c81de3db04c0ea8e9367cb96f836e400b19b83157588
lrwxrwxrwx 1 oem oem 149 Aug 9 09:36 pytorch_model-00002-of-00002.bin -> ../../../../../home/oem/.cache/huggingface/hub/models--remyxai--ffmperative-7b/blobs/530ee273fc7eb08af0d1a4a64ccf467984fbc73e28c6623645ba9c9989bd4d4f
-rw-rw-r-- 1 oem oem 24K Aug 9 09:36 pytorch_model.bin.index.json
-rw-rw-r-- 1 oem oem 4.6K Aug 9 09:36 README.md
-rw-rw-r-- 1 oem oem 414 Aug 9 09:36 special_tokens_map.json
-rw-rw-r-- 1 oem oem 749 Aug 9 09:36 tokenizer_config.json
-rw-rw-r-- 1 oem oem 489K Aug 9 09:36 tokenizer.model
UPDATE
I hacked some code to force guidance to load specific model. but it blows up.
I guess I need a wrapper around this.
def setup_models(model_name: str):
"""
model_name: a Huggingface path like TheBlock/tulu-13B-GPTQ
"""
# A slight improvement in memory usage by using xformers attention:
if "--xformers" in sys.argv:
hijack_llama_attention_xformers()
model = None
print(f"Loading model with executor: {MODEL_EXECUTOR}")
if MODEL_EXECUTOR == "autogptq":
from llama_autogptq import LLaMAAutoGPTQ
model = LLaMAAutoGPTQ(model_name)
elif MODEL_EXECUTOR == "gptq":
from llama_gptq import LLaMAGPTQ
model = LLaMAGPTQ(model_name)
elif MODEL_EXECUTOR == "exllama":
from llama_exllama import ExLLaMA
model = ExLLaMA(model_name)
elif MODEL_EXECUTOR == "ctransformers":
from llama_ctransformers import LLaMATransformer
model = LLaMATransformer(model_name)
elif MODEL_EXECUTOR == "transformers":
from llama_transformer import LLaMATransformer
model = LLaMATransformer(model_name)
elif MODEL_EXECUTOR == "llamacpp":
from llama_cpp_hf import LlamacppHF
model = LlamacppHF(model_name)
global memory
memory = Memory(embedding_model)
print("Memory initialized.")
guidance.llms.Transformers.cache.clear()
# guidance.llm = model
guidance.llm = guidance.llms.transformers.LLaMA("remyxai/ffmperative-7b", device_map="auto")
print(f"Token healing enabled: {guidance.llm.token_healing}")
Initializing ExLlamaGPTQ with model remyxai/ffmperative-7b
Fetching 10 files: 100%|██████████████████████████████| 10/10 [00:00<00:00, 34.08it/s]
Loading config from remyxai/ffmperative-7b/config.json
Traceback (most recent call last):
File "/media/2TB/guider/guider_server.py", line 271, in <module>
run()
File "/media/2TB/guider/guider_server.py", line 262, in run
setup_models(model_name)
File "/media/2TB/guider/guider_server.py", line 58, in setup_models
model = ExLLaMA(model_name)
File "/home/oem/miniconda3/envs/torch2/lib/python3.10/site-packages/guidance/llms/_transformers.py", line 32, in __init__
self.model_obj, self.tokenizer = self._model_and_tokenizer(model, tokenizer, **kwargs)
File "/media/2TB/guider/llama_exllama.py", line 27, in _model_and_tokenizer
(model, tokenizer) = _load(model_dir)
File "/media/2TB/guider/llama_exllama.py", line 57, in _load
exllama_hf = ExllamaHF.from_pretrained(model_dir)
File "/media/2TB/guider/exllama_hf.py", line 112, in from_pretrained
return ExllamaHF(config)
File "/media/2TB/guider/exllama_hf.py", line 21, in __init__
self.ex_model = ExLlama(self.ex_config)
File "/media/2TB/guider/exllama/model.py", line 732, in __init__
with safe_open(self.config.model_path, framework = "pt", device = "cpu") as f:
safetensors_rust.SafetensorError: Error while deserializing header: HeaderTooLarge
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.