Comments (30)
stevenlu137
commented on May 28, 2024
Hello, the latest version of the master branch has eliminated the default dependence on docker. You only need to delete config.json and reinstall the latest version. If you want to continue to use the old version, you can use the --localExecution option.
from ailice.
LanceLake
commented on May 28, 2024
Ok. This is confusing to me then.
I run Ooga Booga on the same computer. I was hoping there was someplace I can put in my local IP, tell it what character I want it to be and then have it use that as the model.
I.e. The model will be able to do command line stuff, browse the web and so on as it wants to do. I'm looking to give this thing total freedom (it's a virtual machine) to do whatever it wants.
Is there something I'm missing? Where do I tell it "Access the OpenAI that OogaBooga is running at this address and port, tell it what character to use and when it responds, show the text it got and perform the requested action.
Is that even possible currently?
from ailice.
stevenlu137
commented on May 28, 2024
I'm not sure if I understood your question correctly.
Firstly, AIlice has built-in support for some open-source models and the OpenAI API, but it cannot share the backend language model (LLM) with Ooga Booga. You can enable support for GPT-4 by configuring your OpenAI API key directly in the config.json file. After that, you can run the command ailice_main --modelID=oai:gpt-4-1106-preview --prompt="main" to initiate it. Running AIlice in a virtual machine allows you to execute code in a local environment without modifying the config.json, providing complete freedom.However, if you have previously installed an older version of AIlice, you need to delete the old config.json before launching the new version to generate a correct configuration file.
Currently, communication with AIlice is primarily done through the command line. Although there is a web version called "ailice_web", it is not yet fully developed, resulting in a suboptimal user experience. In essence, AIlice is a chatbot that can assist you in performing various tasks through a command-line interface.
Role definition is a common usage method in other agent frameworks, but it is not necessary in AIlice for regular users. You can simply chat with AIlice, present your requirements, and she will automatically create agents with various roles to address your needs.
For those interested in developing a new agent, you may need to read the README.md and the code under the prompts/ directory, as there is currently no comprehensive development documentation available.
from ailice.
LanceLake
commented on May 28, 2024
Ok.. Simple question then.
I have an LLM file.. mixtral-8x7b-moe-rp-story.Q6_K.gguf..
How do I put it into the system? Where do I enter in that line into the config so that it will find it in X directory? I think that is where I am misunderstanding. Once I can specify a model, I think I can get the rest.
from ailice.
stevenlu137
commented on May 28, 2024
In AIlice, the model used is specified through command-line parameters (--modelID), where the ID has a prefix of "oai:" or "hf:", representing whether it is from OpenAI or Hugging Face. For more details, you can refer to the README.md.
If you want to run AIlice using a locally available weight file, there might be some challenges at the moment. AIlice utilizes the transformers library to load models. Therefore, for weight formats supported by the transformers library, you can specify the location of the weight file by setting --modelID=file:path_to_the_weights. However, if the transformers library does not support the format of the weight file, it may not be possible.
AIlice only supports a limited number of open-source LLMs for now. While it's possible to add support for other models, it still requires coding effort. The model you mentioned is not currently supported. You can check the list of supported open-source models by opening core/llm/ALLMMeta.py. Alternatively, you can refer to the relevant sections in the README.md to understand how to add support for an LLM.
It's worth noting that currently, many open-source models, including Mixtral MoE, may not function properly on AIlice. On the other hand, smaller model like Mistral-7B-OpenOrca performs better. Therefore, it is recommended to use built-in support with --modelID=oai:gpt-4-1106-preview or --modelID=hf:Open-Orca/Mistral-7B-OpenOrca.
from ailice.
stevenlu137
commented on May 28, 2024
My apologies, currently setting the modelID to file:... is not an official feature. You need to manually add this modelID in ALLMMeta.py for it to work. Please disregard my previous suggestion. Considering the gguf format you mentioned, it is likely not supported by the transformers library.
from ailice.
LanceLake
commented on May 28, 2024
That's ok. I'll keep an eye on the releases and perhaps in the future, this will work for what I have planned. Thank you again for the help.
from ailice.
LanceLake
commented on May 28, 2024
So I tried the command you suggested and got this error.. Figured I would post it here just in case it's useful.
E:\AIlice\ailice>ailice_main --modelID=hf:ehartford/dolphin-2.5-mixtral-8x7b --prompt="main" --quantization=4bit --contextWindowRatio=0.3
config.json need to be updated.
********************** Initialize *****************************
Your openai chatgpt key (press Enter if not):
********************** End of Initialization *****************************
config.json is located at C:\Users\Lance Lake\AppData\Local\Steven Lu\ailice/config.json
In order to simplify installation and usage, we have set local execution as the default behavior, which means AI has complete control over the local environment. To prevent irreversible losses due to potential AI errors, you may consider one of the following two methods: the first one, run AIlice in a virtual machine; the second one, install Docker, use the provided Dockerfile to build an image and container, and modify the relevant configurations in config.json. For detailed instructions, please refer to the documentation.
The port range of the ext-modules has been changed from 2005-2016 to 59000-59200. If you are using an old version, startup failure will occur after updating the code. Please modify the port number in config.json and rebuild the docker image.
'ps' is not recognized as an internal or external command,
operable program or batch file.
storage started.
browser started.
arxiv started.
google started.
duckduckgo started.
scripter started.
Encountered an exception, AIlice is exiting: Protocol not supported (addr='ipc:///tmp/AIliceStorage.ipc')
File "C:\Python311\Lib\site-packages\ailice-0.2.0-py3.11.egg\ailice\AIliceMain.py", line 100, in main
mainLoop(**kwargs)
File "C:\Python311\Lib\site-packages\ailice-0.2.0-py3.11.egg\ailice\AIliceMain.py", line 50, in mainLoop
clientPool.Init()
File "C:\Python311\Lib\site-packages\ailice-0.2.0-py3.11.egg\ailice\common\ARemoteAccessors.py", line 18, in Init
self.pool = {storage: makeClient(storage),
^^^^^^^^^^^^^^^^^^^
File "C:\Python311\Lib\site-packages\ailice-0.2.0-py3.11.egg\ailice\common\lightRPC.py", line 126, in makeClient
socket.connect(url)
File "C:\Python311\Lib\site-packages\zmq\sugar\socket.py", line 328, in connect
super().connect(addr)
File "zmq\\backend\\cython\\socket.pyx", line 597, in zmq.backend.cython.socket.Socket.connect
File "zmq\\backend\\cython\\checkrc.pxd", line 28, in zmq.backend.cython.checkrc._check_rc
E:\AIlice\ailice>
from ailice.
stevenlu137
commented on May 28, 2024
This is likely due to Windows not supporting zmq's ipc communication. You can try changing all instances of ipc:///... to tcp://127.0.0.1:5910x(choose a port as your prefer) in the config.json file. Please make sure to modify both "cmd" and "addr" entries accordingly.
You're probably one of the first people to run AIlice on Windows (I haven't tested it on Windows yet), so encountering issues is normal. I apologize for any inconvenience. If you have any new questions or issues, please feel free to contact me.
from ailice.
LanceLake
commented on May 28, 2024
{
"maxMemory": {},
"quantization": null,
"openaiGPTKey": null,
"temperature": 0.0,
"flashAttention2": false,
"speechOn": false,
"contextWindowRatio": 0.6,
"services": {
"storage": {
"cmd": "python3 -m ailice.modules.AStorageChroma --addr=tcp://127.0.0.1:59101",
"addr": "tcp://127.0.0.1:59101"
},
"browser": {
"cmd": "python3 -m ailice.modules.ABrowser --addr=tcp://127.0.0.1:59102",
"addr": "tcp://127.0.0.1:59102"
},
"arxiv": {
"cmd": "python3 -m ailice.modules.AArxiv --addr=tcp://127.0.0.1:59103",
"addr": "tcp://127.0.0.1:59103"
},
"google": {
"cmd": "python3 -m ailice.modules.AGoogle --addr=tcp://127.0.0.1:59104",
"addr": "tcp://127.0.0.1:59104"
},
"duckduckgo": {
"cmd": "python3 -m ailice.modules.ADuckDuckGo --addr=tcp://127.0.0.1:59105",
"addr": "tcp://127.0.0.1:59105"
},
"scripter": {
"cmd": "python3 -m ailice.modules.AScripter --addr=tcp://127.0.0.1:59006",
"addr": "tcp://127.0.0.1:59106"
},
"speech": {
"cmd": "python3 -m ailice.modules.ASpeech --addr=tcp://127.0.0.1:59107",
"addr": "tcp://127.0.0.1:59107"
}
}
}
Gets me..
E:\AIlice\ailice>ailice_main --modelID=hf:ehartford/dolphin-2.5-mixtral-8x7b --prompt="main" --quantization=4bit --contextWindowRatio=0.3
config.json is located at C:\Users\Lance Lake\AppData\Local\Steven Lu\ailice/config.json
In order to simplify installation and usage, we have set local execution as the default behavior, which means AI has complete control over the local environment. To prevent irreversible losses due to potential AI errors, you may consider one of the following two methods: the first one, run AIlice in a virtual machine; the second one, install Docker, use the provided Dockerfile to build an image and container, and modify the relevant configurations in config.json. For detailed instructions, please refer to the documentation.
The port range of the ext-modules has been changed from 2005-2016 to 59000-59200. If you are using an old version, startup failure will occur after updating the code. Please modify the port number in config.json and rebuild the docker image.
'ps' is not recognized as an internal or external command,
operable program or batch file.
storage started.
browser started.
arxiv started.
google started.
duckduckgo started.
scripter started.
Encountered an exception, AIlice is exiting: Resource temporarily unavailable
File "C:\Python311\Lib\site-packages\ailice-0.2.0-py3.11.egg\ailice\AIliceMain.py", line 100, in main
mainLoop(**kwargs)
File "C:\Python311\Lib\site-packages\ailice-0.2.0-py3.11.egg\ailice\AIliceMain.py", line 50, in mainLoop
clientPool.Init()
File "C:\Python311\Lib\site-packages\ailice-0.2.0-py3.11.egg\ailice\common\ARemoteAccessors.py", line 18, in Init
self.pool = {storage: makeClient(storage),
^^^^^^^^^^^^^^^^^^^
File "C:\Python311\Lib\site-packages\ailice-0.2.0-py3.11.egg\ailice\common\lightRPC.py", line 128, in makeClient
ret=ReceiveMsg(socket)
^^^^^^^^^^^^^^^^^^
File "C:\Python311\Lib\site-packages\ailice-0.2.0-py3.11.egg\ailice\common\lightRPC.py", line 23, in ReceiveMsg
return pickle.loads(conn.recv())
^^^^^^^^^^^
File "zmq\\backend\\cython\\socket.pyx", line 805, in zmq.backend.cython.socket.Socket.recv
File "zmq\\backend\\cython\\socket.pyx", line 841, in zmq.backend.cython.socket.Socket.recv
File "zmq\\backend\\cython\\socket.pyx", line 199, in zmq.backend.cython.socket._recv_copy
File "zmq\\backend\\cython\\socket.pyx", line 194, in zmq.backend.cython.socket._recv_copy
File "zmq\\backend\\cython\\checkrc.pxd", line 22, in zmq.backend.cython.checkrc._check_rc
E:\AIlice\ailice>
I tried just 5910 for all of them. Then I tried 59100 for all of them. Then I tried seperate ports in the range of the yellow warning. Same error for all of them.
Also, don't say you are sorry for this. I don't mind helping debug. :) What should I try next. :)
from ailice.
LanceLake
commented on May 28, 2024
As a side note..
"speech": {
"cmd": "python3 -m ailice.modules.ASpeech --addr=tcp://127.0.0.1:59107",
"addr": "tcp://127.0.0.1:59107"
}
If you are running it with a command line switch of addr, then you should have that be an override. The default of running it without that extension should be it looks up the address in the config.
Perhaps there is a reason you are doing it this way.. But it would seem safer in the long run to not include the --addr and have it look up the command address in the config file itself (so that you don't end up with someone putting one address in the cmd line and another in the addr line. :)
from ailice.
stevenlu137
commented on May 28, 2024
Hey, go get the latest code of the master branch. I have run it in the local windows environment successfully.
It is best to delete the previous config.json first.
Regarding the repeated addr settings, I also realize that it confuses users, and I am considering modifying this. There are indeed reasons why it is like this, but we still make it the first priority to eliminate user confusion.
from ailice.
stevenlu137
commented on May 28, 2024
In addition, in order to allow scripts to execute code, WSL still needs to be installed on Windows (I think you have already installed it). We do not yet have a Windows command line scripter.
from ailice.
LanceLake
commented on May 28, 2024
I pulled the latest version
E:\AIlice\ailice>git pull
remote: Enumerating objects: 14, done.
remote: Counting objects: 100% (14/14), done.
remote: Compressing objects: 100% (3/3), done.
remote: Total 14 (delta 11), reused 14 (delta 11), pack-reused 0
Unpacking objects: 100% (14/14), 1.63 KiB | 3.00 KiB/s, done.
From github.com:myshell-ai/AIlice
cdb9858..045b681 master -> origin/master
Updating cdb9858..045b681
Fast-forward
ailice/AServices.py | 10 +++++++++-
ailice/common/AConfig.py | 29 ++++++++++++++++++++---------
ailice/modules/AScripter.py | 3 ++-
3 files changed, 31 insertions(+), 11 deletions(-)
and I deleted the config. I also reedited it to include the tcp.
{
"maxMemory": {},
"quantization": null,
"openaiGPTKey": null,
"temperature": 0.0,
"flashAttention2": false,
"speechOn": false,
"contextWindowRatio": 0.6,
"services": {
"storage": {
"cmd": "python3 -m ailice.modules.AStorageChroma --addr=tcp://127.0.0.1:59101",
"addr": "tcp://127.0.0.1:59101"
},
"browser": {
"cmd": "python3 -m ailice.modules.ABrowser --addr=tcp://127.0.0.1:59102",
"addr": "tcp://127.0.0.1:59102"
},
"arxiv": {
"cmd": "python3 -m ailice.modules.AArxiv --addr=tcp://127.0.0.1:59103",
"addr": "tcp://127.0.0.1:59103"
},
"google": {
"cmd": "python3 -m ailice.modules.AGoogle --addr=tcp://127.0.0.1:59104",
"addr": "tcp://127.0.0.1:59104"
},
"duckduckgo": {
"cmd": "python3 -m ailice.modules.ADuckDuckGo --addr=tcp://127.0.0.1:59105",
"addr": "tcp://127.0.0.1:59105"
},
"scripter": {
"cmd": "python3 -m ailice.modules.AScripter --addr=tcp://127.0.0.1:59000",
"addr": "tcp://127.0.0.1:59000"
},
"speech": {
"cmd": "python3 -m ailice.modules.ASpeech --addr=tcp://127.0.0.1:59106",
"addr": "tcp://127.0.0.1:59106"
}
}
}
gets me
E:\AIlice\ailice>ailice_main --modelID=hf:ehartford/dolphin-2.5-mixtral-8x7b --prompt="main" --quantization=4bit --contextWindowRatio=0.3
config.json is located at C:\Users\Lance Lake\AppData\Local\Steven Lu\ailice/config.json
In order to simplify installation and usage, we have set local execution as the default behavior, which means AI has complete control over the local environment. To prevent irreversible losses due to potential AI errors, you may consider one of the following two methods: the first one, run AIlice in a virtual machine; the second one, install Docker, use the provided Dockerfile to build an image and container, and modify the relevant configurations in config.json. For detailed instructions, please refer to the documentation.
The port range of the ext-modules has been changed from 2005-2016 to 59000-59200. If you are using an old version, startup failure will occur after updating the code. Please modify the port number in config.json and rebuild the docker image.
'ps' is not recognized as an internal or external command,
operable program or batch file.
storage started.
browser started.
arxiv started.
google started.
duckduckgo started.
scripter started.
Encountered an exception, AIlice is exiting: Resource temporarily unavailable
File "C:\Python311\Lib\site-packages\ailice-0.2.0-py3.11.egg\ailice\AIliceMain.py", line 100, in main
mainLoop(**kwargs)
File "C:\Python311\Lib\site-packages\ailice-0.2.0-py3.11.egg\ailice\AIliceMain.py", line 50, in mainLoop
clientPool.Init()
File "C:\Python311\Lib\site-packages\ailice-0.2.0-py3.11.egg\ailice\common\ARemoteAccessors.py", line 18, in Init
self.pool = {storage: makeClient(storage),
^^^^^^^^^^^^^^^^^^^
File "C:\Python311\Lib\site-packages\ailice-0.2.0-py3.11.egg\ailice\common\lightRPC.py", line 128, in makeClient
ret=ReceiveMsg(socket)
^^^^^^^^^^^^^^^^^^
File "C:\Python311\Lib\site-packages\ailice-0.2.0-py3.11.egg\ailice\common\lightRPC.py", line 23, in ReceiveMsg
return pickle.loads(conn.recv())
^^^^^^^^^^^
File "zmq\\backend\\cython\\socket.pyx", line 805, in zmq.backend.cython.socket.Socket.recv
File "zmq\\backend\\cython\\socket.pyx", line 841, in zmq.backend.cython.socket.Socket.recv
File "zmq\\backend\\cython\\socket.pyx", line 199, in zmq.backend.cython.socket._recv_copy
File "zmq\\backend\\cython\\socket.pyx", line 194, in zmq.backend.cython.socket._recv_copy
File "zmq\\backend\\cython\\checkrc.pxd", line 22, in zmq.backend.cython.checkrc._check_rc
Sadly, I can't run WSL. There's some weird problem with my system that prevents me from running it. When you release a version with a windows command line scripter, then I will recheck out the project.
from ailice.
stevenlu137
commented on May 28, 2024
The "Resource temporarily unavailable" error could be caused by the "python3" you used in "cmd", I replaced it with "python" and it worked. Running AIlice on Windows without WSL would disable scripter, which is essential to automatic programming, so yes, maybe you should wait for a complete support for windows. I'll put this on my agenda.
from ailice.
LanceLake
commented on May 28, 2024
The Python3 in the config was what it auto-generated. But I changed it to python and success... Kind of..
E:\AIlice\ailice>ailice_main --modelID=hf:ehartford/dolphin-2.5-mixtral-8x7b --prompt="main" --quantization=4bit --contextWindowRatio=0.3
config.json is located at C:\Users\Lance Lake\AppData\Local\Steven Lu\ailice/config.json
In order to simplify installation and usage, we have set local execution as the default behavior, which means AI has complete control over the local environment. To prevent irreversible losses due to potential AI errors, you may consider one of the following two methods: the first one, run AIlice in a virtual machine; the second one, install Docker, use the provided Dockerfile to build an image and container, and modify the relevant configurations in config.json. For detailed instructions, please refer to the documentation.
The port range of the ext-modules has been changed from 2005-2016 to 59000-59200. If you are using an old version, startup failure will occur after updating the code. Please modify the port number in config.json and rebuild the docker image.
'ps' is not recognized as an internal or external command,
operable program or batch file.
storage started.
browser started.
arxiv started.
google started.
duckduckgo started.
scripter started.
tokenizer_config.json: 100%|██████████████████████████████████████████████████████████████| 1.70k/1.70k [00:00<?, ?B/s]
tokenizer.model: 100%|██████████████████████████████████████████████████████████████| 493k/493k [00:00<00:00, 3.65MB/s]
added_tokens.json: 100%|████████████████████████████████████████████████████████████████████| 51.0/51.0 [00:00<?, ?B/s]
special_tokens_map.json: 100%|████████████████████████████████████████████████████████████████| 443/443 [00:00<?, ?B/s]
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
config.json: 100%|████████████████████████████████████████████████████████████████████████████| 783/783 [00:00<?, ?B/s]
Encountered an exception, AIlice is exiting: 'mixtral'
File "C:\Python311\Lib\site-packages\ailice-0.2.0-py3.11.egg\ailice\AIliceMain.py", line 100, in main
mainLoop(**kwargs)
File "C:\Python311\Lib\site-packages\ailice-0.2.0-py3.11.egg\ailice\AIliceMain.py", line 65, in mainLoop
llmPool.Init([modelID])
File "C:\Python311\Lib\site-packages\ailice-0.2.0-py3.11.egg\ailice\core\llm\ALLMPool.py", line 22, in Init
self.pool[id] = AModelLLAMA(locType=locType, modelLocation=location)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Python311\Lib\site-packages\ailice-0.2.0-py3.11.egg\ailice\core\llm\AModelLLAMA.py", line 16, in __init__
self.LoadModel(modelLocation)
File "C:\Python311\Lib\site-packages\ailice-0.2.0-py3.11.egg\ailice\core\llm\AModelLLAMA.py", line 30, in LoadModel
self.LoadModel_Default(modelLocation=modelLocation)
File "C:\Python311\Lib\site-packages\ailice-0.2.0-py3.11.egg\ailice\core\llm\AModelLLAMA.py", line 36, in LoadModel_Default
self.model = transformers.AutoModelForCausalLM.from_pretrained(modelLocation,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Python311\Lib\site-packages\transformers\models\auto\auto_factory.py", line 526, in from_pretrained
config, kwargs = AutoConfig.from_pretrained(
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Python311\Lib\site-packages\transformers\models\auto\configuration_auto.py", line 1095, in from_pretrained
config_class = CONFIG_MAPPING[config_dict["model_type"]]
~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Python311\Lib\site-packages\transformers\models\auto\configuration_auto.py", line 792, in __getitem__
raise KeyError(key)
E:\AIlice\ailice>
from ailice.
stevenlu137
commented on May 28, 2024
This is a weird situation. It seems like the underlying package, transformers, can not recognize the mixtral model. If you are still interested on this issue, you can try another model instead, it can be caused by a lower transformers version or mixtral not supporting windows well, I don't know exactly.
from ailice.
LanceLake
commented on May 28, 2024
Which model do you suggest I use at HF?
from ailice.
stevenlu137
commented on May 28, 2024
ailice_main --modelID=hf:Open-Orca/Mistral-7B-OpenOrca --prompt="main" --quantization=8bit --contextWindowRatio=0.6
There is no open source model that can really do this job, and this model is relatively the best.
from ailice.
LanceLake
commented on May 28, 2024
Ah.. My GPU is very small (8GB), but I have 64GB of core memory. So when I ran that..
killing proc with PID 33372
killing proc with PID 37928
killing proc with PID 54656
killing proc with PID 56356
killing proc with PID 59260
killing proc with PID 63648
storage started.
browser started.
arxiv started.
google started.
duckduckgo started.
scripter started.
tokenizer_config.json: 100%|██████████████████████████████████████████████████████████████| 1.69k/1.69k [00:00<?, ?B/s]
tokenizer.model: 100%|██████████████████████████████████████████████████████████████| 493k/493k [00:00<00:00, 4.26MB/s]
added_tokens.json: 100%|████████████████████████████████████████████████████████████████████| 90.0/90.0 [00:00<?, ?B/s]
special_tokens_map.json: 100%|████████████████████████████████████████████████████████████████| 101/101 [00:00<?, ?B/s]
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
config.json: 100%|████████████████████████████████████████████████████████████████████████████| 623/623 [00:00<?, ?B/s]
Encountered an exception, AIlice is exiting: No GPU found. A GPU is needed for quantization.
File "e:\ailice\ailice\AIliceMain.py", line 100, in main
mainLoop(**kwargs)
File "e:\ailice\ailice\AIliceMain.py", line 65, in mainLoop
llmPool.Init([modelID])
File "e:\ailice\ailice\core\llm\ALLMPool.py", line 22, in Init
self.pool[id] = AModelLLAMA(locType=locType, modelLocation=location)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "e:\ailice\ailice\core\llm\AModelLLAMA.py", line 16, in __init__
self.LoadModel(modelLocation)
File "e:\ailice\ailice\core\llm\AModelLLAMA.py", line 30, in LoadModel
self.LoadModel_Default(modelLocation=modelLocation)
File "e:\ailice\ailice\core\llm\AModelLLAMA.py", line 36, in LoadModel_Default
self.model = transformers.AutoModelForCausalLM.from_pretrained(modelLocation,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Python311\Lib\site-packages\transformers\models\auto\auto_factory.py", line 566, in from_pretrained
return model_class.from_pretrained(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Python311\Lib\site-packages\transformers\modeling_utils.py", line 2864, in from_pretrained
raise RuntimeError("No GPU found. A GPU is needed for quantization.")
E:\AIlice\ailice>
Got anything that can be loaded all in the main memory?
from ailice.
stevenlu137
commented on May 28, 2024
Theoretically you can put it all in memory, but it will be very slow. In the future I will consider integrating faster inference engine, but the current 8GB GPU memory is really not enough. Looks like we can only wait for the new version of AIlice:-)
from ailice.
LanceLake
commented on May 28, 2024
This is why I initially asked that instead of contacting OpenAI, you have some kind of selection that will let me contact my OogaBooga via it's OpenAI URL and use that as the interface to communicate with the AI. That means I can load my model in there and let it run stuff and AIlice can communicate with that to do it's thing like it communicates with OpenAI.
from ailice.
stevenlu137
commented on May 28, 2024
I checked OogaBooga, and now I think I can understand your intentions. If you want to use an LLM service with the same interface as OpenAI to provide reasoning support for AIlice, you need to modify the code that calls self.client.chat.completions.create in core/llm/AModelChatGPT.py. Replace the reasoning logic implemented using the openai library with a lower-level implementation like requests. However, since you are currently unable to run the scripter locally, this effort will not yield good results.
from ailice.
LanceLake
commented on May 28, 2024
Good to know for the future. :)
and yes. I'm requesting in the future to be able to do exactly that. It appears that it's calling openai from the pip install. Once it's built the JSON for sending, then it just needs to send it to the local address. It operates the same as sending to OpenAPI.
But yeah. When I can actually run it, I will return to this and hopefully, something can be coded so I can just plug in the URL for the local LLM and go from there.
from ailice.
robonxt-ai
commented on May 28, 2024
@LanceLake I believe you add in openai.base_url() or something similar (forgot the exact line of code) to set a url to your own api.
EDIT: I believe it is openai.api_base = <your own address>, according to openai/openai-python#209 (comment)
There was other examples with llama.cpp, but at the moment I'm blanking out on where it mentioned changing the base url.
from ailice.
robonxt-ai
commented on May 28, 2024
Got anything that can be loaded all in the main memory?
Go into the code in core/llm folder, find and replace .cuda( with .cpu( (main files to change are AModelLLAMA.py)
I've successfully ran AIlice on a 32GB RAM NUC with that change.
from ailice.
LanceLake
commented on May 28, 2024
s with llama.cpp, but at the moment I'm blanking out on where it mentioned changing the base url.
Where would I place that line?
from ailice.
robonxt-ai
commented on May 28, 2024
Where would I place that line?
Unfortunately I do not know, as I haven't looked deep into the code yet.
One possible place is right above line 12 of core/llm/AModelChatGPT.py; you'll have to figure out what the exact code to put in there though, I don't know enough 😅
from ailice.
jushe
commented on May 28, 2024
@LanceLake
I think I have did the same work in my local. I add a new LLM and can access it form Internet like use the openai.
Here is the list you need to do.
First, add a llm script at alice/core/llm. You can refer to the code I wrote in here. Replace the init and Generate functions with yours. If the prompt need some special format, you may need to add a new Formatter.
Then, add your model to the pool in ailice/core/llm/ALLMPool.py. Ensure it can be access by your cmd.
Third, add your LLM information in ailice/core/llm/ALLMeta.py.
Finally, just run AIlice like ailice_main --modelID={your model id set in ALLMPool and ALLMMeta} --prompt-"main"
Hope this helps you.
from ailice.
stevenlu137
commented on May 28, 2024
Done. please checkout the latest version of dev branch.
from ailice.
Related Issues (20)
- Will there be gguf support? HOT 4
- Recommended Python Version? HOT 9
- Errors running on mac HOT 4
- Agents looping at task execution HOT 3
- The code AIlice wrote always have invalid syntax. HOT 3
- Cannot install on Windows HOT 29
- Speech.PrepareModel not downloading voice model HOT 3
- offline huggingface HOT 8
- Problems with ailice calling the model after installing it through lm studio HOT 4
- 502 Network Error HOT 1
- Installation path HOT 2
- Not actually lightweight ;) HOT 1
- Cannot build Docker container using provided Dockerfile HOT 6
- Adding ollama support fails HOT 3
- Suggestion for claude HOT 1
- Add GPT-4o support HOT 1
- can't run correctly HOT 1
- module storage failed HOT 3
- Error when trying to list current directory content HOT 3
- Small features / corrections HOT 9
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from ailice.