lazyagi / lazyllm Goto Github PK
View Code? Open in Web Editor NEWEasiest and laziest way for building multi-agent LLMs applications.
Home Page: https://lazyllm.readthedocs.io/
License: Apache License 2.0
Easiest and laziest way for building multi-agent LLMs applications.
Home Page: https://lazyllm.readthedocs.io/
License: Apache License 2.0
I traced the request processing, it seems in lazyllm/components/deploy/relay/server.py generate
function
if getattr(getattr(func, '_meta', func.__class__), '__enable_request__', False):
output = func(h.make_request(input, **kw))
else:
output = func(input, **kw)
As the TrainableModule's __enable_request__
is indeed False, the keyword extraction 'h.make_request' won't be executed. Consequently, the context set by 'llm_chat_history' won't take effect.
# coding:utf-8
import lazyllm
from lazyllm import launchers, deploy, LazyLlmRequest
# ====================================== Setting parameters:
base_model = "xxxx/internlm2-chat-20b"
deploy_config = {"launcher": launchers.slurm(partition="pat_rd", nnode=1, nproc=1, ngpus=1, sync=False), "port": 22332}
prompter = lazyllm.ChatPrompter(instruction=" 模仿孔子的说话方式进行对话")
tm = lazyllm.TrainableModule(base_model, "").prompt(prompter).deploy_method(deploy.vllm, **deploy_config)
m = lazyllm.ServerModule(tm)
m.start()
params = {
tm._module_id: {"llm_chat_history": [["**的首都是哪个城市", "**的首都是北京"]]},
}
req = LazyLlmRequest(
input="它有哪些著名的景点",
global_parameters=params,
)
# llm_chat_history takes effect if it's sent to tm driectly
print(tm("它有哪些著名的景点", llm_chat_history=[["**的首都是哪个城市", "**的首都是北京"]]))
# BUG: if tm._module_id's history will not take effect
print(m(req))
slurm 20.11.8-7.7
vllm 0.4.3
Develop a simple version is enough currently.
Assigned to @yewentao256
Hi, I found that the input and output of toolManager in lazyLLM need to be passed in the input and output results of LLM. I am confused, why does toolManager rely on these contents? Shouldn't it only accept the tool name and input parameters?
import lazyllm
t = lazyllm.TrainableModule('internlm2-chat-7b')
Traceback (most recent call last):
File "", line 1, in
File "/root/miniconda3/envs/py11/lib/python3.11/site-packages/lazyllm/module/module.py", line 382, in init
self.base_model = ModelDownloader(lazyllm.config['model_source']).download(base_model)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/miniconda3/envs/py11/lib/python3.11/site-packages/lazyllm/components/utils/downloader/model_downloader.py", line 68, in download
if model in model_name_mapping.keys() and self.model_source in model_name_mapping[model][['source']].keys():
~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^
TypeError: unhashable type: 'list'
I encountered an error while trying to import the module lazyllm.components.deploy.relay.server
using importlib.import_module
.
import importlib
importlib.import_module('lazyllm.components.deploy.relay.server')
Error Traceback (most recent call last)
Cell In[1], line 2
1 import importlib
----> 2 importlib.import_module('lazyllm.components.deploy.relay.server')
File ~/.conda/envs/lazyllm_docs/lib/python3.10/importlib/__init__.py:126, in import_module(name, package)
124 break
125 level += 1
--> 126 return _bootstrap._gcd_import(name[level:], package, level)
File <frozen importlib._bootstrap>:1050, in _gcd_import(name, package, level)
File <frozen importlib._bootstrap>:1027, in _find_and_load(name, import_)
File <frozen importlib._bootstrap>:1006, in _find_and_load_unlocked(name, import_)
File <frozen importlib._bootstrap>:688, in _load_unlocked(spec)
File <frozen importlib._bootstrap_external>:883, in exec_module(self, module)
File <frozen importlib._bootstrap>:241, in _call_with_frames_removed(f, *args, **kwds)
File ~/project/lazyllm/lazyllm/components/deploy/relay/server.py:38
35 def load_func(f):
36 return cloudpickle.loads(base64.b64decode(f.encode('utf-8')))
---> 38 func = load_func(args.function)
39 if args.before_function:
40 before_func = load_func(args.before_function)
File ~/project/lazyllm/lazyllm/components/deploy/relay/server.py:36, in load_func(f)
35 def load_func(f):
---> 36 return cloudpickle.loads(base64.b64decode(f.encode('utf-8')))
File ~/.conda/envs/lazyllm_docs/lib/python3.10/base64.py:87, in b64decode(s, altchars, validate)
85 if validate and not re.fullmatch(b'[A-Za-z0-9+/]*={0,2}', s):
86 raise binascii.Error('Non-base64 digit found')
---> 87 return binascii.a2b_base64(s)
Error: Incorrect padding
lazyllm_docs
)~/project/lazyllm/lazyllm/components/deploy/relay/server.py
The error is related to incorrect padding in a base64 encoded string used in the load_func
function.
This occurs at the following line in server.py
:
func = load_func(args.function)
Description:
Hi team,
I noticed that the CI workflow defined in main.yml
for pull requests shows significant time differences between the BasicTests
and AdvancedTests
stages. Specifically:
BasicTests
takes approximately 24m 54sAdvancedTests
takes approximately 28m 8sCould someone help clarify why there is such a notable time difference between these two test stages? Are there specific tests or steps within AdvancedTests
that inherently require more time, or is there potential for optimization?
Additionally, any insights into how we might reduce the overall testing time for both stages would be greatly appreciated.
Model support must accommodate both inference and fine-tuning, with a higher priority on inference. Even if only inference is available, it should still be added to the TrainableModule.
Text-to-Image Model
Stable Diffusion 3 has seen significant improvements in image quality, text rendering, and multi-theme generation.
Stable Diffusion 3 Medium
Image-Text Understanding Model
InternVL 1.5 supports image description and image Q&A. While it performs slightly worse than Gemini Ultra in TextQA (more focused on image description), it is the best open-source multimodal solution for comprehensive scenarios.
InternVL-Chat V1-5 (inference using lmdeploy)
Text-to-Speech Model
A new highly praised open-source speech synthesis project trained on 10 million hours of data.
ChatTTS
The open-source model generates 90-second music clips. The non-open-source Stable Audio 2.0 can produce up to 3-minute-long music, songs, melodies, or vocals.
Stable Audio Open 1.0
Speech-to-Text Model
OpenAI's Whisper is significantly outperformed by DAMO Academy's FunASR in Chinese. The best results are achieved by combining a recognition model with endpoint detection and punctuation prediction. Three models are needed, listed from top to bottom: speech recognition, endpoint detection, and punctuation prediction.
Speech Recognition
Endpoint Detection
Punctuation Prediction
hi,
I am reaching out to invite you to participate in our upcoming AGI2B streaming session on WeChat, which focuses on the AI agent business. We are planning to discuss the AI agent business landscape in America, and your insights would be incredibly valuable.
Your expertise in demoing AI products and engaging local customers in China would add great depth to our discussion. We believe that your participation would be beneficial not only for our audience but also for expanding the reach and impact of AI agent technologies.
Could you please let me know if you are available to join us? We are flexible with the scheduling and can accommodate your availability.
my wechat: jachenms
```
if source.lower() == "openai":
return OpenAIModule(**params)
elif source.lower() == "glm":
return GLMModule(**params)
elif source.lower() == "kimi":
return MoonshotAIModule(**params)
elif source.lower() == "sensenova":
return SenseNovaModule(**params)
elif source.lower() == "qwen":
return QwenModule(**params)
elif source.lower() == "doubao":
```
maybe you can use locals()[f'{source}Module']
, source shoule be 'OpenAI', 'GLM', 'SenseNova' and so on, and config check should use lower()
3. move argument-check of doubao to DoubaoModule.init()
Refactor Creation and Usage of TrainableModule and ServerModule:
TrainableModule
and ServerModule
to support accessing deployed services via URL.TrainableModule
should be able to share a module that uses the same service but has an independent prompt.Add Global Configuration Pool:
session-id
(default to thread ID if not provided).LazyLLMRequest
to pass global parameters.Refactor Trace/Exception Mechanism:
ModuleResponse
class.Support Streaming of Intermediate Results:
During inference, when the available number of GPUs exceeds the demand, the tasks are distributed across multiple computing nodes. This is achieved through a load balancing mechanism to manage the task execution.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.