victordibia / llmx Goto Github PK

Is there paras like 'num_gpus=2'。How can I load model into mult GPUS

Using Hf provider gives llmx.generators.text.hf_textgen.hftextgenerator() got multiple values for keyword argument 'provider' error with version 0.0.15a0.

With version 0.0.14a0 works fine.

python3 --version
Traceback (most recent call last):
File "", line 1, in
NameError: name 'python3' is not defined

What

For certain use cases e.g., cli scripts and web apps, it is important to provide some mechanism to externally set the default LLM provider and configuration

Work Items

• set LLMX_CONFIG environment variable which should be a path to the configuration for a model
• in llmx.generators.text.textgen.py , check for an available config and use to instantiate llm() . Merge provided kwargs with the content of the config file, with kwargs taking precedence

Hi @victordibia, curious - what was missing from litellm to be useful here?

https://github.com/BerriAI/litellm

What

In many cases it is useful to keep track of the token usage for each query and associated costs. Different model provider apis handle this differently (some provide this information as part of a generate query response) and the task is to return accurate usage states in a unified format for all model providers

Work Items

• OpenAI/AzureOpenAI
• HuggingFace
• PALM
  • Extend response from palm api to include usage information.
  • Standardize on usage format for all apis

Palm api response provides the following fields.

"metadata": {
    "tokenMetadata": {
      "input_token_count": {
        "total_tokens": integer,
        "total_billable_characters": integer
      },
      "output_token_count": {
        "total_tokens": integer,
        "total_billable_characters": integer
      }
    }
  }

It would be interesting to try out the recently released lida library with LLMs running locally using Llama.cpp.

Could llmx support such "offline"/embedded or standalone more resource constrained scenarios with LLMs using only CPUs?

If so, can you provide an outline of steps required?

Hi,
could you please add support for Makersuite PALM API (https://makersuite.google.com/)?
Thank you for previously adding support for Vertex AI PALM API.

Thanks in advance

Currently the data needs to be stored as a pandas dataframe in memory. Is there any scope to extend this project to large scale databases that use pyspark dataframes or sql

We should replace api_base to base_url as in openai documentation instead of removing it.

llmx package is used with lida.

Update the openai version please

You tried to access openai.ChatCompletion, but this is no longer supported in openai>=1.0.0 - see the README at https://github.com/openai/openai-python for the API.

You can run openai migrate to automatically upgrade your codebase to use the 1.0.0 interface.

Alternatively, you can pin your installation to the old version, e.g. pip install openai==0.28

A detailed migration guide is available here: openai/openai-python#742

Hello Victor :)

I really like what you are doing and want to use the recently released lida library for projects in my organization.
However, my organization uses Azure OpenAI.
For this we need to be able to specfiy three openai properties:

• openai.api_type
• openai.api_base
• openai.version

Here is an example authorization for an Azure OpenAI instance:

import openai
openai.api_type = "azure"
openai.api_base = "https://yourendpoint.openai.azure.com/"
openai.api_version = "2023-07-01-preview"
openai.api_key = os.getenv("OPENAI_API_KEY")

Can you please add these properties to llmx so it is possible to use lida?

Best regards
Marvin

I've been trying to set up a working example of Azure OpenAI but in vain...

I think the documentation about this LLM provider should be very clear and not just dealt with in a sentence.. it's not about just setting up the env vars.. I've been trying to work around it for almost one hour now, but I don't know how to set the model, etc. I think it's really poorly documented.

Could anyone help please ?

MODEL_NAME=demo-llm
AZURE_OPENAI_BASE=https://******************.openai.azure.com/
AZURE_OPENAI_API_KEY=*********************
AZURE_OPENAI_API_TYPE=azure
AZURE_OPENAI_API_VERSION=2023-07-01-preview

I tried this:

provider = sanitize_provider("azureopenai")

text_gen = OpenAITextGenerator(
                # provider="azureopenai",
               api_type="azure",
               api_base=os.environ["AZURE_OPENAI_BASE"],
               api_key=os.environ["AZURE_OPENAI_API_KEY"],
               api_version="2023-07-01-preview",
               models =[os.environ["MODEL_NAME"]],
                provider=provider,
)

and this

text_gen = llm(provider="azureopenai",
               api_type="azure",
               api_base=os.environ["AZURE_OPENAI_BASE"],
               api_key=os.environ["AZURE_OPENAI_API_KEY"],
               api_version="2023-07-01-preview",
               model =os.environ["MODEL_NAME"],
               ) # for azure openai

I get Error processing file: The API deployment for this resource does not exist. If you created the deployment within the last 5 minutes, please wait a moment and try again.