einstack / glide Goto Github PK

Buildout the Unified response based on GEP 0002.

Define language model pools based on the configuration passed.

Investigate if the default slogs module added in go1.21 is a good option to rely for logging functionality-wise.

If not, lets configure zap.

Requirements:

• it's possible to set log level/severity via configs(account for that since configs are still in TODO)
• it should be possible to log records as colored plain text and as JSON
• it's possible to log a message and additional fields
• it's fine to just write logs to stdout

Receiving following error when deploying the following config:

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x2 addr=0x0 pc=0x10509150c]

goroutine 1 [running]:
glide/pkg.NewGateway(0x1400052e0c0)
        /Users/max/code/Glide/pkg/gateway.go:39 +0x3c
glide/pkg/cmd.NewCLI.func1(0x14000690600, {0x1050c419c?, 0x7?, 0x1050c173f?})
        /Users/max/code/Glide/pkg/cmd/cli.go:26 +0x54
github.com/spf13/cobra.(*Command).execute(0x14000690600, {0x14000122220, 0x2, 0x2})
        /Users/max/go/pkg/mod/github.com/spf13/[email protected]/command.go:983 +0x840
github.com/spf13/cobra.(*Command).ExecuteC(0x14000690600)
        /Users/max/go/pkg/mod/github.com/spf13/[email protected]/command.go:1115 +0x344
github.com/spf13/cobra.(*Command).Execute(0x14000070728?)
        /Users/max/go/pkg/mod/github.com/spf13/[email protected]/command.go:1039 +0x1c
main.main()
        /Users/max/code/Glide/main.go:25 +0x20
exit status 2
make: *** [run] Error 1

telemetry:
logging:
level: info
encoding: console # console, json
# other configs

api:
http:
listen_addr: 0.0.0.0:7685
max_body_size: "2Mi"
tls:
  ca_path:
  cert_path:
# other configs

routes:
language:
- id: openai-pool
  strategy: priority # round-robin, weighted-round-robin, priority, least-latency, priority, etc.
  models:
    - id: primary
      openai: # cohere, azureopenai, gemini, other providers we support
        model: gpt-3.5-turbo
        api_key: "sk-"
        default_params:
          temperature: 0.1

    - id: secondary
      cohere:
        model: command-light
        apiKey: ""
        default_params: # set the default request params
          temperature: 0.1

- id: latency-critical-pool
  strategy: least-latency
  models:
    - id: primary
      timeout_ms: 200
      openai:
        model: gpt-3.5-turbo
        api_key: "sk-"
     
    - id: secondary
      timeout_ms: 200
      cohere:
        api_key: ""

- id: cohere-openai-ab-test
  strategy: weighted-round-robin
  models:
    - id: openai
      weight: 30
      openai:
        api_key: "sk-"
    - id: cohere
      weight: 70
      cohere:
        api_key: ""

API keys redacted.

Create a Helm chart to deploy Glide into a Kubernetes cluster easier

Max has API key/access on Azure

• Need to read endpoint data from config

Configure a Github Action pipeline to build a new version of Glide on merges into the main branch.

GEP: EinStack/geps#7

The field under 'id' needs to be updated from 'openai' to 'provider'

routers:
  language:
    - id: test-router
      models:
        - id: openai
          openai:
            api_key: "sk"
        - id: cohere
          openai:
            api_key: ""

Probably need to move provider config details to a YAML or other config setup

Init a CLI and add there one command that would run the Glide server with a health API

We need to find a way to track model latency that would be independent from the response size.

Options:

• use the time to fist byte metric (needs to instrument clients to get that info)
• count the generated tokens in the response and dividing it by the response time
• something else?

Reference:

• #70 (comment)

Reference:

• https://github.com/hertz-contrib/swagger

Transforms the request body to match the structure required by the AI provider.
It also ensures the values for each parameter are within the minimum and maximum
constraints defined in the provider's configuration. If a required parameter is missing,
it assigns the default value from the provider's configuration.

Simplify representation

Investigate why goleak reports that netpoll leaves pending goroutines in:
https://github.com/modelgateway/Glide/blob/38c56a2cf0c650ec56f7476453a14a0ec1ed2a10/leak_test.go#L7-L6
and enable the test back.

A bug report in Netpoll's Github: cloudwego/netpoll#305

Update Unified Request schema to accept different prompts for different models

Integrate with Codecov to track and visualize codebase coverage

https://github.com/zilliztech/GPTCache/blob/main/docs/usage.md#Use-GPTCache-server

Setup documentation for Glide and deploy/host it somewhere.

Docs frameworks

Mkdocs + Material Theme

A popular option in Python community. I have been using it for my Hyx project.

Reference:

• https://squidfunk.github.io/mkdocs-material/
  Example of Real Docs:
• https://hyx.readthedocs.io/en/latest/
• https://fastapi.tiangolo.com/

Hugo + Docsy

A popular option in CNCF community.

Reference:

• https://www.docsy.dev/

Example of Real Docs:

• https://kubernetes.io/docs/concepts/scheduling-eviction/pod-overhead/
• https://opentelemetry.io/docs/demo/

Docusaurus

A popular option in general, gives a good looking UI but unsure about of effort to get it working/maintain.

Reference:

Examples of Real Docs:

• https://chaos-mesh.org/docs/
• https://apisix.apache.org/docs/apisix/getting-started/README/

Docs Hosting

Popular Options are:

• https://readthedocs.org/ (using it for all my OS projects)
• https://www.netlify.com/ (using it for my website)
• https://vercel.com/dashboard (was using it before)

Docs Licence

We can go with Apache 2.0 for code snippets + CC-BY-4.0 for the main content.

Allow to setup TLS for Glide HTTP API, so it's possible to:

• securely connect to API
• (potentially) setup mTLS authentication

Add a new set of API to support embeddings:

• come up with the unified embedding request/response schemas
• add a new model type to the routers config
• define at least one provider with embedding capabilities

• Map the OpenAI response to a struct then feed into Unified response
• Change ProviderResponse to ModelResponse

Max has API key

I want to test building via docker

Implement fallbacking on provider failures to another healthy provider.

Create a GEP to explain how will that work (GEP0005).

Create a Grafana dashboard to monitor Glide and health of models it uses

Let Glide be configured via a YAML file.

Requirements:

• define the structure of the config file via a GEP (e.g. routing config, server configs, etc). [GEP0001]
• implement the config subsystem that would include reading the file content and its validation
• make the config accessible across the service
• it should be possible to include env vars into the config (e.g. to load sensitive data like API keys)
• explore ways to include separate files as a value of a config (e.g. to load sensitive data like API keys)

Reloading the service on config file change is outside of this task (#28)

Implement a mock provider called FaultyProvider that would enable us to do resiliency testing of Glide emulating super hostile conditions like:

• temporary transient errors
• rate limits
• sudden response time increases
• canceled in-flight responses

Readme.md is a face of the project. Let's fix the broken links and update copy to make good impression on people visiting the repo

In order to collect metrics on different aspects of the system (e.g. how many times OpenAI failed/accepted requests), we need to setup OpenTelemetry.

Also, we could use OpenTelemetry to implemented distributed tracing, so it's seamlessly integrated into business services that communicates with it and instrumented by OTEL. Also, this should be useful for us to have a better debugging capabilities.

Requirements:

• setup OTEL
• make it possible to disable traces/metrics via config if that's unused by users
• emit gateway-specific metrics (instrument Fiber with OTEL)
• allow to control cardinality of metrics

Tasks

• #219
• #220
• #221
• #222
• #223
• #224

Ignore Nilaway false alarmsand enable the check in the CI

Setup a new GH Action pipeline on each PR that includes:

• linting
• tests
• try to build the app