This is a simple REST API wrapper around GPT4All
It is built using the Nest framework for running locally or on your own server.
There is also an AWS CDK stack for AWS Lambda deployment of the API. It works, but it is rather slow even on the highest memory setting.
$ pnpm install
Download the latest models.json into the models folder. Then download one of the models in that JSON file also into the models folder. You can see better overview over the models on the GPT4All website.
Create a .env
file in the root folder and add the following content:
GPT4ALL_MODEL=FILE_NAME_OF_THE_MODEL
Where FILE_NAME_OF_THE_MODEL
is the name of the model file you downloaded, e.g. mistral-7b-openorca.gguf2.Q4_0.gguf
.
You can optionally set the number of CPU threads by adding
CPU_THREADS=8
# development
$ pnpm run start
# watch mode
$ pnpm run start:dev
# production mode
$ pnpm run start:prod
You can run the TypeScript file cli/cli.ts
as an example.
For this deployment the model mistral-7b-openorca.gguf2.Q4_0.gguf
will be used. It can be changed by editing the source code. Download this model into cdk/docker/models
We will build an amd64 docker image.
Go into cdk/docker
and run ./build.sh
You now have an image with the name gpt4all-api:latest
.
It is possible to try out the AWS lambda image locally. Call the script ./run_image.sh
and afterward the script ./run_lambda.sh
.
Set the URI of your ECR repository as the URI environment variable: URI=ECR_REPOSITORY_URI
First login into the ECR docker repository by calling ./login.sh
Then tag the image with the URI of your repository by calling ./tag_image.sh
Finally, push the image to the repository by calling ./push.sh
Go to the cdk
folder. If the name of your repository is not gpt4all-api
then set it as an environment variable in you terminal:
REPOSITORY_NAME=your-repository-name
Install all packages by calling pnpm install
.
Bootstrap the deployment: pnpm cdk bootstrap
Deploy the stack using pnpm cdk deploy
On the terminal you will see the output Gpt4AllStack.gpt4alllambdaname
that shows you the function name. Go to that function in the AWS web console. This lambda is supposed to be run behind an API Gateway. This Gateway is not yet implemented in that CDK stack. Here is a test event to run the lambda:
{
"body": "{\"prompt\":\"What is 1 + 1?\",\"temperature\":0.5}"
}
- Author - Achim Rohn
This project is MIT licensed.