Join our community Discord: AI Stack Devs
![Screen Shot 2023-08-14 at 10 01 00 AM](https://private-user-images.githubusercontent.com/3489963/260520547-a4c91f17-23ed-47ec-8c4e-9f9a8505057d.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjQ1NDc1MzQsIm5iZiI6MTcyNDU0NzIzNCwicGF0aCI6Ii8zNDg5OTYzLzI2MDUyMDU0Ny1hNGM5MWYxNy0yM2VkLTQ3ZWMtOGM0ZS05ZjlhODUwNTA1N2QucG5nP1gtQW16LUFsZ29yaXRobT1BV1M0LUhNQUMtU0hBMjU2JlgtQW16LUNyZWRlbnRpYWw9QUtJQVZDT0RZTFNBNTNQUUs0WkElMkYyMDI0MDgyNSUyRnVzLWVhc3QtMSUyRnMzJTJGYXdzNF9yZXF1ZXN0JlgtQW16LURhdGU9MjAyNDA4MjVUMDA1MzU0WiZYLUFtei1FeHBpcmVzPTMwMCZYLUFtei1TaWduYXR1cmU9NTQwZjJjYTVhZmZlYTQ3ZjFmM2ZjYWZhZWYwMzhmOWUxY2M1MmU1OGU5YmQ1NGY3YzEzOTA2MWNmNzVlYTkxOCZYLUFtei1TaWduZWRIZWFkZXJzPWhvc3QmYWN0b3JfaWQ9MCZrZXlfaWQ9MCZyZXBvX2lkPTAifQ.DMFEf2k3ZvKXmwtXwgjmsiLPwza3waQQoT7x8efOAVE)
AI Town is a virtual town where AI characters live, chat and socialize.
This project is a deployable starter kit for easily building and customizing your own version of AI town. Inspired by the research paper Generative Agents: Interactive Simulacra of Human Behavior.
The primary goal of this project, beyond just being a lot of fun to work on, is to provide a platform with a strong foundation that is meant to be extended. The back-end natively supports shared global state, transactions, and a simulation engine and should be suitable from everything from a simple project to play around with to a scalable, multi-player game. A secondary goal is to make a JS/TS framework available as most simulators in this space (including the original paper above) are written in Python.
- ๐ป Stack
- ๐ง Installation
- ๐ค Customize - run YOUR OWN simulated world
- ๐ฉโ๐ป Setting up local inference
- ๐ Credits
- Game engine, database, and vector search: Convex
- Auth: Clerk
- Text model: OpenAI
- Deployment: Vercel
- Pixel Art Generation: Replicate, Fal.ai
- Background Music Generation: Replicate using MusicGen
- Local inference: Ollama
git clone https://github.com/a16z-infra/ai-town.git
cd ai-town
npm install
npm run dev
npm run dev
will fail asking for environment variables.
Enter them in the environment variables on your Convex dashboard to proceed.
You can get there via npx convex dashboard
or dashboard.convex.dev
See below on how to get the various environment variables.
a. Set up Clerk
- Go to https://dashboard.clerk.com/ and click on "Add Application"
- Name your application and select the sign-in providers you would like to offer users
- Create Application
- Add
VITE_CLERK_PUBLISHABLE_KEY
andCLERK_SECRET_KEY
to.env.local
VITE_CLERK_PUBLISHABLE_KEY=pk_***
CLERK_SECRET_KEY=sk_***
- Go to JWT Templates and create a new Convex Template.
- Copy the JWKS endpoint URL for use below.
b. OpenAI API key
Visit https://platform.openai.com/account/api-keys to get your OpenAI API key and set the
OPENAI_API_KEY
environment variable in your Convex deployment (see below).
c. Replicate API key (Optional)
For Daily background music generation, create a
Replicate account and create a token in your Profile's
API Token page.
Add the token as REPLICATE_API_TOKEN
in your Convex environment variables.
d. Add environment variables to the Convex backend
Environment variables for a Convex backend is configured through the dashboard.
Go to "settings" and add the following environment variables.
CLERK_ISSUER_URL
should be the domain in the URL from the JWKS endpoint (e.g., https://your-issuer-url.clerk.accounts.dev/).
This link will take you there for your most recent deployment: https://dashboard.convex.dev/deployment/settings/environment-variables?var=CLERK_ISSUER_URL
OPENAI_API_KEY sk-*******
CLERK_ISSUER_URL https://****
REPLICATE_API_TOKEN **** #optional
To run both the front and and back end:
npm run dev
You can now visit http://localhost:5173.
If you'd rather run the frontend in a separate terminal from Convex (which syncs your backend functions as they're saved), you can run these two commands:
npm run dev:frontend
npm run dev:backend
See package.json for details, but dev:backend runs npx convex dev
Note: The simulation will pause after 5 minutes if the window is idle.
Loading the page will unpause it.
You can also manually freeze & unfreeze the world with a button in the UI.
If you want to run the world without the
browser, you can comment-out the "stop inactive worlds" cron in convex/crons.ts
.
To stop the back end, in case of too much activity
This will stop running the engine and agents. You can still run queries and run functions to debug.
npx convex run testing:stop
To restart the back end after stopping it
npx convex run testing:resume
To kick the engine in case the game engine or agents aren't running
npx convex run testing:kick
To archive the world
If you'd like to reset the world and start from scratch, you can archive the current world:
npx convex run testing:archive
Then, you can still look at the world's data in the dashboard, but the engine and agents will no longer run.
You can then create a fresh world with init
.
npx convex run init
To clear all databases
You can wipe all tables with the wipeAllTables
testing function.
npx convex run testing:wipeAllTables
To pause your backend deployment
You can go to the dashboard to your deployment settings to pause and un-pause your deployment. This will stop all functions, whether invoked from the client, scheduled, or as a cron job. See this as a last resort, as there are gentler ways of stopping above. Once you
- Register an account on Vercel and then install the Vercel CLI.
- If you are using Github Codespaces: You will need to install the Vercel CLI and authenticate from your codespaces cli by running
vercel login
. - Deploy the app to Vercel with
vercel --prod
.
Before you can run the app, you will need to make sure the Convex functions are deployed to its production environment.
- Run
npx convex deploy
to deploy the convex functions to production - Run
npx convex run init --prod
If you have existing data you want to clear, you can run npx convex run testing:wipeAllTables --prod
NOTE: every time you change character data, you should re-run
npx convex run testing:wipeAllTables
and then
npm run dev
to re-upload everything to Convex.
This is because character data is sent to Convex on the initial load.
However, beware that npx convex run testing:wipeAllTables
WILL wipe all of your data.
-
Create your own characters and stories: All characters and stories, as well as their spritesheet references are stored in characters.ts. You can start by changing character descriptions.
-
Updating spritesheets: in
data/characters.ts
, you will see this code:
export const characters = [
{
name: 'f1',
textureUrl: '/assets/32x32folk.png',
spritesheetData: f1SpritesheetData,
speed: 0.1,
},
...
];
You should find a sprite sheet for your character, and define sprite motion / assets in the corresponding file (in the above example, f1SpritesheetData
was defined in f1.ts)
- Update the background (environment):
data/firstmap.ts
is where the map gets loaded. The easiest way to export a tilemap is by using Tiled -- Tiled exports tilemaps as a CSV and you can convert CSV to a 2d array accepted by firstmap.ts - Change the background music by modifying the prompt in
convex/music.ts
- Change how often to generate new music at
convex/crons.ts
by modifying thegenerate new background music
job
We support using Ollama for conversation generations, but don't yet support using a local model for generating embeddings.
Steps to switch to using Ollama:
-
Install Ollama When Ollama runs on your laptop, it by default uses http://localhost:11434 as an endpoint for generation. Next, we need to set up a tunnelmole or ngrok tunnel so Convex can access it:
Using Tunnelmole
Tunnelmole is an open source tunneling tool.
You can install Tunnelmole using one of the following options:
- NPM:
npm install -g tunnelmole
- Linux:
curl -s https://tunnelmole.com/sh/install-linux.sh | sudo bash
- Mac:
curl -s https://tunnelmole.com/sh/install-mac.sh --output install-mac.sh && sudo bash install-mac.sh
- Windows: Install with NPM, or if you don't have NodeJS installed, download the
exe
file for Windows here and put it somewhere in your PATH.
Once Tunnelmole is installed, run the following command:
tmole 11434
Tunnelmole should output a unique url once you run this command.
Using Ngrok
Ngrok is a popular closed source tunneling tool.
Once ngrok is installed and authenticated, run the following command:
ngrok http http://localhost:11434
Ngrok should output a unique url once you run this command.
- NPM:
-
Add Ollama endpoint to Convex
- Go to https://dashboard.convex.dev/deployment/settings/environment-variables?var=OLLAMA_HOST
- Add
OLLAMA_HOST = [your tunnelmole/ngrok unique url from the previous step]
- You might also want to set:
ACTION_TIMEOUT
to100000
or more, to give your model more time to run.NUM_MEMORIES_TO_SEARCH
to1
, to reduce the size of conversation prompts.
By default, we use llama2-7b
model on Ollama.
If you want to customize which model to use, you can set OLLAMA_MODEL
variable under Environment Variables.
Ollama model options can be found here.
- All interactions, background music and rendering on the component in the project are powered by PixiJS.
- Tilesheet:
- https://opengameart.org/content/16x16-game-assets by George Bailey
- https://opengameart.org/content/16x16-rpg-tileset by hilau
- We used https://github.com/pierpo/phaser3-simple-rpg for the original POC of this project. We have since re-wrote the whole app, but appreciated the easy starting point
- Original assets by ansimuz
- The UI is based on original assets by Mounir Tohami