This is a project dedicated to building a real-life version of GLaDOS.
This is a hardware and software project that will create an aware, interactive, and embodied GLaDOS.
This will entail:
- Train GLaDOS voice generator
- Generate a prompt that leads to a realistic "Personality Core"
- Generate a MemGPT medium- and long-term memory for GLaDOS
- Give GLaDOS vision via LLaVA
- Create 3D-printable parts
- Design the animatronics system
The initial goals are to develop a low-latency platform, where GLaDOS can respond to voice interactions within 600ms.
To do this, the system constantly records data to a circular buffer, waiting for voice to be detected. When it's determined that the voice has stopped (including detection of normal pauses), it will be transcribed quickly. This is then passed to streaming local Large Language Model, where the streamed text is broken by sentence, and passed to a text-to-speech system. This means further sentences can be generated while the current is playing, reducing latency substantially.
- The other aim of the project is to minimize dependencies, so this can run on constrained hardware. That means no PyTorch or other large packages.
- As I want to fully understand the system, I have removed a large amount of redirection: which means extracting and rewriting code. i.e. as GLaDOS only speaks English, I have rewritten the wrapper around espeak and the entire Text-to-Speech subsystem is about 500 LOC and has only 3 dependencies: numpy, onnxruntime, and sounddevice.
This will be based on servo- and stepper-motors. 3D printable STL will be provided to create GlaDOS's body, and she will be given a set of animations to express herself. The vision system will allow her to track and turn toward people and things of interest.
If you want to install the TTS Engine on your machine, please follow the steps below. This has only been tested on Linux, but I think it will work on Windows with small tweaks. If you are on windows, I would recommend WSL with an Ubuntu image. Proper Windows and Mac support is in development.
-
Install the
espeak
synthesizer according to the installation instructions for your operating system. -
Install the required Python packages, e.g., by running
pip install -r requirements.txt
-
For the LLM, you have two option:
- compile llama.cpp:
- use:
git submodule update --init --recursive
to pull the llama.cpp repo - move the the right subdirectory:
cd submodules/llama.cpp
- compile llama.cpp:
make server LLAMA_CUDA=1
- use:
- install an inference backend yourself, such as Ollama or Llamafile:
- Find and install a backend with an OpenAI compatible API (most of them)
- then edit the glados_config.yaml
- update
completion_url
to the URL of your local server - remove the LlamaServer configurations (make them null)
- compile llama.cpp:
-
For voice recognition, install I have precompiled Whisper.cpp, using
make libwhisper.so
and supplied the "libwhisper.so" file in the "glados" folder. For Windows, check out the discussion in my whisper pull request. This is a temporarily solution until we have build scripts for each platform. -
Download the models:
and put them in the "models" directory.
To start GLaDOS, use:
python glados.py
You can stop with "Ctrl-c".
You can test the systems by exploring the 'demo.ipynb'.