This project allows operating a computer using a guitar, or bass guitar, or other instrument. Notes on the instrument can be mapped to key presses, key combos, or mouse movements/clicks. Check out this tweet for a demo.

This software was used in the creation of these epic YouTube videos:

I Played Minecraft Using ONLY A Bass

The project works by running fundamental-frequency estimation on microphone input, and mapping these to keyboard/mouse inputs. State is kept to ensure each played note is pressed only once (rather than the full note duration), to allow certain keys to be "held down" as long as the note is played, to allow chorded keys/key combos to be entered, to detect when a note is stopped, etc.

Modify the note-mappings in config.py to configure what note maps to what keyboard or mouse action. For more complicated configuration, you might need to modify the code in keymaps.py.

1. Ensure you have Python 3 available on your system

2. Run the following shell commands (macOS):

brew install portaudio
pipenv shell
pip install --global-option='build_ext' --global-option='-I/usr/local/include' --global-option='-L/usr/local/lib' pyaudio
pipenv install

3. Run offkeyboard:

python offkeyboard/__init__.py

As long as the script is active, playing notes defined in the configuration will send the associated keyboard/mouse events to the OS.

Press CTRL+C to exit.

It's difficult to send simulated mouse events on modern macOS. To send mouse events, we need to install a kernel extension and communicate with it.

For this, I used the (very cool!) foohid library. This will typically require disabling SIP, so continue at your own risk.

I'm intentionally leaving out detailed instructions for this bit for now, as disabling SIP and installing random kernel drivers is a security risk.

MIT License