This script helps you backup your RoamResearch graphs!
This script automatically
- Downloads a markdown archive of your RoamResearch workspace
- Downloads a json archive of your RoamResearch workspace
- Download the full EDN of your RoamResearch workspace
- Unzips them to your git directory
- Format your markdown, including with backlinks
- Commits and push the difference to GitHub
V.02:
- Use Selenium library, and roam-to-git seems to be much faster and stable π₯
- Download the EDN archive
See it in action!. This repo is updated using roam-to-git.
ππ NEW ππ : The Unofficial backup of the official RoamResearch Help Database
- You have a backup if RoamResearch loses some of your data.
- You have a history of your notes.
- You can browse your GitHub repository easily with a mobile device
Note: Erik Newhard's guide shows an easy way of setting up GitHub Actions without using the CLI.
With gh: gh repo create notes (yes, it's private)
Or manually
- Go to github.com/your/repository/settings/secrets
Due to the limitations of OAuth and complexities with tokens, we are unable to snapshot accounts that are set up with the Login with Google option as of now.
To set up backup in this case, you will need to create(not exactly) a native account from your old Google Account, which is as simple as using the reset password link found in Roam.
Once you've reset your password, use the following steps to finish setting up your backup!
Add 3 (separate) secrets where the names are
ROAMRESEARCH_USER
ROAMRESEARCH_PASSWORD
ROAMRESEARCH_DATABASE
-
Refer to env.template for more information
-
when inserting the information, there is no need for quotations or assignments
cd notes
mkdir -p .github/workflows/
curl https://raw.githubusercontent.com/MatthieuBizien/roam-to-git-demo/master/.github/workflows/main.yml > \
.github/workflows/main.yml
git add .github/workflows/main.yml
git commit -m "Add github/workflows/main.yml"
git push --set-upstream origin master
- Go to github.com/your/repository/actions
- Your CI job should start in a few seconds
If the backup does not automatically start, try pushing to the repository again
This section is based on this article on GitLab blog: https://about.gitlab.com/blog/2017/11/02/automating-boring-git-operations-gitlab-ci/
Create a project for your notes. We will refer to it as
YOUR_USER/YOUR_PROJECT.
Generate a new key pair that will be used by the CI job to push the new commits.
$ ssh-keygen -f gitlab-ci-commit
Generating public/private rsa key pair.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in gitlab-ci-commit-test
Your public key has been saved in gitlab-ci-commit-test.pub
The key fingerprint is:
SHA256:HoQUcbUPJU2Ur78EineqA6IVljk8ZD9XIxiGFUrBues agentydragon@pop-os
The key's randomart image is:
+---[RSA 3072]----+
| .o=O*..o++. |
| .*+.o. o+o |
| +.=. .oo. . |
| X o.. o . |
| . = oS o. |
| + .. o ... |
| + . .o o ... |
| . E .. o .. |
| .o. .. |
+----[SHA256]-----+DO NOT commit the private key (gitlab-ci-commit).
This step allows the CI job to push when identified by the public key.
Go to Project Settings β Repository (https://gitlab.com/YOUR_USER/YOUR_PROJECT/-/settings/repository) β Deploy Keys.
Paste the content of the public key file (gitlab-ci-commit-test.pub), and
enable write access for the public key.
Click "Add Key".
This step gives the CI job the private key so it can authenticate against GitLab.
Go to Project Settings β Pipelines (https://gitlab.com/YOUR_USER/YOUR_PROJECT/-/settings/ci_cd) β Variables.
Click "Add variable", with name GIT_SSH_PRIV_KEY, and paste in the content
of the private key file (gitlab-ci-commit). You probably want to mark
"Protect". You might want to look up GitLab docs on protected branches.
Click "Add variable".
Also add the following variables with appropriate values:
ROAMRESEARCH_USERROAMRESEARCH_PASSWORDROAMRESEARCH_DATABASE
In your repo, create and commit a gitlab_known_hosts file containing the
needed SSH known_hosts entry/entries for the GitLab instance. It will be used
by the CI job to check that it's talking to the right server.
This should work as of 2021-01-04:
# Generated by @agentydragon at 2021-01-04 at by `git fetch`ing a GitLab repo
# with an empty ~/.ssh/known_hosts file.
|1|zIQlCRxv+s9xhVCAfGL2nvaZqdY=|jbPpD9GNaS/9Z4iJzE9gw2XCo20= ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFSMqzJeV9rUzU4kWitGjeR4PWSa29SPqJ1fVkhtj3Hw9xjLVXVYrU9QlYWrOLXBpQ6KWjbjTDTdDkoohFzgbEY=
|1|uj60xYhsW2vAM8BpQ+xZz51ZarQ=|BNIJlvu4rNcmrxd60fkqpChrf9A= ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFSMqzJeV9rUzU4kWitGjeR4PWSa29SPqJ1fVkhtj3Hw9xjLVXVYrU9QlYWrOLXBpQ6KWjbjTDTdDkoohFzgbEY=
Create a .gitlab-ci.yml file:
backup:
when: manual
before_script:
- 'which ssh-agent || ( apt-get update -y && apt-get install openssh-client -y )'
- eval $(ssh-agent -s)
- ssh-add <(echo "$GIT_SSH_PRIV_KEY")
- git config --global user.email "[email protected]"
- git config --global user.name "roam-to-git automated backup"
- mkdir -p ~/.ssh
- cat gitlab_known_hosts >> ~/.ssh/known_hosts
# (Taken from: https://github.com/buildkite/docker-puppeteer/blob/master/Dockerfile)
# We install Chrome to get all the OS level dependencies, but Chrome itself
# is not actually used as it's packaged in the pyppeteer library.
# Alternatively, we could include the entire dep list ourselves
# (https://github.com/puppeteer/puppeteer/blob/master/docs/troubleshooting.md#chrome-headless-doesnt-launch-on-unix)
# but that seems too easy to get out of date.
- apt-get install -y wget gnupg ca-certificates
- wget -q -O - https://dl-ssl.google.com/linux/linux_signing_key.pub | apt-key add -
- echo "deb [arch=amd64] http://dl.google.com/linux/chrome/deb/ stable main" >> /etc/apt/sources.list.d/google.list
- apt-get update
- apt-get install google-chrome-stable libxss1 python3-pip -y
- pip3 install git+https://github.com/MatthieuBizien/roam-to-git.git
# TODO(agentydragon): Create and publish Docker image with all deps already
# installed.
script:
# Need to clone the repo again over SSH, since by default GitLab clones
# the repo for CI over HTTPS, for which we cannot authenticate pushes via
# pubkey.
- git clone --depth=1 [email protected]:YOUR_USER/YOUR_PROJECT
- cd YOUR_PROJECT
# --no-sandbox needed because Chrome refuses to run as root without it.
- roam-to-git --browser-arg=--no-sandbox .Commit and push.
Go to https://gitlab.com/YOUR_USER/YOUR_PROJECT/-/jobs,
press the Play button:
Shortly after you should see an automated commit added to master.
To run the script, say, every hour, go to the project's
CI / CD β Schedules (https://gitlab.com/YOUR_USER/YOUR_PROJECT/-/pipeline_schedules).
Click "New schedule", fill out and submit (like this for every 15 minutes):
Note: if your file system is not case-sensitive, you will not backup notes that have the same name in different cases
With pipx (if you don't know pipx, you should look at it, it's wonderful!)
pipx install git+https://github.com/MatthieuBizien/roam-to-git.git
With gh: gh repo create notes (yes, it's private)
Or manually
Then run git push --set-upstream origin master
curl https://raw.githubusercontent.com/MatthieuBizien/roam-to-git/master/env.template > notes/.env- Fill the .env file:
vi .env - Ignore it:
echo .env > notes/.gitignore; cd notes; git add .gitignore; git commit -m "Initial commit"
- Run the script:
roam-to-git notes/ - Check your GitHub repository, it should be filled with your notes :)
One-liner to run it with a cron every hour:
echo "0 * * * * '$(which roam-to-git)' '$(pwd)/notes'" | crontab -
NB: there are issues on Mac with a crontab.
Making roam-to-git foolproof is hard, as it depends on Roam, on GitHub Action or the local environment,
on software not very stable (pyppeteer we still love you π )
and on the correct user configuration.
For debugging, please try the following:
- Check that the environment variables
ROAMRESEARCH_USER,ROAMRESEARCH_PASSWORD,ROAMRESEARCH_DATABASEare correctly setup - Login into Roam using the username and the password. You may want to ask a new password if you have enabled Google Login, as it solved some user problems.
- Run
roam-to-git --debugto check the authentication and download work - Look at the traceback
- Look for similar issues
- If nothing else work, create a new issue with as many details as possible. I will try my best to understand and help you, no SLA promised π
- Download automatically from RoamResearch
- Create Cron
- Write detailed README
- Publish the repository on GitHub
- Download images (they currently visible in GitHub, but not in the archive so not saved in the repository π)
- Format
[[links]] - Format
#links - Format
attribute:: - Format
[[ [[link 1]] [[link 2]] ]] - Format
((link))
- Add backlinks reference to the notes files
- Integrate the context into the backlink
- Manage
/in file names
- Format
{{TODO}}to be compatible with GitHub markdown - Format `{{query}}``
- Push it to GitHub
- Add example repository
- Make the backup directory configurable
- Publicize it
- Test it/make it work on Windows
- Pre-configure a CI server so it can run every hour without a computer Thanks @Stvad for #4!
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent