stevejenkins / pihole-cloudsync Goto Github PK

I run Pihole and Unbound in a Docker-container, so I guess I have to put the script in the docker-image as well and let it run from there. I'll check out how I can add this to it. Might indeed be convenient. Added a 2nd pihole in another network a few days ago, but adding manually all the lists is one hell of a job, so I quit after a few. :-)

I've written and tested pihole-cloudsync-docker. It's really as simple as packaging this script up into layers on top of the regular pihole docker image, but it also does some clever things not mentioned here by hooking into gravity.sh and list.sh to push immediately to git when config changes.

I'm currently only supporting v4 (latest) but I have builds on docker hub for all architectures supported by pihole (Model 2B - Model 4b)

I'd like to continue testing this for a while on my own, but also talk about how we could either bring my work into this repository or add some docs to link out to my repo.

I think an auto-syncing docker image would provide a lot of value.

Cheers!

It is saying authentication failed

I have confirmed this on two systems. After the cron job runs, the Dashboard in PiHole says there are 0 domains on adlists. I ran the cron command manually and noticed this:

[i] Creating new gravity databases...
 [✗] Unable to copy data from /etc/pihole/gravity.db to /etc/pihole/gravity.db_temp
 Error: near line 15: in prepare, foreign key mismatch - "domainlist_by_group" referencing "domainlist" (1)
Error: near line 19: in prepare, foreign key mismatch - "adlist_by_group" referencing "adlist" (1)

More info: if I run pihole -g from the command line, logged in as root, it works fine.

Everything seems to work smoothly, but after the --initpush and the subsequent --push dont update with the local pihole's lists, only the examples from your github repository.

I can confirm nothing has happened as the commits do not show in github.

Am I missing something here? Thanks.

I've added a few whitelisted domains to my main Pi-hole and unable to push the changes to my Github repo. Whenever I run the --push command, Local Pi-hole lists match remote Git repo. No further action required. is printed to the terminal window and the files in my repo remain unchanged.

Hi,

Starting from Pihole Beta 5.0 is migrating from list storage to a database.
Link: https://pi-hole.net/2020/01/19/announcing-a-beta-test-of-pi-hole-5-0/

Hi - First off, thanks for pulling this together, it is awesome.

I followed the instructions, and when I manually run the script on my 'push' pihole, everything in my git repo is updated as expected. However, when I try to run the script via cron, my repo is not updated.

I piped the output of the cron job to a log file, and it looks like everything is doing what it should, so I"m not sure what I'm doing wrong. Here is the relevant details:

Pihole is running on Ubuntu server 18.04

Line in my root crontab: (this is just for testing purposes, which is why I have it running every minute)

• • • • • sudo /usr/local/bin/pihole-cloudsync/pihole-cloudsync --push > "/home/kevin/logs/pihole-cloudsync.log"

Output of above:

--push option detected. Running in Push/Upload mode.
Local Pi-hole lists are different than remote Git repo. Updating remote repo...
Done!

No matter how many times I let the cron job run, my git repo is never updated.

When I run the script manually with the following (no sudo or anything is needed):

/usr/local/bin/pihole-cloudsync/pihole-cloudsync --push

I get the same output, but the git repo is updated.

Any idea on what I could be doind wrong?

Hi,

I don't use cloud-sync as I have only ever had one RPI and I just live with no DNS if things break lol.

In any case, I saw your suggestion on Reddit that one of my other projects may have been suitable for keeping the file size down to allow you to sync. In actual fact, this wouldn't be appropriate as it wouldn't copy enough information across to make a full 1:1 sync.

Anyway, I made this, this morning.

It basically removes all domains from the gravity table which is usually where the bulk of the DB size is, vacuums the DB (which reduces size down to around 0.1MB), exports it to /etc/pihole/PyPhTP/ and then repopulates the gravity table. If you sync the output directory to your slave RPI, you can use the same script to 'inject', or rather overwrite the pi-hole db.

On inject, it copies the gravity.db from /etc/pihole/PyPhTP/ to /etc/pihole/ and then repopulates the gravity table.

I'm not 100% sure whether I need to stop the Pi-hole service to try to prevent locks to the DB but I am trying to avoid that.

Feel free to give it a try though. If nothing else, it may give you an idea of how you might adapt your script to achieve the same outcome.

https://github.com/mmotti/PyPhTP

Example (eject) with loads of random blocklists:

mmotti@ubuntu-server:~$ curl -sSl https://raw.githubusercontent.com/mmotti/PyPhTP/master/PyPhTP.py | sudo python3 - --eject
[i] Pi-hole directory located
[i] Write access is available to Pi-hole directory
[i] Pi-hole DB located
[i] Connection established to Pi-hole DB
[i] gravity.db size: 90.24 MB
[i] Emptying the gravity table
[i] Updating the gravity count in the info table
[i] Running Vacuum
[i] gravity.db size: 0.11 MB
[i] Restarting Pi-hole
[i] Closing connection to the Pi-hole DB
[i] Copying gravity.db to /etc/pihole/PyPhTP/gravity.db
[i] Correcting permissions
[i] Refreshing Gravity for source database

90.24MB down to 0.11MB just by removing the gravity entries.

I only have a single Pihole server at this point in time. I am using the custom.list file to store DNS records for systems on my internal network.

I went through the installation routine as documented:

1. Forked the repo
2. Marked as private
3. Cloned pihole-cloudsync to /usr/local/bin
4. Cloned my repo to /usr/local/bin (full path is now /usr/local/bin/my-pihole-lists)
5. Ran sudo pihole-cloudsync --initpush
6. Ran sudo pihole-cloudsync --push

When I go and look at the status of my repo, the following files were updated within the past few minutes:

• adlists.list
• black.list
• blacklist.txt
• regex.list
• whitelist.txt

The following are marked as updated "Last month" (i.e. not synced):

• custom.list
• gravity.db

Shared hosts is not enabled at this time, so I don't expect to see sharedhosts.txt updated at all.

Why would pihole-cloudsync not pick up changes in my custom.list and gravity.db?

Hello,

I'm trying this out for the first time. My secondary pihole is running from within docker.

When I run "pihole-cloudsync --initpull" I eventually run into the following error:

"Unable to copy data from /etc/pihole/gravity.db to /etc/pihole/gravity.db_temp"

Any pointers on where to start troubleshooting.

Thanks for looking!

In the step-by-step directions, under "On GitHub" step 3 calls for creating a fork, then step 4 (optional) is on marking the forked repo as private.

I just tried this and, in the Danger Zone section, there's a message that states "Public forks can't be made private. Please duplicate the repository."

I've never tried forking a repository before, so this is all new to me. I am going to try the process that GitHub outlines in terms of duplicating a repo but I thought I would bring this to your attention in the event you wanted to add details to your documentation.

Thanks so much for sharing this!!!

Whenever I try to run pihole-cloudsync --push, I get the following error:

fatal: Authentication failed for 'https://github.com/NGatti1997/my-pihole-lists.git/'
error: Could not fetch origin
Local Pi-hole lists are different than remote Git repo. Updating remote repo...

*** Please tell me who you are.

Run

  git config --global user.email "[email protected]"
  git config --global user.name "Your Name"

to set your account's default identity.
Omit --global to set the identity only in this repository.

fatal: unable to auto-detect email address (got 'root@raspberrypi.(none)')

Not sure where the problem lies. I'm sure the username and password are right, and I've run git config.

The ssh key info is there github but can we also update the setup help on front page here , ssh / or token base logins for future unattended pull/push of cloudsync lists thanks!

My gravity.db file is just over 200mb and, after spending the day setting up Git LFS on my devices, I’m now nearing my monthly data cap for the service.

Would it be possible to compare the gravity.db file currently in use with the one stored in the local repo and delete whatever is redundant, then push that shrunken file to the remote repo? Would the secondary piholes then be able to use that shrunken file to fill in the gaps of their own local gravity.db files?

The groups are not synced at all, and groups assignments does not work for Whitelist and Blacklist.

I guess that if there is no assigned group to a whitelist/blacklist rule, it won't work, so the rules won't be applied the same on the master than on the clones?

Here is an example:

Master

Clone

I'm using systemd to automate the push/pull and I can remove that part easily enough but was wondering what else I might miss. I've been using the script to sync 2 pihole instances in my home lab and have run into issues where the pull instance has lost it's gravity db. I'd like to remove pihole-cloudsync completely while I try to resolve this issue.

when doing step 6 at primairy pihole:
pi@raspberrypi:~ $ /usr/local/bin/pihole-cloudsync/pihole-cloudsync --initpush
-bash: /usr/local/bin/pihole-cloudsync/pihole-cloudsync: No such file or directory

What am I doing wrong?

both PIs running v5

push on primary works
pull on secondary is causing pihole (FTL) stopping to work
/usr/local/bin/pihole-cloudsync/pihole-cloudsync --initpull
Shared Hosts Mode DISABLED.
--initpull option detected. Initializing local Git repo for Pull/Download.
Error: no such table: info
[✗] DNS service is not running
[✓] Restarting DNS server

https://github.com/pi-hole/FTL/issues/743

Adding a comment to an adlist that contains a quote mark or apostrophe, i.e. "Person**'**s Blocklist", breaks the gravity update on the "pull" client after the database update.

After updating the database, the pull client attempts a gravity update and receives an error:
[i] Creating new gravity databases... [✗] Unable to copy data from /etc/pihole/gravity.db to /etc/pihole/gravity.db_temp Runtime error near line 24: FOREIGN KEY constraint failed (19) [✗] Unable to create gravity database. Please try again later. If the problem persists, please contact support.

Removing the quote marks from blocklist comments on the "push" client allows the "pull" client to update gravity correctly after the next pull.

Examining the db_dump.sql file shows that the insert command attempts to escape the single quote by doubling it. This appears not to be handled well on the other end.

I just set up a secondary pihole for my home network using Docker and I was looking to use this tool to sync the two together. To make this work with my docker container, I'm planning on running it on my docker host (Ubuntu 20.04) and changing the pihole_dir to match the volume for the pihole dir of the Docker container. This should work fine but the pihole -g command won't be able to run on the host, it has to be directed to the container instead - docker exec <container_name> pihole -g

Note: I considered running docker-compose restart or other docker equivalent command but I don't feel it is necessary, as pihole runs the gravity update on a cron anyway, so restarting the whole container feels like overkill in this use case.

I'm planning on simply replacing the two pihole -g commands in the bash script to said docker command instead, and it should work. Perhaps an option can be included similar to the shared hosts option where you if you specify a container name, it will run the docker exec command instead of simply pihole -g

The only other way I could think of to get this to work for a docker container would be to build your own docker image from the pihole image, but have the dockerbuild bring in Git and this tool. However, that's a bit more complicated - especially for the average user - so I figured adding an option as explained above would be a simple yet effective alternative.

Been trying to understand where Local CNAMES are stored in 5.2 but these scripts don't replicate them. More of a heads-up then a bug.

I entered username/email and password correctly but getting error message

fatal: Authentication failed for 'https://github.com/xxx/my-pihole-lists.git/'

Is this because I have 2FA activated on GitHub?
Any workaround?

thx
Mike

Hi @stevejenkins

Has anyone run into this error?

root@raspberrypi-hole1:/usr/local/bin/pihole-cloudsync# ./pihole-cloudsync --push
--push option detected. Running in Push/Upload mode.
Local Pi-hole lists are different than remote Git repo. Updating remote repo...
To github.com:REDACTED/my-pihole-lists.git
! [rejected] main -> main (non-fast-forward)
error: failed to push some refs to '[email protected]:REDACTED/my-pihole-lists.git'
hint: Updates were rejected because the tip of your current branch is behind
hint: its remote counterpart. Integrate the remote changes (e.g.
hint: 'git pull ...') before pushing again.
hint: See the 'Note about fast-forwards' in 'git push --help' for details.
Done!

I have raised a 'feature request' on the pihole discourse for this but just FYI the future group sync feature (which I have been looking at coding) will not fully work due to the use of the group keyword for the table name (only this table is affected).

The use of a keyword results in a syntax error when trying to export the table to CSV.

A workaround is to manually create the groups to match but I'm not sure if this will cause issues as internally the DB's will be different.

Coding wise the change is very easy with additional commands for the 3 group tables and 2 client tables but the client side would raise issues if the piholes are in different locations and therefore see different clients. This might require the client sync aspect to be optional via command line flag ??

Working to set this up to Sync 3x Pi-Hole's and was given the reply from GitHub when trying to setup my Repo that Password Auth is no longer allowed.

Suggestions?

Is it possible to add a script line for rebooting the pi when the pull command is finished?

I always get the API error when the pull request is finished and a reboot fixed it.

If not, just tell me where I should add the line and I will do it myself.

Thank you for taking the time to answer.

If I run /usr/local/bin/pihole-cloudsync/pihole-cloudsync --pull this works 100% without issue letting me know the stored credentials are working. When my systemd task runs this command I get a issue.

pihole-cloudsync[65509]: fatal: could not read Username for 'https://github.com': No such device or address

Jan 02 11:24:10 pihole2 pihole-cloudsync[65507]: error: Could not fetch origin

Jan 02 11:24:10 pihole2 pihole-cloudsync[65500]: Local Pi-hole lists match remote Git repo. No further action required.

Jan 02 11:24:10 pihole2 systemd[1]: pihole-cloudsync-update.service: Succeeded.

Even though it says succeeded it does not actually succeed in downloading updated data from the repo (because of failed authentication). I have tested and verified the changes did not pull down, and then ran the command manually and verified THAT pulled the change. I'm not sure why the command works manually but something in the automated command isn't recognizing the saved credentials. I went through the info in your guide and have verified the credential.helper is working so I'm thinking this is a 2FA specific issue with the personal access token I have setup to make this work.

For the README.me, Automating with systemd, systemd Details, .service its show this line
ExecStart=/usr/local/bin/pihole-cloudsync --pull

shouldn't it be?
ExecStart=/usr/local/bin/pihole-cloudsync/pihole-cloudsync --pull

When running sudo /usr/local/bin/pihole-cloudsync/pihole-cloudsync --pull I get the error

fatal: ambiguous argument 'HEAD..origin/master': unknown revision or path not in the working tree.
Use '--' to separate paths from revisions, like this:
'git <command> [<revision>...] -- [<file>...]'

The error seems harmless as my lists are successfully pulled from my repo and gravity is updated as expected. I'm fairly inexperienced with using Git, but I'm fairly sure I've configured everything according to the instructions outlines in the readme.

Instead of using /etc/hosts for DNS you can use this method and also have the list synced.

https://discourse.pi-hole.net/t/howto-using-pi-hole-as-lan-dns-server/533

Noticed you didn't provide the files for automating the push, I assume it is just changing the description and the command in the service file, but figured I would flag it.

The title says it all. Is this still a viable solution for multi-pi setups?

Wanted to make a suggestion to update the guide instructions for setup as a private repo.

I ran into this issue when making my forked repo private. I tracked down information that GitHub will NOT allow public forks to be private, which is what your guide advises to do. That option will be grayed out in the settings and unselectable if you do a fork.

You need to create a new repository using your "my-pihole-lists" as a template instead. When you "fork" it is as template it will give you the prompt upon creation to make it public or private and then set the repository name to the same thing of my-pihole-lists. Everything works as normal doing it that way and I could continue the guide. I had never done a private repo before so this was all new to me.

/usr/local/bin/my-pihole-lists$ sudo /usr/local/bin/pihole-cloudsync/pihole-cloudsync --push
--push option detected. Running in Push/Upload mode.
/usr/local/bin/pihole-cloudsync/pihole-cloudsync: line 99: sqlite3: command not found
Local Pi-hole lists are different than remote Git repo. Updating remote repo...
On branch master
Your branch is up to date with 'origin/master'.

Untracked files:
(use "git add ..." to include in what will be committed)
db_dump.sql

nothing added to commit but untracked files present (use "git add" to track)
Done!

pihole-cloudsync: line 69: sqlite3: command not found
./pihole-cloudsync: line 70: sqlite3: command not found

Would it be possible to add extra options? One for each of the items synchronized @ pull?
So a ‘--NOadlists’, a ‘--NOdomainlists’, a ‘--NOdnsnames’ and a ‘--NOcnames’.
Each of these options will skip the pull/import action.

It would help if you, for instance, do want to sync all but custom DNSnames or all but custom CNAMEs.

Hello.

I am able to confirm that my setup is working. The Master Pihole uploads to my github repo via Cronjob and my Secondary Pihole is pulling down those changes (new adlists at least) via Cronjob fine./

However the Domains blocked number on the Dashboard for the secondary Pihole never updates.

Pihole Master

Pihole Slave

My repo:
https://github.com/Wh1t3Rose/my-pihole-lists

Debug logs:
Master Pihole: https://pastebin.com/T8isvcJ9
Secondary: https://pastebin.com/PnHptpwZ