Issues with mounts in v2.1.0 about gooby HOT 85 CLOSED

I'm sure we can get around this, no worries... I'm pretty much sure I know where the problem lies, so all that needs to be done now is implement the correct solution!

from gooby.

Pfew!! Now if deedeefink and others can confirm it's working for them too, we may be able to finally close this nasty chapter 🗡

All seems to be working again. Thank you :-)

from gooby.

Guys, it's definitely something to do with whatever change happened between v18.04.0 and 18.04.1 - in fact I can reproduce the issue 100% now. That also explains why it ran flawlessly on our own systems for months before going live.

The positive news is that I am pretty sure I have a solution... which I'm going to implement in a test branch and will probably ask you to test before going live with it.

If you want the technical details:

Currently Gooby creates a single service that calls upon Rclone and MergerFS and boots them up one by one in sequence through the script. This worked like a charm in the past, but it's broken on the latest Ubuntu. It's not even intermittent any longer, it just crashes.

The solution is to go back to two separate scripts, which is the conventional way of running scripts anyway and seems to work fine.

I hope to have more news tomorrow and a place for you to test my theories :)

from gooby.

Lol, I have no idea, but frankly it's driving me nuts... I can make it work now on one machine (Hetzner) but not on my regular dedi... but eh I'm not going to give up that easily! I love using MergerFS because it prevents so many bans and other problems.

New day tomorrow with some fresh and innovative thoughts - hopefully 😆

from gooby.

Thanks guys, I'll try to tackle this with renewed zeal today. WE SHALL LICK THIS! Or my name is not Tech Perplexed - oh wait it's not 👩

from gooby.

Thanks 😄 Now, thinking out loud, the only difference between the uninstaller and the rclean script is that the uninstaller actually stops and then disables the service, whereas the rclean script only stops the service and restarts it. Now I will try @bdschuster's solution to wait 10 seconds in combination with stopping the service and restarting it... give me 10 minutes and see if that does something positive to the rclean.... again fingers crossed!

from gooby.

I accept full responsibility ;-)

from gooby.

I'll live (hopefully) lol

from gooby.

3 cleanups and 4 reboots performed and both /mnt/google & /mnt/clone come back each time.

from gooby.

Yeah I should have said....in case it's relevant to deedeefink.
The containers take a couple of minutes to load (which is fine) Plex and and Emby especially a minute or so longer still.
When I tested I think I got the 502 (or whichever) error. Leaving it a few minutes all was good.

Enjoy your day off Sophia and have a great holiday deedeefink.

from gooby.

lol...i will install and check everything out shorty and report back...get some rest.

from gooby.

Pfew!! Now if deedeefink and others can confirm it's working for them too, we may be able to finally close this nasty chapter 🗡

from gooby.

Yep. I just had real life cause to use a system clean up.
It definitely works for me here too. Nice work Sophia and Kelinger.

from gooby.

after several system cleanups, everything is still working.
case closed!

from gooby.

When I navigate via CLI, I can access /mnt/rclone and my files show up, but when I navigate to /mnt/google/ the folder is empty.

When I navigate via Plex to add a library, the folder /Media is empty

from gooby.

The exact same thing happened to me yesterday. A reboot seemed to fix things, will watch out if it happens again.

from gooby.

Hmmm a few of us have run it on Ubuntu 18 for several months. It's solid as a rock, I wouldn't release an unstable product (at least not willingly).

How often do you experience a drop of the mount? Is it consistent with certain actions?

If it's truly unstable for you you can always go back to the old Rclone only solution, but I'm hoping this can be sorted :)

from gooby.

I know, I've been using 18.04.0 since it came out and with Gooby 2.0 it worked like a charm. Yesterday I installed 18.04.1 and Gooby 2.1 to get the new mergerfs up and running. But not so lucky I guess.

It seems to drop after each System reset. Then I have to reboot 1 or more times to get the mount online. So yes, it's consistent. I'm not aware of where a log would be store, but if you point into the direction I'll be happy to share it.

from gooby.

From what you're telling me, it's just MergerFS that is giving you trouble right? As in: /mnt/rclone is still showing your files, but /mnt/google is empty?

When that happens, please type

sudo systemctl status mnt-google.mount

What does it report back to you?

(you can escape that with Ctrl-Z)

from gooby.

I just performed a System Clean and same problem. See the result below. I also noted that when I entered Midnight Commander to visually check the mount, it complains about the permission. Could this be linked?

● mnt-google.mount - MergerFS overlay writable storage
Loaded: loaded (/etc/systemd/system/mnt-google.mount; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Mon 2018-11-12 19:22:39 CET; 4min 4s ago
Where: /mnt/google
What: ${UPLOADS}:${RCLONEMOUNT}

Nov 12 19:22:39 Ubuntu-1804-bionic-64-minimal systemd[1]: Mounting MergerFS overlay writable storage...
Nov 12 19:22:39 Ubuntu-1804-bionic-64-minimal mount[28158]: fuse: missing mountpoint parameter
Nov 12 19:22:39 Ubuntu-1804-bionic-64-minimal systemd[1]: mnt-google.mount: Mount process exited, code=exited status=1
Nov 12 19:22:39 Ubuntu-1804-bionic-64-minimal systemd[1]: mnt-google.mount: Failed with result 'exit-code'.
Nov 12 19:22:39 Ubuntu-1804-bionic-64-minimal systemd[1]: Failed to mount MergerFS overlay writable storage.
Nov 12 19:24:43 Ubuntu-1804-bionic-64-minimal systemd[1]: /etc/systemd/system/mnt-google.mount:10: Not an absolute path, ignoring: ${MOUNTTO}
Nov 12 19:25:19 Ubuntu-1804-bionic-64-minimal systemd[1]: /etc/systemd/system/mnt-google.mount:10: Not an absolute path, ignoring: ${MOUNTTO}
Nov 12 19:25:19 Ubuntu-1804-bionic-64-minimal systemd[1]: /etc/systemd/system/mnt-google.mount:10: Not an absolute path, ignoring: ${MOUNTTO}
Nov 12 19:25:19 Ubuntu-1804-bionic-64-minimal systemd[1]: /etc/systemd/system/mnt-google.mount:10: Not an absolute path, ignoring: ${MOUNTTO}

from gooby.

Here is what MC is reporting. Again. It only happens after System Clean:

plexuser@Ubuntu-1804-bionic-64-minimal:~$ sudo mc
Directory /tmp/mc-root is not owned by you
Temporary files will be created in /tmp
Press any key to continue...

from gooby.

Hmmm interesting... I researched the message and it dates back to 2012!

You said it only happens after a system clean - but not when you reboot right? Or how do you get the mount to come back at all?

Can you try this please:

sudo systemctl stop gooby.service gooby-rclone.service gooby-find.service mnt-google.mount
sudo systemctl disable gooby.service gooby-rclone.service gooby-find.service mnt-google.mount

Check if both /mnt/rclone and /mnt/google are empty before you proceed... (they should be) then

sudo chown -R $USER:$USER /tmp /mnt
sudo systemctl enable gooby.service gooby-rclone.service gooby-find.service mnt-google.mount
sudo systemctl daemon-reload
sudo systemctl start gooby.service

First check if your mounts are there (both /mnt/rclone and /mnt/google, there should also be a /mnt/uploads/Downloads present)

Then run another system clean - did the problem come back?

from gooby.

Yes, only after system clean

1. Stopped service = mounts disappear
2. Ran commands = mounts are back, also download folder
3. Ran System Clean= All is working :-)

However, I still get the error with MC as mentioned above after step 2

from gooby.

Ok if everything is working properly I wouldn't worry about that particular error.... the real question is of course: will it keep on working?! Please let me know 😸

from gooby.

Wait... I were too trigger happy.
After the system clean the mounts are back, but:

• Plex URL not working. I get Chrome privacy error
• Sonarr 502 Bad Gateway
• Radarr, Jackett, Netdata, Tautulli and OMBI are working just fine

A reboot fixed this though.

Seems like I should be careful about System clean :-)

from gooby.

I'll close now and cross my fingers :-)

from gooby.

LOL well a system clean doesn't do too many exciting things... basically it

• Takes all containers and gooby.service down
• Updates Rclone if there is a newer version
• Updates Gooby to latest version
• Brings gooby.service up
• Brings all containers back up, updating the ones that have an update
• Runs Linux patches for updated packages
• Makes sure all permissions are set properly

It really doesn't need to be run on a VERY regular basis, just when you experience some issues really.

That said, it SHOULD work properly and not CAUSE any issues 😆

from gooby.

Hmmm, something is not right.
I still have issues which a reboot cannot fix. Now Rclone service is also down, also after reboot.

I'll reset the server and reinstall ubuntu / gooby and let you know if the issues persist afterwards

from gooby.

Oof, sorry you're having such a hard time with it!

from gooby.

@deedeefink I just did a system clean and I ran into the same issue as yourself! I made a change in the script, please update Gooby and see if that sorts it for you - once again apologies it's causing you grief!

from gooby.

Mind is suddenly behaving weird too... It's not you. I'll look into this tomorrow!

Edit: I think I know what it is, it's not quitting the mount properly so after a reboot it won't have the proper permissions... I promise this will be sorted :)

from gooby.

Pheww. I was going crazy...
Did a fresh install of ubuntu and Gooby and so far it seems to work as intended. Hopefully that will be the end of this issue :-)

from gooby.

I guess we're not that lucky :-(

1. Did a fresh ubuntu and gooby install.
2. All worked great until I did a System Clean. After that the Rclone mount went offline and only after a reboot I got it working again.
3. I then remove an app/container and the Google mount stopped working. Rclone worked
4. And now even after a reboot I have issue with the Rclone mount (Several reboots fixed it though)

Could this still be a permissions problem or what kind of witchcraft is going on?

Here are the error codes I get with the services AFTER reboot:

gooby-find.service - Gooby Pre-Cacher
Loaded: loaded (/etc/systemd/system/gooby-find.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Tue 2018-11-13 14:36:20 CET; 7min ago
Process: 1823 ExecStart=/bin/sh -c /bin/sleep 10 ; /usr/bin/find ${RCLONEMOUNT} > ${LOGS}/precache (code=exited, status=1/FAILURE)
Main PID: 1823 (code=exited, status=1/FAILURE)

Nov 13 14:36:20 Ubuntu-1804-bionic-64-minimal sh[1823]: /usr/bin/find: ‘./proc/5004/task/5004/fd’: Permission denied
Nov 13 14:36:20 Ubuntu-1804-bionic-64-minimal sh[1823]: /usr/bin/find: ‘./proc/5004/task/5004/fdinfo’: Permission denied
Nov 13 14:36:20 Ubuntu-1804-bionic-64-minimal sh[1823]: /usr/bin/find: ‘./proc/5004/task/5004/ns’: Permission denied
Nov 13 14:36:20 Ubuntu-1804-bionic-64-minimal sh[1823]: /usr/bin/find: ‘./proc/5004/fd’: Permission denied
Nov 13 14:36:20 Ubuntu-1804-bionic-64-minimal sh[1823]: /usr/bin/find: ‘./proc/5004/map_files’: Permission denied
Nov 13 14:36:20 Ubuntu-1804-bionic-64-minimal sh[1823]: /usr/bin/find: ‘./proc/5004/fdinfo’: Permission denied
Nov 13 14:36:20 Ubuntu-1804-bionic-64-minimal sh[1823]: /usr/bin/find: ‘./proc/5004/ns’: Permission denied
Nov 13 14:36:20 Ubuntu-1804-bionic-64-minimal systemd[1]: gooby-find.service: Main process exited, code=exited, status=1/FAILURE
Nov 13 14:36:20 Ubuntu-1804-bionic-64-minimal systemd[1]: gooby-find.service: Failed with result 'exit-code'.
Nov 13 14:36:20 Ubuntu-1804-bionic-64-minimal systemd[1]: Failed to start Gooby Pre-Cacher.

gooby-rclone.service - Mount Google Drive (rclone)
Loaded: loaded (/etc/systemd/system/gooby-rclone.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Tue 2018-11-13 14:34:50 CET; 7min ago
Process: 1193 ExecStart=/usr/bin/rclone mount --allow-other --buffer-size 1G --dir-cache-time 72h --drive-chunk-size 32M --fast-list --gid ${USERID} --log-level INFO --log-file ${
Process: 1867 ExecStartPre=/bin/mkdir -p ${RCLONEMOUNT} (code=exited, status=1/FAILURE)
Main PID: 1193 (code=killed, signal=TERM)

Nov 13 14:34:50 Ubuntu-1804-bionic-64-minimal systemd[1]: Starting Mount Google Drive (rclone)...
Nov 13 14:34:50 Ubuntu-1804-bionic-64-minimal systemd[1]: gooby-rclone.service: Control process exited, code=exited status=1
Nov 13 14:34:50 Ubuntu-1804-bionic-64-minimal systemd[1]: gooby-rclone.service: Failed with result 'exit-code'.
Nov 13 14:34:50 Ubuntu-1804-bionic-64-minimal systemd[1]: Failed to start Mount Google Drive (rclone).
Nov 13 14:34:50 Ubuntu-1804-bionic-64-minimal systemd[1]: gooby-rclone.service: Service hold-off time over, scheduling restart.
Nov 13 14:34:50 Ubuntu-1804-bionic-64-minimal systemd[1]: gooby-rclone.service: Scheduled restart job, restart counter is at 5.
Nov 13 14:34:50 Ubuntu-1804-bionic-64-minimal systemd[1]: Stopped Mount Google Drive (rclone).
Nov 13 14:34:50 Ubuntu-1804-bionic-64-minimal systemd[1]: gooby-rclone.service: Start request repeated too quickly.
Nov 13 14:34:50 Ubuntu-1804-bionic-64-minimal systemd[1]: gooby-rclone.service: Failed with result 'exit-code'.
Nov 13 14:34:50 Ubuntu-1804-bionic-64-minimal systemd[1]: Failed to start Mount Google Drive (rclone).

mnt-google.mount - MergerFS overlay writable storage
Loaded: loaded (/etc/systemd/system/mnt-google.mount; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Tue 2018-11-13 14:24:14 CET; 4min 38s ago
Where: /mnt/google
What: ${UPLOADS}:${RCLONEMOUNT}
Process: 5105 ExecUnmount=/bin/umount /mnt/google -c (code=exited, status=0/SUCCESS)
Process: 5903 ExecMount=/bin/mount ${UPLOADS}:${RCLONEMOUNT} /mnt/google -t fuse.mergerfs -o use_ino,hard_remove,auto_cache,sync_read,allow_other,category.action=all,category.crea
Tasks: 0 (limit: 4915)
CGroup: /system.slice/mnt-google.mount

Nov 13 14:24:14 Ubuntu-1804-bionic-64-minimal systemd[1]: Mounting MergerFS overlay writable storage...
Nov 13 14:24:14 Ubuntu-1804-bionic-64-minimal mount[5903]: fuse: missing mountpoint parameter
Nov 13 14:24:14 Ubuntu-1804-bionic-64-minimal systemd[1]: mnt-google.mount: Mount process exited, code=exited status=1
Nov 13 14:24:14 Ubuntu-1804-bionic-64-minimal systemd[1]: mnt-google.mount: Failed with result 'exit-code'.
Nov 13 14:24:14 Ubuntu-1804-bionic-64-minimal systemd[1]: Failed to mount MergerFS overlay writable storage.

from gooby.

This is really making me think that 18.04.1 has changed some things. I kept having issues like this, I reverted back to 18.04.0 and no issues anymore. It also seems like they changed the repository as well since you can no longer access some common tools like fail2ban from the repository any longer in 18.04.1.

from gooby.

Could very well be. In this case @TechPerplexed should experience the same issues as I believe we're both using Hetzner who only offers 16.04.5 and 18.04.1 as default Ubuntu images.

But I'll consider to downgrade if we cannot get around this

from gooby.

Woohoo!! I knew something happened! Lol...what about missing packages in 18.04.1?

from gooby.

Not only that, it really helps out with unresponsive sonarr/radarr when it's moving to gdrive. I haven't had any unresponsiveness since you moved this to mergerfs

from gooby.

Well I just had a bit of a brainwave.... can you try updating Gooby to the latest version and see if that works better? If not, I WILL use tomorrow for fresh, creative and brilliant thoughts LOL

from gooby.

Just update? Or do a system cleanup after as well?

from gooby.

Update, then see if a system cleanup or reboot kills the mount (if it does, I'm gonna kill Ubuntu!)

from gooby.

BAHHH HUMBUG!!!!!!!!!!!!!!!!!!!!!! STILL no good!!! Weirdly enough a manual

sudo systemctl start gooby

Will bring the mount back instantly (at least for me), but it's supposed to come back in the SCRIPT! GRR, ah well, tomorrow!

from gooby.

Ok, so I'm not the only one having that issue... Everytime I do a system cleanup, my mount to /mnt/google is empty, but /mnt/rclone is there. But I just run 'sudo service gooby start' and it's all good, but then I have to restart all my docker containers to remount in the container.

from gooby.

@deedeefink Do you still have issues with the Rclone mount /mnt/rclone, or just MergerFS /mnt/google?

@bdschuster Yes, that is exactly what is happening to me too now. Yet it worked fine for me until yesterday - then suddenly all hell broke loose. The mount is stable once it's up, but a reboot or system clean WILL break it 100% now. Crazy!

from gooby.

I know this sounds crazy...but try adding a pause...maybe 10 seconds to start with, right above the the start gooby command in the script.

from gooby.

@deedeefink Do you still have issues with the Rclone mount /mnt/rclone, or just MergerFS /mnt/google?

So,

1. I updated Gooby and verified that I got your last commit
2. Performed a system Clean = Both Rclone and MergerFS went down
3. I rebooted twice = MergerFS is mounted, but no content as Rclone is down
4. Ran sudo systemctl start gooby but no effect = MergerFS is mounted, but no content as Rclone is down
5. Ran sudo systemctl start gooby-rclone.service which fails as per below:

Status are:
gooby-rclone.service - Mount Google Drive (rclone)

   Loaded: loaded (/etc/systemd/system/gooby-rclone.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Wed 2018-11-14 13:14:06 CET; 1min 3s ago
  Process: 17519 ExecStartPre=/bin/mkdir -p ${RCLONEMOUNT} (code=exited, status=1/FAILURE)

mnt-google.mount - MergerFS overlay writable storage

   Loaded: loaded (/etc/systemd/system/mnt-google.mount; enabled; vendor preset: enabled)
   Active: active (mounted) since Wed 2018-11-14 13:11:11 CET; 5min ago
    Where: /mnt/google
     What: /mnt/uploads
  Process: 6398 ExecUnmount=/bin/umount /mnt/google -c (code=exited, status=0/SUCCESS)
  Process: 6479 ExecMount=/bin/mount ${UPLOADS}:${RCLONEMOUNT} /mnt/google -t fuse.mergerfs -o use_ino,hard_remove,auto_cache,sync_read,allow_other,category.action=all,category.crea
    Tasks: 4 (limit: 4915)
   CGroup: /system.slice/mnt-google.mount
           └─6493 mergerfs /mnt/uploads:/mnt/rclone /mnt/google -o rw,use_ino,hard_remove,auto_cache,sync_read,allow_other,category.action=all,category.create=ff,dev,suid

Command sudo journalctl -xe

Nov 14 13:29:29 Ubuntu-1804-bionic-64-minimal systemd[1]: gooby-rclone.service: Start request repeated too quickly.
Nov 14 13:29:29 Ubuntu-1804-bionic-64-minimal systemd[1]: gooby-rclone.service: Failed with result 'exit-code'.
Nov 14 13:29:29 Ubuntu-1804-bionic-64-minimal systemd[1]: Failed to start Mount Google Drive (rclone).
-- Subject: Unit gooby-rclone.service has failed
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
-- 
-- Unit gooby-rclone.service has failed.
-- 
-- The result is RESULT.

from gooby.

@deedeefink one more question for you: when you do a fresh install of Rclone, do both mounts ALWAYS come up properly? In other words, do your woes only happen after a reboot/system clean?

from gooby.

@deedeefink one more question for you: when you do a fresh install of Rclone, do both mounts ALWAYS come up properly? In other words, do your woes only happen after a reboot/system clean?

Yes that is correct. But, I only removed Rclone by reinstalling Ubuntu and then install Rclone on a fresh box. I did not try sudo apt-get remove rclone and the reinstall Rclone. Do you want me to try this?

from gooby.

Would you please try to uninstall it from the menu and then reinstall it (again from the menu)? No need to delete your config... just uninstall and reinstall. I'm trying to figure out if a fresh installation is behaving different from the system clean and then take it from there :)

from gooby.

Yes that works. mounts are back up again :-)

from gooby.

SUPER, ok, and I made a few changes to the uninstall script, would you please update Gooby and see if it's still behaving well? (And thank you for testing this with me)

from gooby.

Yes, still works
(and no worries. Happy to help)

from gooby.

Ok, the update is live.... go for it 🙈

from gooby.

Sorry :-(

Rclone is running
MergerFS is not

 mnt-google.mount - MergerFS overlay writable storage
   Loaded: loaded (/etc/systemd/system/mnt-google.mount; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Wed 2018-11-14 19:24:34 CET; 1min 15s ago
    Where: /mnt/google
     What: ${UPLOADS}:${RCLONEMOUNT}
  Process: 29000 ExecMount=/bin/mount ${UPLOADS}:${RCLONEMOUNT} /mnt/google -t fuse.mergerfs -o use_ino,hard_remove,auto_cache,sync_read,allow_other,category.action=all,category.create=ff (code=exited, status=1/FAILURE)

Nov 14 19:24:34 Ubuntu-1804-bionic-64-minimal systemd[1]: Mounting MergerFS overlay writable storage...
Nov 14 19:24:34 Ubuntu-1804-bionic-64-minimal mount[29000]: fuse: missing mountpoint parameter
Nov 14 19:24:34 Ubuntu-1804-bionic-64-minimal systemd[1]: mnt-google.mount: Mount process exited, code=exited status=1
Nov 14 19:24:34 Ubuntu-1804-bionic-64-minimal systemd[1]: mnt-google.mount: Failed with result 'exit-code'.
Nov 14 19:24:34 Ubuntu-1804-bionic-64-minimal systemd[1]: Failed to mount MergerFS overlay writable storage.

from gooby.

Tested system clean twice with the same result. . Will try a reboot

from gooby.

WHAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

from gooby.

Is it working on your server?
Can there be something else that is interfering on my side?

from gooby.

Yeah it just behaved weird on mine too, it's definitely not you. It behaved well for 2 reboots, then suddenly my mount dropped too. This is unbelievable... I mean if uninstalling/reinstalling works, then there isn't a reason in the world why it wouldn't behave the same in an almost identical script.

Ok, back to the drawing board..... ackpff.

from gooby.

As mentioned a couple of time I like to use Midnight Commander and it's my first go-to tool to see if my mount is available. After each System Clean I get an error starting MC:

Directory /tmp/mc-root is not owned by you
Temporary files will be created in /tmp
Press any key to continue...

In the system clean script you're resetting permissions. As I remember this is a relatively new addition to the script. Could it be that this resetting permissions are impacting the mounts?

I'm thinking this as I typically use sudo mc to launch Midnight Commander. When I use just mc I do not get the permission error. I mean, it's weird that I get the permissions error when I used sudo

from gooby.

Thank you, I must look into that deeper then. Now here is the REALLY weird issue:

The two scripts are basically identical now. In fact I went one further and added the command to uninstall and reinstall Rclone (on the test script, not live) - just so it matches the two scripts 100%.

Yet it's STILL not coming back properly for me.

The resetting permissions were added because I wanted to match the working environment to a T (but if you see something I don't, all suggestions welcome!!!)

All this started when you started this thread - it's all your fault (TEASING) 😋

Seriously though I do feel we're very close... if one script works, with another one it MUST be possible to make it toe the line too right?

from gooby.

If he didn't open, I was going too 😃😃 ... And sorry I haven't been around, been dealing with medical issues.

from gooby.

@bdschuster welcome back. Hope things are okay?

from gooby.

I'll soon be signing out for tonight and leave you too it. Will test in the morning.

from gooby.

@deedeefink sleep well, and @bdschuster take care, hope you're all better soon <3

from gooby.

Ok, here is a test environment for you guys with a new service (Rclone and MergerFS each separately). To install:

sudo rm -r /opt/Gooby
sudo git clone -b rcleantest https://github.com/TechPerplexed/Gooby /opt/Gooby
sudo chmod +x -R /opt/Gooby/install
sudo chmod +x -R /opt/Gooby/menus
sudo chmod +x -R /opt/Gooby/scripts/cron
sudo rsync -a /opt/Gooby/install/gooby /bin
sudo chmod 755 /bin/gooby

Then uninstall Rclone from the menu and reinstall (without deleting the config).

Let's see how that goes! Don't update Gooby from the menu as that will take you back to the stable (cough, make that no-longer-stable) branch.

I rebooted and ran a system clean quite a few times and it seems alright to me... very curious how it goes for you!

from gooby.

Oh one more note: I actually tested the reboot while the Crontab that runs on reboot was disabled and the mount came back each and every time (it didn't with the other service scripts). I haven't tested it WITH cron yet... but will do so tomorrow. In the meantime let me know your findings.

I'm a little encouraged now that it does come up after a reboot by itself. So if the script breaks that, at least now we FINALLY have a basis to work with!

from gooby.

After 2x system clean and 2x reboot both Rclone and MergerFS mounts are coming back online... Wuhuuuu.... but (sorry).....theres more......

After my first system clean non of the containers worked and I could not access any apps - But mounts were up though :-). I rebooted and checked mounts. All okay. But still Docker were not working. I performed a new system clean but it never completed as it got stuck at "Bringing services back online". It gave me time to notice the following error:

Taking containers down
Removing network docker_default
WARNING: Network docker_default not found.

I wanted to double check that I could reproduce the above. When I tired, I encountered no problems with docker_network, but the mounts were empty...arrrggghhhh....I tried to system clean and switching back and forth between the "stable" and test version for the N number of times and:

• Sometime it works
• Sometimes I get stuck at "Bringing services back online"
• Sometime docker_default is not working
• Sometimes mounts are failing
• Sometimes I smile and sometimes I cry

@nesbcn, @bdschuster and @HitsvilleUK can you please check. I'm loosing track of all the things I tried and need a second opinion

from gooby.

Interesting @deedeefink
After all my system cleans all the containers loaded fine. I just double checked again now and they're all up including my custom containers,
I suddenly started getting issues a day or so ago scanning my TV shows into Emby and even that's working now, Sorry I cant be of more help.

from gooby.

I tried a system clean again. It completed and the mounts are working, but the docker_network is not responding and I cannot access my apps. Also after a reboot.

I then make another system clean and get this error again:

Taking containers down
Removing network docker_default
WARNING: Network docker_default not found.

And "Bringing services back online" gets stuck

@TechPerplexed / @ALL I'll fly off to the beach now , so I might not be online for testing that much the coming days.

from gooby.

@deedeefink first of all, have a wonderful vacation (or holiday, depending on where you live 🤣 )

I'm glad the mount problem seems to be resolved... tentatively of course.

As for the message "docker_default not found", this is normal and should not have any other effect (it's just to make sure everything is cleaned out properly). It is also normal for the containers not to always show the "Ok" message (that always makes me nervous too).

HOWEVER your system should come back of course... unfortunately I can't reproduce it myself. For the record there have been absolutely no changes in the network settings or the rest of the script - I have only been "messing" with the mount settings.

I suggest that if the mount is fine (in the new test setting), but if you continue to have problems with the docker_network not responding, that you create a new issue so we can start troubleshooting that separately...

If you guys all confirm that the new services setting solves the mount problem I'll push that to stable - and then we can start afresh with new problems (YIKES I hope not...)

Today I am going to take a day off too, because three days of non stop troubleshooting sort of made me neglect all my regular activities, heh. Tomorrow I'll be back with renewed vim and vigor!

from gooby.

Ok my friendlies, this is what I'm going to do... since it seems that is new script has solved the mount problems, I'll push it to the "master" repository tomorrow. If you wish you can update Gooby from the menu (option E - A) and go with that (tomorrow that is, not just yet).

However, we're not done yet with the test script.... with the help of my good friend kelinger we have made more improvements to the mount check, basically that it won't start any of the containers until it verifies that both the Rclone AND the MergerFS mount are in place and running.

After I push this test repository to the live environment I'll update the test place with the improvements he has made - which will then be pushed soon afterwards hopefully after your approval :)

But all that is for tomorrow, today was strictly a script free day for me 😴

from gooby.

Done! The version you guys tested is now live - and I have updated the test environment with the built in mount checks (to make sure they are really up before proceeding). Hopefully this can go live by the weekend 👍

from gooby.

Hi from the beach 👋
I updated to the new test version and when I run a system clean, I get this looping error:

EDIT: This also applies to the Master version (I'll do a clean ubuntu install).

Job for rclonefs.service failed because the control process exited with error code.
See "systemctl status rclonefs.service" and "journalctl -xe" for details.
Job for mergerfs.service failed because the control process exited with error code.
See "systemctl status mergerfs.service" and "journalctl -xe" for details.
mountpoint: /mnt/rclone: No such file or directory
mountpoint: /mnt/google: No such file or directory
Waiting on mounts to be created...
mountpoint: /mnt/rclone: No such file or directory
mountpoint: /mnt/google: No such file or directory
Waiting on mounts to be created...
mountpoint: /mnt/rclone: No such file or directory
mountpoint: /mnt/google: No such file or directory
Waiting on mounts to be created...
mountpoint: /mnt/rclone: No such file or directory
mountpoint: /mnt/google: No such file or directory
Waiting on mounts to be created...

I wouldn't necessarily trust my setup as I previously experienced issues you guys didn't. I will soon make a fresh install and test it again. But would be good to know if it's just me or a general problem though.

from gooby.

So did a clean install of ubuntu and installed master version of Gooby and I still got this error when I ran a system clean

Cleaning mount leftovers

Job for rclonefs.service failed because the control process exited with error code.
See "systemctl status rclonefs.service" and "journalctl -xe" for details.
Job for mergerfs.service failed because the control process exited with error code.
See "systemctl status mergerfs.service" and "journalctl -xe" for details.
mountpoint: /mnt/rclone: No such file or directory
mountpoint: /mnt/google: No such file or directory
Waiting on mounts to be created...
mountpoint: /mnt/rclone: No such file or directory
mountpoint: /mnt/google: No such file or directory
Waiting on mounts to be created...
mountpoint: /mnt/rclone: No such file or directory
mountpoint: /mnt/google: No such file or directory
Waiting on mounts to be created...

rclonefs.service - Mount Google Drive (rclone)
   Loaded: loaded (/etc/systemd/system/rclonefs.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Sat 2018-11-17 16:21:12 CET; 3min 7s ago
  Process: 31838 ExecStartPre=/bin/mkdir -p ${RCLONEMOUNT} (code=exited, status=1/FAILURE)
 Main PID: 29298 (code=exited, status=0/SUCCESS)

mergerfs.service - MergerFS overlay writable storage
   Loaded: loaded (/etc/systemd/system/mergerfs.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Sat 2018-11-17 16:21:12 CET; 2min 23s ago
  Process: 31839 ExecStartPre=/bin/mkdir -p ${UPLOADS} ${MOUNTTO} (code=exited, status=1/FAILURE)
 Main PID: 29455 (code=exited, status=0/SUCCESS)

from gooby.

Well at least with the new built in checks it will show you that there is a problem somewhere... rather than ignoring it altogether and proceeding without the mount. Which I realize is only a very small comfort...

That said, I have no idea why yours would still be so stubborn! In fact we now reverted almost all the way back to the old tried and proven GooPlex method of first mounting Rclone (absolutely no changes from how it ran before, except it's properly mounted as your user instead of root), then MergerFS (which should not be able to cause Rclone to crash).

As for the messages "mountpoint: /mnt/rclone: No such file or directory" etc - that is what is supposed to happen (though obviously not in a never ending loop). The way the script runs now, first it shuts down the containers, then the mergerfs service, then rclone. Those seem to work fine for you (it loops until done).
Then it makes sure the services are up and loops until that is done too - that is where yours seems to halt.

Ok, some troubleshooting actions:

Can you please verify that the line user_allow_other is uncommented in the file /etc/fuse.conf?
(sudo nano /etc/fuse.conf)

Also please verify the permissions are correct (they should be... but...)
sudo chown -R $USER:$USER /mnt

When the script is stuck in its loop, is it possible for you to open another session and verify that when you type nano /var/local/Gooby/Docker/.env all the fields are present like so:

RCLONESERVICE=Gdrive
RCLONEFOLDER=Media
RCLONEMOUNT=/mnt/rclone
MOUNTTO=/mnt/google
UPLOADS=/mnt/uploads
GOOGLE=/mnt/google
MEDIA=/mnt/google
MOVIES=/mnt/google/Movies
TV=/mnt/google/TV
DOWNLOADS=/home/you/Downloads

Something else for you to try: rebooting with and without the cron enabled. To disable the cron, type crontab -e and find the line
@reboot /opt/Gooby/scripts/cron/rclean.sh > /dev/null 2>&1
Then just add a # in front of it:
#@reboot /opt/Gooby/scripts/cron/rclean.sh > /dev/null 2>&1

When you uncomment the line some unexpected things may happen with the rest of your containers (because the script makes sure things happen in the correct order), but I'd like to focus on getting your mount stable first.... so when you do a "vanilla" reboot and or "cron assisted" reboot, do you have your mounts?

Oh and have a wonderful time at the beach 🌞 , I'm jealousssssssssssss (looking at the first snow here) ❄️

from gooby.

My system is also stuck at this:

mountpoint: /mnt/rclone: No such file or directory
Waiting on mounts to be created...
mountpoint: /mnt/rclone: No such file or directory
Waiting on mounts to be created...

this is after updating gooby and system clean. Any suggestion?

from gooby.

How long does it stay stuck in that loop? Forever or a certain amount of time?

Also lots of suggestions in the post above you :)

Edit: can you update Gooby and try again please?

from gooby.

I tried everything in the post, still looping forever:

The latest version of rclone rclone v1.44 is already installed.


Cleaning mount leftovers

Job for rclonefs.service failed because the control process exited with error code.
See "systemctl status rclonefs.service" and "journalctl -xe" for details.
Job for mergerfs.service failed because the control process exited with error code.
See "systemctl status mergerfs.service" and "journalctl -xe" for details.
mountpoint: /mnt/rclone: No such file or directory
mountpoint: /mnt/google: No such file or directory
Waiting on mounts to be created...
mountpoint: /mnt/rclone: No such file or directory
mountpoint: /mnt/google: No such file or directory
Waiting on mounts to be created...
mountpoint: /mnt/rclone: No such file or directory
mountpoint: /mnt/google: No such file or directory
Waiting on mounts to be created...
mountpoint: /mnt/rclone: No such file or directory
mountpoint: /mnt/google: No such file or directory
Waiting on mounts to be created...
mountpoint: /mnt/rclone: No such file or directory
mountpoint: /mnt/google: No such file or directory
Waiting on mounts to be created...
mountpoint: /mnt/rclone: No such file or directory
mountpoint: /mnt/google: No such file or directory
Waiting on mounts to be created...
mountpoint: /mnt/rclone: No such file or directory
mountpoint: /mnt/google: No such file or directory
Waiting on mounts to be created...
mountpoint: /mnt/rclone: No such file or directory
mountpoint: /mnt/google: No such file or directory
Waiting on mounts to be created...
mountpoint: /mnt/rclone: No such file or directory
mountpoint: /mnt/google: No such file or directory
Waiting on mounts to be created...

from gooby.

Can you try once more to set the permissions? Problem is I can't reproduce this any longer... testing on two systems and it just seems to work fine for me.

So try this:

sudo systemctl stop mergerfs
sudo systemctl stop rclonefs
sudo systemctl daemon-reload
sudo chown -R $USER:$USER /mnt

Then make sure Gooby is updated and try the cleanup again from the menu...

from gooby.

Well I didn't try the permissions, I updated Gooby again, rebooted then reinstalled rclone (Beta version this time), then proceeded to system clean. It worked!

from gooby.

Also, would it be possible to launch rclone mount with --rc option? I like to review the stats from time to time, to check everything is going smoothly...

from gooby.

I tested what you suggested in the above comment #36 (comment)

• user_allow_other is uncommented
• I chown /mnt folder

Here is my env.:

RCLONESERVICE=Gdrive
RCLONEFOLDER=
RCLONEMOUNT=/mnt/rclone
MOUNTTO=/mnt/google
UPLOADS=/mnt/uploads
GOOGLE=/mnt/google
MEDIA=/mnt/google
MOVIES=/mnt/google/Movies
TV=/mnt/google/TV
DOWNLOADS=/home/plexuser/Downloads

Note that I define the root folder in my rclone settings. Don't think it's relevant, but my TV folder is actual TVshows

When i reboot with / without disabled crontab mounts are not working. When I test it, the result is:

$ sudo systemctl status gooby-rclone.service  
Unit gooby-rclone.service could not be found.

$ sudo systemctl status mnt-google.mount
Unit mnt-google.mount could not be found.

from gooby.

Well that was too quick... Mounts seem to be working, now Plex somehow doesn't find the files when transcoding ... I'm looking into it atm

from gooby.

sudo systemctl stop mergerfs
sudo systemctl stop rclonefs
sudo systemctl daemon-reload

Followed this, updated Gooby, reinstalled rclone and ran system clean and it worked :-)

I did not add my media just yet, so can't verify @nesbcn Plex problem

from gooby.

Also, would it be possible to launch rclone mount with --rc option? I like to review the stats from time to time, to check everything is going smoothly...

Absolutely! You can edit your rclonefs.service to your liking - it won't be overwritten unless you uninstall/reinstall :)
sudo nano /etc/systemd/system/rclonefs.service then run a daemon reload sudo systemctl daemon-reload

Well that was too quick... Mounts seem to be working, now Plex somehow doesn't find the files when transcoding ... I'm looking into it atm

Did you restart the containers though? After all we stopped the services without stopping the containers first... just to make sure the permissions were correct. Do you still have the problem after a "proper" system clean?

Followed this, updated Gooby, reinstalled rclone and ran system clean and it worked :-)

Oh I hope so.... fingers crossed... toes crossed... eyes crossed........

from gooby.

I'm not having any issues so far with the new version. :-)

from gooby.