Improve backup/update process about mycodo HOT 32 CLOSED

Testing by @zsole2:

First, I checked diff/patch. It was used on the og formatting I was mentioned earlier, it looks I got all pieces resolved after a few bumps. The main thing was to find a proper replacement for the explode function. I attach here the diff file for you to see what it was changed, and the resulting relay log file. I only could check the relay and HT-sensor logs for correctness. The diff was made from the home folder, based on a backup:

diff -Naur --exclude="log" --exclude="config" --exclude="*.pyc" --exclude="img" Mycodo-backups/Mycodo-2015-09-24_13-33-22/3.5 Mycodo/3.5 > test.patch

and checked with

patch -p0 --dry-run < test.patch

end patched with

patch -p0 < test.patch

This diff/patch should work on the whole Mycodo directory, but custom changes would be difficult to follow.
After this, I googled the following page:

http://stackoverflow.com/questions/10414769/git-pull-keeping-local-changes

This looked easy to follow, so I tried it. In fact, after a first try, the real run took about 20 minutes...
Here is the sequence of commands I used from the terminal (somehow my .bash_history file was not updated, but anyway, the whole terminal output is attached):

root@chz:/home/pi/Mycodo# git log --oneline
root@chz:/home/pi/Mycodo# git reset --hard 4024e62
root@chz:/home/pi/Mycodo# /etc/init.d/mycodo restart
root@chz:/home/pi/Mycodo# cd ..
root@chz:/home/pi# patch -p0 --dry-run < test.patch
root@chz:/home/pi# patch -p0 < test.patch
root@chz:/home/pi# /etc/init.d/mycodo restart
root@chz:/home/pi# cd Mycodo
root@chz:/home/pi/Mycodo# git status
root@chz:/home/pi/Mycodo# git stash
root@chz:/home/pi/Mycodo# git pull
root@chz:/home/pi/Mycodo# /etc/init.d/mycodo restart
root@chz:/home/pi/Mycodo# git stash pop
root@chz:/home/pi/Mycodo# /etc/init.d/mycodo restart

In short:
I started from my patched version of log files
then first reverted 2 commits back
then applied the patch
then stashed the modified files
then pulled the newest version from github (coincidentally you just pushed an update in this small timeframe, I am worried about the database update, but everything still looks working, I'll do a proper update soon).
then reapplied my modified files.

The log formatting shows this nicely.

As for the conflict changes, I had none in this case. I suppose to be able to automate it, some ideas are needed, but I think regular users face no problem, and whoever applies custom code, should be at least knowledgeable enough to resolve it manually if the update process gives him a flag.

Edit: Since this information was compiled, the update process has changed. The backup directory no longer contains the "3.5" directory.

terminal.txt
test.patch.txt

from mycodo.

I feel a little lame here. I dont have 'Mycodo-backups' folder at my home directory and in case of this think your diff/patch command doesnt work properly. Am I wrong?
update:
But it works I see...
update2:
And why you are using /etc/init.d/mycodo restart isnt it correct service mycodo stop when we need to stop and service mycodo start to start?
update3:
patching was done, cause I see different formatting in my log (will try few times more, because Im not shure that understand full process correctly)

from mycodo.

I assume you're referring to mycodo-wrapper.c?

Either will work. The service command is the same as executing the init.d script. I chose to execute the init.d script directly to bypass the use of another program. The service command is just more convenient when working in the terminal. I also wanted to use the full path... the service command may not necessarily be in the same location across linux distributions, but the script should always reside in /etc/init.d/

from mycodo.

Im asking about service command because we need to stop service while patching(I think so), but if we stop it(not restart), then I cant run it with /etc/init.d/mycodo start
p.s. or we doesnt need to stop process?

from mycodo.

There generally isn't a problem with leaving the daemon running while patching/updating unless the database changes. The daemon is running in memory, so even the daemon (mycodo.py) can be modified without repercussions.

However, if the database is being rewritten (updated) when the daemon is trying to read it, there could be unanticipated consequences (no idea what could happen, but worst case would be the daemon crashing or relays turning on or off when they're not supposed to). To avoid this, the update script stops the daemon before the update and starts it after it finishes.

from mycodo.

What do you mean by "I cant run it with /etc/init.d/mycodo start"

from mycodo.

I mean it starting not in background process

this I get after patching when try to restart

root@mycodo:/home/pi# /etc/init.d/mycodo restart
/etc/init.d/mycodo: line 146: /var/www/mycodo/cgi-bin/mycodo-client.py: No such file or directory
Checking if daemon has stopped (please wait 10 seconds)..........root     10450  2.7  1.4  23772 12488 pts/1    Sl   21:11   0:01 python /var/www/mycodo/cgi-bin/mycodo.py
 Daemon is still running. Attempting to stop... mycodo.py (pid 10450) has been killed after 1 seconds.
Because daemon had to be killed, turning off all GPIOs
/etc/init.d/mycodo: line 197: /var/www/mycodo/cgi-bin/GPIO-initialize.py: No such file or directory
chown: cannot access `/var/www/mycodo/cgi-bin/mycodo-wrapper': No such file or directory
chmod: cannot access `/var/www/mycodo/cgi-bin/mycodo-wrapper': No such file or directory

from mycodo.

I have no idea why you are getting those errors. The files should exist. I think I will need to know first why those files don't exist to fix the problem.

from mycodo.

Ok, I will investigate this.

from mycodo.

It because my HEAD was somewhere unknown, and after git reset --hard origin/master all files come to their places.

from mycodo.

I'm not sure I'm following you. Is it moving the files? Deleting them and creating them elsewhere?

from mycodo.

I really dont know why this happens, but after git fetch --all and git reset --hard origin/master files are on their places, I will try to find the moment when they are gone(will try whole update process few times more)

I think its not about update, but when I am restarting daemon with /etc/init.d/mycodo restart and the daemon is not running I got this errors:

Traceback (most recent call last):
  File "/var/www/mycodo/cgi-bin/mycodo-client.py", line 253, in <module>
    menu()
  File "/var/www/mycodo/cgi-bin/mycodo-client.py", line 85, in menu
    c = rpyc.connect("localhost", 18812)
  File "/usr/local/lib/python2.7/dist-packages/rpyc/utils/factory.py", line 89, in connect
    s = SocketStream.connect(host, port, ipv6 = ipv6, keepalive = keepalive)
  File "/usr/local/lib/python2.7/dist-packages/rpyc/core/stream.py", line 132, in connect
    return cls(cls._connect(host, port, **kwargs))
  File "/usr/local/lib/python2.7/dist-packages/rpyc/core/stream.py", line 102, in _connect
    s.connect(sockaddr)
  File "/usr/lib/python2.7/socket.py", line 224, in meth
    return getattr(self._sock,name)(*args)
socket.error: [Errno 111] Connection refused
Checking if daemon has stopped (please wait 10 seconds).......... No daemon running.

from mycodo.

That error is normal. If the daemon is not running, the connection is refused because the threaded communication server of the daemon is not alive to accept the connection. mycodo-client.py is attempting to make a connection with the daemon to instruct it to shut down, but because the daemon is not running, it cannot connect.

However, because this error is expected when issuing the restart command when the daemon is not running or has crashed/stopped responding/etc., It will still attempt to start the server after this error.

from mycodo.

Is it okay that I dont have directory Mycodo-backups/Mycodo-2015-09-24_13-33-22/3.5?
Because how diff will work without it?
update:
A! I will try to make backup now...

from mycodo.

I recently changed the backup process. It should now back up the contents of /home/user/Mycodo/3.5/ to /home/user/Mycodo-backups/Mycodo-2015-09-24_13-33-22/ and there shouldn't be a 3.5 directory. This was done so there would [hopefully] be less confusion about what the backup is (only the contents of /var/www/mycodo/).

from mycodo.

Oops. I said that @boomstick8x was the person who provided the information in #25 (comment), however it was @zsole2. I've updated the comment with the correct attribution. Sorry for the confusion.

from mycodo.

my testing update process:

1. move head at 4024e62 remove display of password in settings (insecure)
2. run backup from web update
3. make patch with:
   diff -Naur --exclude="log" --exclude="config" --exclude="*.pyc" --exclude="img" Mycodo-backups/Mycodo-XXX-XX-XX_XX-XX-XX/3.5 Mycodo/3.5 > test.patch
   Agree with all Reversed (or previously applied) patch detected! Assume -R? [n]
4. git stash
5. git pull
6. /etc/init.d/mycodo restart
7. git stash pop
8. /etc/init.d/mycodo restart
9. and finally i got `The log formatting shows this nicely' :)

Thank you for confusion, now I know a little more about git :)

from mycodo.

To my understanding this process is supposed to allow updating to the latest Mycodo version as well as add custom code from the previous version to the updated version.

How does the patch know what is custom code and what is actually being updated?

from mycodo.

Sorry about all that, even though you know a bit more. I can see now that it seemed like I was requesting you, @boomstick8x, to investigate this. I was merely trying to put the information here that @zsole2 provided in an email, from his efforts to restore a backup.

from mycodo.

Im using terminal version of git only few days, but like you write in first comment, we do:

1. git stash - to save uncommited changes to stack
2. git pull - do update to newest version
3. git stash apply - to apply our last saved changes to updated version(git stash pop will also delete changes from stack after applying)
4. check all is on right places
5. git stash drop - to drop last saved stash (if we dont use git stash pop before)

making patch file with git:
0. git stash - to stash our local changes

1. git stash show -p > name.patch - to stash output in a patch file
   (add --binary option after -p if we also need to stash binary files git stash show -p --binary > name.patch)
2. git apply --stat name.patch - to view all looks good
3. git apply --check name.patch - to verify no errors
4. git apply name.patch - to apply our patch

p.s. git stash list - to see our stack

from mycodo.

@boomstick8x, Thanks for that explanation and for the work to figure it out!

I'll probably add this to the manual after I confirm everything works as expected.

I'm hesitant to add it as an automated feature because it's such a niche area where someone would be adding custom code. There's also the possibility of conflicts if the patch tries to change something that was already updated or if the custom code breaks the system, so I don't want someone accidentally using this option without knowing fully what it's for and the risks involved.

from mycodo.

Dont know about helpful of my work to your project, but for me it was very interesting, I like to learn something new.
If you except that I could somehow help to project I am waiting and always ready :)

from mycodo.

Guys,
Isn't the purpose of these project to learn and accomplish something new and useful?
Personally I did not know about git anything before Mycodo, so already learned a lot. My whole investigation was for finding out why I could not copy back a backup and go forward from there.

Now with all this info, an automated restore is feasible, at least for a regular user where no custom code is used. In this case, simple way reverting back to a working state would be helpful. As in my case was, some bug prevented the humidity PID working, leading me to wanting to restore a backup.

The update script can be modified to include a restore function, and if 'git status' shows any sign of modified files, the script would stop and inform the (probably advanced) user to do it by hand, for which the manual could show a simple guideline, as @boomstick8x summarized quite nicely. The regular user would not notice anything, other than his install went back to an earlier working state, and he can try the update later, when the bug possibly fixed.

from mycodo.

I agree that the function should exist, but I want to make sure that it's foolproof in it's implementation.

For reverting to a previous state, git reset --hard x, where x is the commit ID, should reset everything to that commit. If I incorporate the commit ID in the backup folder title, it would be easy to identify which databases should be restored after the git reset. This would be very easy to implement and there's a reasonable assurance that it can't be incorrectly used. I could add a "Revert" button next to each commit ID (that there is a backup of) of the output of the "Git Commits" button in the Data tab.

I think the use of git stash should be reserved for advanced users and would be most appropriate in the manual.

from mycodo.

I agree completely, on all points.

from mycodo.

After much testing, I think I have a working restore system in place, with commit 5da6d73

Please test it if you are inclined and see if you can find any problems.

from mycodo.

I would recommend always doing a restore from the latest version. I should put in the documentation that before doing a restore, make sure you run an update to have the latest code that carries out the restore. Until this feature is well-tested, be weary of doing a restore from old versions.

from mycodo.

After cloning anew, the backup/restore appears to be working.
The restore log does not appear correctly ... but I saw your bug for that
Is there a specific reason for moving the backups directory to /var ?
What if someone is so behind that the current commit is not visible, therefore the user does not see the red highlight and gets confused? Maybe you should show at least that many lines from git log...
In a related suggestion, it would be convenient if the Lines input-box could remember the previous number of lines, defaulting to 30, as in the code now.

I'll try to test more later.

from mycodo.

I moved the backups to /var because it was too difficult to accurately determine what the correct user was where the backups were previously. As root is the user performing the backup/restore, there is no easy way to determine the correct user other than just searching all the user directories in /home. Having a standard location also alleviates a lot of other problems.

Currently, the commits will show the 30 (default) commits you are ahead and an unlimited number you are behind. So, the current commit (red) will always be visible. You just made me realize I did not limit the display of new commits, but this may not be a bad thing, as you will always see ALL the commits ahead.

from mycodo.

Good point on the user...
Also agree on the new commits, I did not look into the code just fired a question.
Thanks for implementing the Lines input.

from mycodo.

Last feature of backup system in place ("create backup" button) with commit 9f81b57

I'll add the patch notes to the manual soon.

from mycodo.

Finished adding stash/patch information to the manual. Closing this issue. Thanks for the work, guys.

from mycodo.