demizer / gds Goto Github PK
View Code? Open in Web Editor NEWEasy multi-mismatched-device backups.
License: MIT License
Easy multi-mismatched-device backups.
License: MIT License
The first major backup of 6.5TB had only a single apparent failure:
time="2015-07-31T19:32:44-07:00" level=fatal msg="Sync failed: createFile: setMetaData chown: chown /mnt/backup/data/home/demizer/code/code/c/pacman/lib/libalpm/util-common.c: no such file or directory"
util-common.c
is a symlink, so the problem could be that chmod is following the symlink but the file it is pointing at doesn't exist.
After a drive has become full, save the metadata log in its current state to the drive before unmounting. This is currently done manually.
When running gds for the first time on my huge dataset, a panic occurred when it encountered my root only easyrsa files with perms 2700/drwx--S---
.
For now the work around is to run as root. gds should gracefully notify the user that files cannot be read and give them the option to continue or cancel.
If the destination drive is nearly full, and the current file is larger than the available space, the next files that can fit into the available space should be copied and the larger files saved for the next mounted drive.
After the user has mounted a drive, the permissions of the mounted drive can be root, which would cause the write to fail if todrives has been run as a regular user.
Need to fully test the command argument parsing.
My test config.yml had "drives" instead of "devices" and gds continued anyways. This caused a nil pointer dereference panic in NewCatalog()
when it attempted to load device info that did not exist.
It takes 30 seconds to run about 20 tests.
Just like copying, but measure bytes being hashed.
The function is pages of code long, break it up! Split out the io.Copy() calls to separate functions so that they can be mocked in testing.
If enter is accidentally pressed before a device switch is made, it would be catastrophic. gds should verify the switch has been made before continuing. If not, ask the user again to switch out the device. Checks can be made against the PARTUUID. This should be done from the start.
If the values passed to the --log
, --config``, and
--contextarguments contain the suffix
.gz`, the the output will be contained in a compressed archive.
It would be nice to be able to sync files to multiple drives at once to speed up the sync process and saturate IO.
The hashes for the files are saved once backup is complete. There should be an option to use these hashes to check the origin files for corruption.
Some parts of the code base use fmt.Errorf() to add context to the error output. gds should really be using https://godoc.org/gopkg.in/inconshreveable/log15.v2#hdr-Context_loggers
Need to dump the catalog to a file. Not really sure if this will be useful when update syncing is implemented, but for now we just need to serialize this data.
XDG_HOME_CONFIG/gds/$(date --iso-8601=seconds)_context.json
The issue is complete when tests have been written for the following scenarios:
backup
vs. backup/
It would be nice to have a command option for copying the gds static binary (for multiple OSes) and catalog copied to backup device for ease of use. This would need to be duplicated across all devices so that the user can plug in any device and get to the files they need quickly without first having to install gds.
Use external program to verify results of saving file checking. This may also speed the process.
Add a command argument to send log output to file.
Need a proper argument parser and arguments!
The scrolling log output is hard to read, especially when being piped to a file due to logrus's logfmt support. It would be nice to have a terminal GUI for showing progress bars and such.
https://github.com/jroimartin/gocui
NewFileList()
https://en.wikipedia.org/wiki/Bash_(Unix_shell)Need to clear the line before sending a new one. Also need to fix carriage return in go-elog.
Support gzip, bzip2, zlib, and lzma (http://godoc.org/code.google.com/p/lzma)
sync()
copy throws an error, and goroutines are racing.
time="2015-07-29T23:47:23-07:00" level=error msg="Error copying file!" deviceSize=5005516800 deviceUsage=4965451465 filePath="/home/demizer/documents/development/HTML5/jQuery eBooks Collection/Packt Publishing jQuery 1.4 Plugin Development, Beginner's Guide (2010).pdf" fileSize=10319491
Actual device size in this case:
demizer@helium$ \df -B1 /mnt/backup1 /mnt/backup2
Filesystem 1B-blocks Used Available Use% Mounted on
/dev/loop0 5284429824 5015994368 0 100% /mnt/backup1
/dev/loop1 5284429824 10477568 5005516800 1% /mnt/backup2
I modified the preSync()
function to dump the file size + file name of all the files that will be saved to the device. Once I got the output, I added up the sizes of the files:
demizer@helium $ cat 2015-07-30T01:01:44-0700_output | cut -d"," -f 1 | awk '{total = total + $0}END{print total}'
5008878454
Something in NewCatalog()
is off.
Basic sync includes getting the files on formatted devices, this is largely done. Update sync is syncing backup paths to devices that already contain files.
func TestSha1Sum(t *testing.T) {
_, err := sha1sum("/root")
if err == nil {
t.Error("EXPECT: error permission denied GOT: No errors")
}
}
Root user can read "/root", so this test will fail if being run by root user. This test should not fail if being run by a privileged user.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.