Comments (10)
Thanks @joelchen,
Could I see the full log of the backup from start -> end? Feel free to remove any sensitive info like hostnames, etc or you can email direct to [email protected] instead.
@islue recently made some changes to this code so maybe they can assist also when there is more info.
from mongodb_consistent_backup.
Here is a full output a few minutes ago @timvaillancourt:
$ sudo mongodb-consistent-backup -c backup.yml
[2017-07-21 23:38:11,150] [INFO] [MainProcess] [Main:init:143] Starting mongodb-consistent-backup version 1.0.3 (git commit: c2a2207ba4e607ab424881f57bff86b086b27fd9)
[2017-07-21 23:38:11,151] [INFO] [MainProcess] [Main:init:144] Loaded config: {"archive": {"method": "tar", "tar": {"compression": "gzip", "threads": 1}, "zbackup": {"binary": "/usr/bin/zbackup", "cache_mb": 128, "compression": "lzma"}}, "authdb": "admin", "backup": {"location": "/var/lib/mongodb-consistent-backup", "method": "mongodump", "mongodump": {"binary": "/usr/bin/mongodump", "compression": "auto"}, "name": "default"}, "configPath": "backup.yml", "environment": "production", "host": "localhost", "lock_file": "/tmp/mongodb-consistent-backup.lock", "log_dir": "/var/log/mongodb-consistent-backup", "notify": {"method": "none"}, "oplog": {"compression": "none", "tailer": {"status_interval": 30}}, "port": 27017, "replication": {"max_lag_secs": 5, "max_priority": 1000}, "sharding": {"balancer": {"ping_secs": 3, "wait_secs": 300}}, "upload": {"method": "s3", "remove_uploaded": true, "s3": {"access_key": "AKIAIJ6OCVNLSST2QT6A", "acl": "none", "bucket_name": "db", "bucket_prefix": "/", "chunk_size_mb": 50, "region": "ap-southeast-1", "retries": 5, "secret_key": "XXXXXXXXXXXXXXXXXXXX", "secure": true, "threads": 1}}}
[2017-07-21 23:38:11,151] [INFO] [MainProcess] [Stage:init:32] Notify stage disabled, skipping
[2017-07-21 23:38:11,156] [INFO] [MainProcess] [State:init:135] Initializing root state directory /var/lib/mongodb-consistent-backup/default
[2017-07-21 23:38:11,157] [INFO] [MainProcess] [State:load_backups:153] Found 0 existing completed backups for set
[2017-07-21 23:38:11,157] [INFO] [MainProcess] [State:init:119] Initializing backup state directory: /var/lib/mongodb-consistent-backup/default/20170721_2338
[2017-07-21 23:38:11,233] [INFO] [MainProcess] [Main:run:268] Running backup in replset mode using seed node(s): localhost:27017
[2017-07-21 23:38:11,246] [INFO] [MainProcess] [Mongodump:choose_compression:48] Mongodump binary supports gzip compression, auto-enabling gzip compression
[2017-07-21 23:38:11,246] [INFO] [MainProcess] [Task:compression:38] Setting Mongodump compression method: gzip
[2017-07-21 23:38:11,247] [INFO] [MainProcess] [Main:run:296] Backup method supports compression, disabling compression in archive step
[2017-07-21 23:38:11,247] [INFO] [MainProcess] [Task:compression:38] Setting Tar compression method: none
[2017-07-21 23:38:11,247] [INFO] [MainProcess] [Stage:run:83] Running stage mongodb_consistent_backup.Backup with task: Mongodump
[2017-07-21 23:38:11,254] [INFO] [MainProcess] [Replset:find_primary:132] Found PRIMARY: prod-rs/mongo0.prod.org:27017 with optime Timestamp(1500651489, 1)
[2017-07-21 23:38:11,254] [INFO] [MainProcess] [Replset:find_secondary:205] Found SECONDARY prod-rs/mongo1.prod.org:27017: {'priority': 1, 'lag': 0, 'optime': Timestamp(1500651489, 1), 'score': 100}
[2017-07-21 23:38:11,254] [WARNING] [MainProcess] [Replset:find_secondary:160] Found down or unhealthy SECONDARY prod-rs/mongo2.prod.org:27017 with state: (not reachable/healthy)
[2017-07-21 23:38:11,255] [INFO] [MainProcess] [Replset:find_secondary:215] Choosing SECONDARY prod-rs/mongo1.prod.org:27017 for replica set prod-rs (score: 100)
[2017-07-21 23:38:11,257] [INFO] [MainProcess] [Mongodump:run:139] Starting backups using mongodump r3.4.5 (options: compression=gzip, threads_per_dump=2)
[2017-07-21 23:38:11,262] [INFO] [MongodumpThread-2] [MongodumpThread:run:125] Starting mongodump backup of prod-rs/mongo1.prod.org:27017
[2017-07-21 23:38:11,288] [INFO] [MongodumpThread-2] [MongodumpThread:wait:92] prod-rs/mongo1.prod.org:27017: writing admin.system.users to
[2017-07-21 23:38:11,289] [INFO] [MongodumpThread-2] [MongodumpThread:wait:92] prod-rs/mongo1.prod.org:27017: done dumping admin.system.users (4 documents)
[2017-07-21 23:38:11,290] [INFO] [MongodumpThread-2] [MongodumpThread:wait:92] prod-rs/mongo1.prod.org:27017: writing admin.system.version to
[2017-07-21 23:38:11,290] [INFO] [MongodumpThread-2] [MongodumpThread:wait:92] prod-rs/mongo1.prod.org:27017: done dumping admin.system.version (2 documents)
[2017-07-21 23:38:11,291] [INFO] [MongodumpThread-2] [MongodumpThread:wait:92] prod-rs/mongo1.prod.org:27017: writing test-prod.provider_notes to
[2017-07-21 23:38:11,291] [INFO] [MongodumpThread-2] [MongodumpThread:wait:92] prod-rs/mongo1.prod.org:27017: writing test-prod.results to
[2017-07-21 23:38:11,295] [INFO] [MongodumpThread-2] [MongodumpThread:wait:92] prod-rs/mongo1.prod.org:27017: done dumping test-prod.results (17 documents)
[2017-07-21 23:38:11,296] [INFO] [MongodumpThread-2] [MongodumpThread:wait:92] prod-rs/mongo1.prod.org:27017: writing test-prod.reports to
[2017-07-21 23:38:11,297] [INFO] [MongodumpThread-2] [MongodumpThread:wait:92] prod-rs/mongo1.prod.org:27017: done dumping test-prod.provider_notes (40 documents)
[2017-07-21 23:38:11,297] [INFO] [MongodumpThread-2] [MongodumpThread:wait:92] prod-rs/mongo1.prod.org:27017: writing test-prod.accounts to
[2017-07-21 23:38:11,299] [INFO] [MongodumpThread-2] [MongodumpThread:wait:92] prod-rs/mongo1.prod.org:27017: done dumping test-prod.reports (14 documents)
[2017-07-21 23:38:11,299] [INFO] [MongodumpThread-2] [MongodumpThread:wait:92] prod-rs/mongo1.prod.org:27017: writing test-prod.measurements to
[2017-07-21 23:38:11,300] [INFO] [MongodumpThread-2] [MongodumpThread:wait:92] prod-rs/mongo1.prod.org:27017: done dumping test-prod.measurements (9 documents)
[2017-07-21 23:38:11,301] [INFO] [MongodumpThread-2] [MongodumpThread:wait:92] prod-rs/mongo1.prod.org:27017: writing test-prod.roles to
[2017-07-21 23:38:11,301] [INFO] [MongodumpThread-2] [MongodumpThread:wait:92] prod-rs/mongo1.prod.org:27017: done dumping test-prod.accounts (12 documents)
[2017-07-21 23:38:11,301] [INFO] [MongodumpThread-2] [MongodumpThread:wait:92] prod-rs/mongo1.prod.org:27017: writing test-prod.assessment to
[2017-07-21 23:38:11,302] [INFO] [MongodumpThread-2] [MongodumpThread:wait:92] prod-rs/mongo1.prod.org:27017: done dumping test-prod.roles (8 documents)
[2017-07-21 23:38:11,302] [INFO] [MongodumpThread-2] [MongodumpThread:wait:92] prod-rs/mongo1.prod.org:27017: writing test-prod.todos to
[2017-07-21 23:38:11,302] [INFO] [MongodumpThread-2] [MongodumpThread:wait:92] prod-rs/mongo1.prod.org:27017: done dumping test-prod.assessment (6 documents)
[2017-07-21 23:38:11,303] [INFO] [MongodumpThread-2] [MongodumpThread:wait:92] prod-rs/mongo1.prod.org:27017: writing test-prod.assessment_filled to
[2017-07-21 23:38:11,303] [INFO] [MongodumpThread-2] [MongodumpThread:wait:92] prod-rs/mongo1.prod.org:27017: done dumping test-prod.todos (2 documents)
[2017-07-21 23:38:11,303] [INFO] [MongodumpThread-2] [MongodumpThread:wait:92] prod-rs/mongo1.prod.org:27017: writing test-prod.broadcasts to
[2017-07-21 23:38:11,304] [INFO] [MongodumpThread-2] [MongodumpThread:wait:92] prod-rs/mongo1.prod.org:27017: done dumping test-prod.assessment_filled (2 documents)
[2017-07-21 23:38:11,304] [INFO] [MongodumpThread-2] [MongodumpThread:wait:92] prod-rs/mongo1.prod.org:27017: writing test-prod.admin_programs to
[2017-07-21 23:38:11,304] [INFO] [MongodumpThread-2] [MongodumpThread:wait:92] prod-rs/mongo1.prod.org:27017: done dumping test-prod.broadcasts (2 documents)
[2017-07-21 23:38:11,305] [INFO] [MongodumpThread-2] [MongodumpThread:wait:92] prod-rs/mongo1.prod.org:27017: writing test-prod.enrolled_programs to
[2017-07-21 23:38:11,305] [INFO] [MongodumpThread-2] [MongodumpThread:wait:92] prod-rs/mongo1.prod.org:27017: done dumping test-prod.admin_programs (1 document)
[2017-07-21 23:38:11,305] [INFO] [MongodumpThread-2] [MongodumpThread:wait:92] prod-rs/mongo1.prod.org:27017: writing test-prod.teams to
[2017-07-21 23:38:11,306] [INFO] [MongodumpThread-2] [MongodumpThread:wait:92] prod-rs/mongo1.prod.org:27017: done dumping test-prod.enrolled_programs (1 document)
[2017-07-21 23:38:11,306] [INFO] [MongodumpThread-2] [MongodumpThread:wait:92] prod-rs/mongo1.prod.org:27017: writing test-prod.organizations to
[2017-07-21 23:38:11,306] [INFO] [MongodumpThread-2] [MongodumpThread:wait:92] prod-rs/mongo1.prod.org:27017: done dumping test-prod.teams (1 document)
[2017-07-21 23:38:11,306] [INFO] [MongodumpThread-2] [MongodumpThread:wait:92] prod-rs/mongo1.prod.org:27017: done dumping test-prod.organizations (1 document)
[2017-07-21 23:38:11,307] [INFO] [MongodumpThread-2] [MongodumpThread:wait:92] prod-rs/mongo1.prod.org:27017: writing captured oplog to
[2017-07-21 23:38:11,605] [INFO] [MongodumpThread-2] [MongodumpThread:run:161] Backup prod-rs/mongo1.prod.org:27017 completed in 0.34 seconds, 0 oplog changes
[2017-07-21 23:38:15,263] [INFO] [MainProcess] [Mongodump:wait:92] All mongodump backups completed successfully
[2017-07-21 23:38:15,265] [INFO] [MainProcess] [Stage:run:92] Completed running stage mongodb_consistent_backup.Backup with task Mongodump in 4.02 seconds
[2017-07-21 23:38:15,266] [INFO] [MainProcess] [Stage:run:83] Running stage mongodb_consistent_backup.Archive with task: Tar
[2017-07-21 23:38:15,275] [INFO] [MainProcess] [Tar:run:56] Archiving backup directories with pool of 2 thread(s)
[2017-07-21 23:38:15,277] [INFO] [PoolWorker-3] [TarThread:run:41] Archiving directory: /var/lib/mongodb-consistent-backup/default/20170721_2338/prod-rs
[2017-07-21 23:38:17,278] [INFO] [MainProcess] [Stage:run:92] Completed running stage mongodb_consistent_backup.Archive with task Tar in 2.01 seconds
[2017-07-21 23:38:17,279] [INFO] [MainProcess] [Stage:run:83] Running stage mongodb_consistent_backup.Upload with task: S3
[2017-07-21 23:38:17,279] [INFO] [MainProcess] [S3:run:81] Starting multipart AWS S3 upload to key: db/default/20170721_2338/prod-rs.tar using 1 threads, 50mb chunks, 5 retries
[2017-07-21 23:38:17,910] [INFO] [PoolWorker-5] [S3UploadThread:run:34] Uploading file: /var/lib/mongodb-consistent-backup/default/20170721_2338/prod-rs.tar (part num: 1)
[2017-07-21 23:38:18,657] [ERROR] [MainProcess] [S3:run:131] Uploading to AWS S3 failed! Error:
[2017-07-21 23:38:18,677] [ERROR] [MainProcess] [Stage:run:95] Stage mongodb_consistent_backup.Upload did not complete!
[2017-07-21 23:38:18,677] [CRITICAL] [MainProcess] [Main:exception:218] Problem performing upload of backup! Error: Stage mongodb_consistent_backup.Upload did not complete!
[2017-07-21 23:38:18,677] [INFO] [MainProcess] [Main:cleanup_and_exit:174] Starting cleanup procedure! Stopping running threads
[2017-07-21 23:38:18,682] [INFO] [MainProcess] [Main:cleanup_and_exit:204] Cleanup complete, exiting
[2017-07-21 23:38:18,682] [INFO] [MainProcess] [Logger:rotate:81] Running rotation of log files
[2017-07-21 23:38:18,682] [INFO] [MainProcess] [Logger:compress:60] Compressing log file: /var/log/mongodb-consistent-backup/backup.default.20170721_0300.log
from mongodb_consistent_backup.
Um, the exception is supposed to be printed out in the error message, but it's empty.
Sorry, I have no idea.
[2017-07-21 23:38:17,910] [INFO] [PoolWorker-5] [S3UploadThread:run:34] Uploading file: /var/lib/mongodb-consistent-backup/default/20170721_2338/prod-rs.tar (part num: 1)
The multipart upload does support 1 part upload according to the document. But if you can, have a try to do a backup with a bigger data set to force a multiple parts upload.
from mongodb_consistent_backup.
I do not have a bigger data set to test in my current environment.
Since this is not reliable, can I suggest this project to integrate with something better maintained like Rclone (https://rclone.org)?
from mongodb_consistent_backup.
I think the fix here is to not use multipart if there is only one chunk to upload (due to the API limitation), or to auto-reduce the chunk size to something smaller than the size of the file.
I will attempt one or both of these ideas in a branch to share soon.
from mongodb_consistent_backup.
@joelchen, I believe this branch will fix this. Please test and confirm.
This git command will check out and build this branch:
cd /a/new/path
git clone -b issue_187_s3_multipart_1chunk https://github.com/timvaillancourt/mongodb_consistent_backup
cd mongodb_consistent_backup
make
./bin/mongodb-consistent-backup <flags here>
URL: https://github.com/timvaillancourt/mongodb_consistent_backup/tree/issue_187_s3_multipart_1chunk
from mongodb_consistent_backup.
FYI, this didn't work for me:
[2017-09-17 02:57:52,729] [ERROR] [MainProcess] [S3:run:140] Uploading to AWS S3 failed! Error: S3ResponseError: 400 Bad Request\
<Error><Code>EntityTooSmall</Code><Message>Your proposed upload is smaller than the minimum allowed size</Message><ProposedSize>5120</ProposedSize><MinSizeAllowed>5242880</MinSizeAllowed><PartNumber>1</PartNumber><ETag>b6b96b0e96964de9ff49d6de2c75c761</ETag><RequestId>2CC14A9DBB3DEB2A</RequestId><HostId>MXm3OKr7Sk+AVcvNZ8rLqqEommOXKLzNAHEqidauJ0UCGY02Z5feuoMqO+gtkq4tDLQ=</HostId></Error>
It looks like only the last part is allowed to be less than the chunk size (I think).
from mongodb_consistent_backup.
Thanks for testing. This minimum limit (5242880 bytes/5.24 megabytes) is good to know.
I'll make another change to skip multipart entirely if the files are less than 5.24mb shortly.
The logic I added to use 2 x threads of 50% of the bytes if the file is less than chunk_size_mb is still valid, I think, so this will add to that logic.
from mongodb_consistent_backup.
I'm not sure dynamic chunk sizing is the right approach. We were able to fix the problem by separating the ACLs logic from the complete_upload
operation. Near as i can tell, there's something wrong with the get_key
operation. I moved the get_key
inside of the ACL flow and everything works (we're not using ACLs).
I realize this isn't a complete solution, but I think it's the right track.
** UPDATE ** Submitted a PR which changes slightly how the ACLs are applied. This fixes the problem in our tests.
from mongodb_consistent_backup.
Hello all,
I believe this "S3 overhaul" branch fixes the issues discussed here with S3 upload. There was a lot of problems to tackle so this is a large change:
https://github.com/timvaillancourt/mongodb_consistent_backup/tree/s3_upload_overhaul_v1
PR #249 will merge these fixes and explains each of them in detail.
Please test out this branch, any feedback is appreciated before it's merged.
from mongodb_consistent_backup.
Related Issues (20)
- Upload Rsync prepare_dest_dir prepares wrong location
- Tailer csReplSet/ra1svlrp4i:27019 received NetworkTimeout exception: ra1svlrp4k:27019: timed out. Attempting retry HOT 3
- Encryption for zbackup failed when there is existing encrypted zbackup folder
- Urgent: Oplog Resolver Intermittently doesn't Complete Resolving HOT 4
- New Bugfix Release HOT 4
- Backup failed: Could not acquire lock on file: /var/lib/.../mongodb-consistent-backup_META.lock HOT 2
- cannot dump with oplog if admin.system.version is modified HOT 3
- docker install problem HOT 1
- get CRITICAL error
- UnicodeDecodeError: 'ascii' codec can't decode byte 0xc2 in position 5: ordinal not in range(128) HOT 1
- Mongodump problem with large deployments HOT 1
- [v1.4.1] Backup to s3 not read from config file HOT 2
- restore data question HOT 4
- can we backup two mongo cluster at the same time? HOT 1
- Unrecognized field snapshot HOT 4
- error reading oplog bson input: unexpected EOF
- the mongodb-consistent-backup is blocked
- Failed to execute PEX file, missing linux_x86_64-cp-27-cp27mu compatible dependencies for:
- Failed to execute PEX file, missing linux_x86_64-cp-27-cp27mu compatible dependencies for: HOT 1
- installation with errors
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from mongodb_consistent_backup.