twindb / backup Goto Github PK

View Code? Open in Web Editor NEW

78.0 78.0 42.0 1.47 MB

TwinDB Backup

Home Page: https://twindb-backup.readthedocs.io/en/stable/

License: Other

Makefile 0.83% Shell 0.94% Ruby 6.26% Python 91.27% Puppet 0.69% Pascal 0.01%

backup's People

Contributors

Stargazers

Watchers

Forkers

gitter-badger aimdotsh soft-way dikang123 10oo01sir tricloud weiyanwei412 jessebaomihua ardabeyazoglu picassio luhouhua xiaohaoyu andrejfedorow liugaohui aliengain el4v howard0su celery-payroll ratanasoth emersongaudencio ernstae liuxinwang gczheng sapisuper jmp12315 woihzh longqidong lightbiaggi wemingtt atkins126 locmai zhouwanchun ryancpeters ricardojosueperezaltamirano kloostermanw mailanetworks openaxon vsabella databaseleague haitaocq dimied

backup's Issues

twindb-backup clone mysql fails to determine free memory

TwinDB Backup version: 2.14.1

Description

$ twindb-backup --config ~/.twindb-backup.cfg clone mysql a.b.c.d e.f.g.h
2017-10-13 13:23:25,730: ERROR: clone.clone_mysql():85: Failed to stop MySQL on <twindb_backup.destination.ssh.Ssh object at 0x7f87b599dcd0>
Traceback (most recent call last):
  File "/home/okuzminskyi/.virtualenvs/backup/bin/twindb-backup", line 11, in <module>
    load_entry_point('twindb-backup', 'console_scripts', 'twindb-backup')()
  File "/home/okuzminskyi/.virtualenvs/backup/lib/python2.7/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/home/okuzminskyi/.virtualenvs/backup/lib/python2.7/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/home/okuzminskyi/.virtualenvs/backup/lib/python2.7/site-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/okuzminskyi/.virtualenvs/backup/lib/python2.7/site-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/okuzminskyi/.virtualenvs/backup/lib/python2.7/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/okuzminskyi/.virtualenvs/backup/lib/python2.7/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/home/okuzminskyi/.virtualenvs/backup/lib/python2.7/site-packages/click/decorators.py", line 64, in new_func
    return ctx.invoke(f, obj, *args[1:], **kwargs)
  File "/home/okuzminskyi/.virtualenvs/backup/lib/python2.7/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/home/okuzminskyi/src/backup/twindb_backup/cli.py", line 231, in clone_mysql_backup
    netcat_port=netcat_port)
  File "/home/okuzminskyi/src/backup/twindb_backup/clone.py", line 130, in clone_mysql
    binlog, position = dst_mysql.apply_backup(datadir)
  File "/home/okuzminskyi/src/backup/twindb_backup/source/remote_mysql_source.py", line 166, in apply_backup
    free_mem = int(stdout_.read()) * 1024
ValueError: invalid literal for int() with base 10: ''

Description

clone doesn't log properly

Traceback (most recent call last):
  File "/opt/rh/python27/root/usr/lib64/python2.7/logging/__init__.py", line 859, in emit
    msg = self.format(record)
  File "/opt/rh/python27/root/usr/lib64/python2.7/logging/__init__.py", line 732, in format
    return fmt.format(record)
  File "/opt/rh/python27/root/usr/lib64/python2.7/logging/__init__.py", line 471, in format
    record.message = record.getMessage()
  File "/opt/rh/python27/root/usr/lib64/python2.7/logging/__init__.py", line 335, in getMessage
    msg = msg % self.args
TypeError: %d format: a number is required, not str
Logged from file clone.py, line 126

twindb-backup doesn't work with old awscli

TwinDB Backup version: 2.8.2

Description

twindb-backup depends on aws streaming feature that was introduces around 1.4.4.

If aws is older than 1.4.4 twindb-backup will fail with something like

2017-01-25 19:58:02,733: ERROR: base_destination._save():71: aws s3 cp - s3://<BUCKET>/ip-172-30-0-245/daily/files/_etc-2017-01-25_19_58_01.tar.gz exited with error code 1
2017-01-25 19:58:02,734: INFO: base_destination._save():73: [Errno 2] No such file or directory: '/etc/twindb/-'
Completed 1 part(s) with ... file(s) remaining

Solution might be either check aws version and notify that it's too old or maybe just use boto3?

tmp directory gets full

TwinDB Backup version: 2.12.0
Operating System: CentOS 7

Description

The script doesn't gracefully handle full disk errors. Also, it probably leaves backup copies in tmp dir.

What I Did

# twindb-backup restore mysql --dst /var/log/mysql/mysql_3307_tmp --cache /var/log/mysql/cache s3://.../.../mysql-2017-05-22_09_30_02.xbstream.gz
Traceback (most recent call last):
  File "/usr/bin/twindb-backup", line 9, in <module>
    load_entry_point('twindb-backup==2.12.0', 'console_scripts', 'twindb-backup')()
  File "/opt/twindb-backup/embedded/lib/python2.7/site-packages/click-6.7-py2.7.egg/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/opt/twindb-backup/embedded/lib/python2.7/site-packages/click-6.7-py2.7.egg/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/opt/twindb-backup/embedded/lib/python2.7/site-packages/click-6.7-py2.7.egg/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/opt/twindb-backup/embedded/lib/python2.7/site-packages/click-6.7-py2.7.egg/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/opt/twindb-backup/embedded/lib/python2.7/site-packages/click-6.7-py2.7.egg/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/opt/twindb-backup/embedded/lib/python2.7/site-packages/click-6.7-py2.7.egg/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/opt/twindb-backup/embedded/lib/python2.7/site-packages/click-6.7-py2.7.egg/click/decorators.py", line 64, in new_func
    return ctx.invoke(f, obj, *args[1:], **kwargs)
  File "/opt/twindb-backup/embedded/lib/python2.7/site-packages/click-6.7-py2.7.egg/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/opt/twindb-backup/embedded/lib/python2.7/site-packages/twindb_backup-2.12.0-py2.7.egg/twindb_backup/cli.py", line 115, in restore_mysql
    restore_from_mysql(cfg, backup_copy, dst, Cache(cache))
  File "/opt/twindb-backup/embedded/lib/python2.7/site-packages/twindb_backup-2.12.0-py2.7.egg/twindb_backup/restore.py", line 365, in restore_from_mysql
    restore_from_mysql_incremental(stream, dst_dir, config)
  File "/opt/twindb-backup/embedded/lib/python2.7/site-packages/twindb_backup-2.12.0-py2.7.egg/twindb_backup/restore.py", line 190, in restore_from_mysql_incremental
    inc_dir = tempfile.mkdtemp()
  File "/opt/twindb-backup/embedded/lib/python2.7/tempfile.py", line 325, in mkdtemp
    dir = gettempdir()
  File "/opt/twindb-backup/embedded/lib/python2.7/tempfile.py", line 269, in gettempdir
    tempdir = _get_default_tempdir()
  File "/opt/twindb-backup/embedded/lib/python2.7/tempfile.py", line 212, in _get_default_tempdir
    ("No usable temporary directory found in %s" % dirlist))
IOError: [Errno 2] No usable temporary directory found in ['/tmp', '/var/tmp', '/usr/tmp', '/root']

option to make backup copy public

TwinDB Backup version: 2.11.0

Description

For s3 backup copies make a command that would share the backup copy and return URL to download it.

Warning: 'wsrep_desync' is already ON.

TwinDB Backup version: 2.5.0

Description

twindb-backup sets wsrep_desync to ON when it takes a backup from Percona XtraDB Cluster. if the backup job fails on whatever reason wsrep_desync if not reverted to OFF.

Next twindb-backup run will produce a warning:

/usr/lib/python2.7/site-packages/twindb_backup/backup.py:49: Warning: 'wsrep_desync' is already ON.
c.execute("set global wsrep_desync=ON")

Description

Please add support for MariaBackup (drop in replacement for xtrabackup) to support MariaDB 10.2+

packages for xenial are missing

TwinDB Backup version: 2.13.0
Python version: N/A
Operating System: Ubuntu xenial

Description

Ubuntu Xenial is an LTS release. We should build twindb-backup packages for it.

Backup failed

  File "/opt/twindb-backup/embedded/lib/python2.7/site-packages/twindb_backup-2.16.1-py2.7.egg/twindb_backup/cli.py", line 135, in backup
    binlogs_only=binlogs_only
  File "/opt/twindb-backup/embedded/lib/python2.7/site-packages/twindb_backup-2.16.1-py2.7.egg/twindb_backup/backup.py", line 371, in run_backup_job
    backup_everything(run_type, cfg, binlogs_only=binlogs_only)
  File "/opt/twindb-backup/embedded/lib/python2.7/site-packages/twindb_backup-2.16.1-py2.7.egg/twindb_backup/backup.py", line 317, in backup_everything
    backup_binlogs(run_type, config)
  File "/opt/twindb-backup/embedded/lib/python2.7/site-packages/twindb_backup-2.16.1-py2.7.egg/twindb_backup/backup.py", line 261, in backup_binlogs
    dst.delete(copy.key + ".gz")
  File "/opt/twindb-backup/embedded/lib/python2.7/site-packages/twindb_backup-2.16.1-py2.7.egg/twindb_backup/destination/ssh.py", line 114, in delete
    self.execute_command(cmd)
  File "/opt/twindb-backup/embedded/lib/python2.7/site-packages/twindb_backup-2.16.1-py2.7.egg/twindb_backup/destination/ssh.py", line 234, in execute_command
    background=background
  File "/opt/twindb-backup/embedded/lib/python2.7/site-packages/twindb_backup-2.16.1-py2.7.egg/twindb_backup/ssh/client.py", line 135, in execute
    % (cmd, exit_code))
twindb_backup.ssh.exceptions.SshClientException: rm localhost.localdomain/binlog/mysql-bin.000001.gz exited with code 1

It seems that SshClient uses invalid path to binlog backup. Similar problem was in previous version of util with mysql backups.

list python-psutil.x86_64 as dependency

TwinDB Backup version: 2.2.0
Operating System: centos 6 and 7

Description

python-psutil.x86_64 should be listed as dependency

Description

malformed status file causes the tool crash

What I Did

2018-01-23 16:44:11,617: DEBUG: backup.run_backup_job():233: daily
2018-01-23 16:44:12,473: DEBUG: backup.set_open_files_limit():169: Setting max files limit to 1048577
2018-01-23 16:44:12,474: DEBUG: __init__.get_directories_to_backup():118: Not backing up files
2018-01-23 16:44:12,474: DEBUG: configuration.get_destination():28: Destination in the config ssh
2018-01-23 16:44:12,475: DEBUG: base_source._get_name():50: Suffix = xbstream
2018-01-23 16:44:12,476: DEBUG: ssh.execute_command():266: Executing: mkdir -p "/backups_mysql/www.xxx.com/daily/mysql"
2018-01-23 16:44:12,873: DEBUG: client.execute():72: Executing mkdir -p "/backups_mysql/www.xxx.com/daily/mysql"
2018-01-23 16:44:13,350: DEBUG: mysql_source.get_stream():123: Running innobackupex --defaults-file=/etc/twindb/my.cnf --stream=xbstream --host=127.0.0.1 .
2018-01-23 16:44:13,355: DEBUG: mysql_source.get_stream():129: Running innobackupex --defaults-file=/etc/twindb/my.cnf --stream=xbstream --host=127.0.0.1 .
2018-01-23 18:02:23,974: DEBUG: mysql_source.get_stream():143: Successfully streamed innobackupex output
2018-01-23 18:02:23,975: DEBUG: mysql_source._update_backup_info():163: innobackupex error log file /tmp/tmpCi7ULD
2018-01-23 18:02:24,271: DEBUG: client.execute():72: Executing bash -c 'if test -s /backups_mysql/www.xxx.com/status; then echo exists; else echo not_exists; fi'
Traceback (most recent call last):
  File "/usr/bin/twindb-backup", line 9, in <module>
    load_entry_point('twindb-backup==2.15.3', 'console_scripts', 'twindb-backup')()
  File "/opt/twindb-backup/embedded/lib/python2.7/site-packages/click-6.7-py2.7.egg/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/opt/twindb-backup/embedded/lib/python2.7/site-packages/click-6.7-py2.7.egg/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/opt/twindb-backup/embedded/lib/python2.7/site-packages/click-6.7-py2.7.egg/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/opt/twindb-backup/embedded/lib/python2.7/site-packages/click-6.7-py2.7.egg/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/opt/twindb-backup/embedded/lib/python2.7/site-packages/click-6.7-py2.7.egg/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/opt/twindb-backup/embedded/lib/python2.7/site-packages/click-6.7-py2.7.egg/click/decorators.py", line 64, in new_func
    return ctx.invoke(f, obj, *args[1:], **kwargs)
  File "/opt/twindb-backup/embedded/lib/python2.7/site-packages/click-6.7-py2.7.egg/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/opt/twindb-backup/embedded/lib/python2.7/site-packages/twindb_backup-2.15.3-py2.7.egg/twindb_backup/cli.py", line 82, in backup
    run_backup_job(cfg, run_type, lock_file=lock_file)
  File "/opt/twindb-backup/embedded/lib/python2.7/site-packages/twindb_backup-2.15.3-py2.7.egg/twindb_backup/backup.py", line 235, in run_backup_job
    backup_everything(run_type, cfg)
  File "/opt/twindb-backup/embedded/lib/python2.7/site-packages/twindb_backup-2.15.3-py2.7.egg/twindb_backup/backup.py", line 186, in backup_everything
    backup_mysql(run_type, config)
  File "/opt/twindb-backup/embedded/lib/python2.7/site-packages/twindb_backup-2.15.3-py2.7.egg/twindb_backup/backup.py", line 119, in backup_mysql
    status = prepare_status(dst, src, run_type, src_name, backup_start)
  File "/opt/twindb-backup/embedded/lib/python2.7/site-packages/twindb_backup-2.15.3-py2.7.egg/twindb_backup/backup.py", line 136, in prepare_status
    status = dst.status()
  File "/opt/twindb-backup/embedded/lib/python2.7/site-packages/twindb_backup-2.15.3-py2.7.egg/twindb_backup/destination/base_destination.py", line 120, in status
    return self._read_status()
  File "/opt/twindb-backup/embedded/lib/python2.7/site-packages/twindb_backup-2.15.3-py2.7.egg/twindb_backup/destination/ssh.py", line 230, in _read_status
    return json.loads(base64.b64decode(stdout.read()))
  File "/opt/twindb-backup/embedded/lib/python2.7/json/__init__.py", line 338, in loads
    return _default_decoder.decode(s)
  File "/opt/twindb-backup/embedded/lib/python2.7/json/decoder.py", line 366, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/opt/twindb-backup/embedded/lib/python2.7/json/decoder.py", line 382, in raw_decode
    obj, end = self.scan_once(s, idx)
ValueError: Unterminated string starting at: line 1 column 170178 (char 170177)

Corrupted status

[root@storage ~]# twindb-backup --debug backup hourly
2018-09-01 18:30:06,219: DEBUG: backup.run_backup_job():269: hourly
2018-09-01 18:30:06,698: DEBUG: backup.set_open_files_limit():205: Setting max files limit to 1048577
2018-09-01 18:30:06,698: DEBUG: _init_.get_directories_to_backup():120: Directories to backup []
2018-09-01 18:30:06,704: DEBUG: configuration.get_destination():29: Destination in the config ssh
2018-09-01 18:30:07,061: DEBUG: client.execute():110: Executing command: bash -c 'if test -s /backup_dir/hostname/status; then echo exists; else echo not_exists; fi'
2018-09-01 18:30:07,380: DEBUG: client.get_remote_handlers():163: Try to get remote handlers: cat /backup_dir/hostname/status
2018-09-01 18:30:07,471: DEBUG: mysql_status.candidate_parent():127: Looking a parent candidate for hourly run
2018-09-01 18:30:07,472: DEBUG: mysql_status.candidate_parent():133: Checking 45 hourly copies
2018-09-01 18:30:07,472: DEBUG: mysql_status.candidate_parent():138: Found parent MySQLCopy(hostname/hourly/mysql/mysql-2018-08-28_17_00_02.xbstream.gz)
2018-09-01 18:30:07,473: DEBUG: mysql_status.candidate_parent():127: Looking a parent candidate for hourly run
2018-09-01 18:30:07,473: DEBUG: mysql_status.candidate_parent():133: Checking 45 hourly copies
2018-09-01 18:30:07,473: DEBUG: mysql_status.candidate_parent():138: Found parent MySQLCopy(hostname/hourly/mysql/mysql-2018-08-28_17_00_02.xbstream.gz)
2018-09-01 18:30:07,473: DEBUG: backup.backup_mysql():141: Creating source

{'backup_type': 'incremental', 'parent_lsn': 97968913006, 'dst': <twindb_backup.destination.ssh.Ssh object at 0x7f3034d98e10>, 'xtrabackup_binary': u'/opt/twindb-backup/embedded/bin/xtrabackup'}
2018-09-01 18:30:07,474: DEBUG: backup._backup_stream():56: keep_local_path is not present in the config file
2018-09-01 18:30:07,474: DEBUG: base_source._get_name():55: Suffix = xbstream.gz
2018-09-01 18:30:07,474: DEBUG: ssh.execute_command():217: Executing: mkdir -p "/backup_dir/hostname/hourly/mysql"
2018-09-01 18:30:07,771: DEBUG: client.execute():110: Executing command: mkdir -p "/backup_dir/hostname/hourly/mysql"
2018-09-01 18:30:08,100: DEBUG: client.get_remote_handlers():163: Try to get remote handlers: cat - > /backup_dir/hostname/hourly/mysql/mysql-2018-09-01_18_30_07.xbstream.gz
2018-09-01 18:30:08,131: DEBUG: mysql_source.is_galera():407: Galera is not supported or not enabled
2018-09-01 18:30:08,131: DEBUG: mysql_source.get_stream():138: Running /opt/twindb-backup/embedded/bin/xtrabackup --defaults-file=/etc/twindb/.my.cnf --stream=xbstream --host=127.0.0.1 --backup --target-dir . --incremental-basedir . --incremental-lsn=97968913006
2018-09-01 18:30:08,135: DEBUG: mysql_source.is_galera():407: Galera is not supported or not enabled
2018-09-01 18:30:08,135: DEBUG: mysql_source.get_stream():144: Running /opt/twindb-backup/embedded/bin/xtrabackup --defaults-file=/etc/twindb/.my.cnf --stream=xbstream --host=127.0.0.1 --backup --target-dir . --incremental-basedir . --incremental-lsn=97968913006

Description

Implement mysqldump as a backups source.
Run mysqldump only if full backup is scheduled.

Part number must be an integer between 1 and 10000, inclusive

TwinDB Backup version: 2.1.9
Python version: na
Operating System: na

Description

2016-11-08 02:56:43,467: ERROR: base_destination._save():68: aws s3 cp - s3://<bucket>/daily/<hostname>/files/<filename>-2016-11-08_00_00_07.tar.gz exited with error code 1
2016-11-08 02:56:43,467: ERROR: base_destination._save():72: upload failed: ./- to s3://<bucket>/daily/<hostname>/files/<filename>-2016-11-08_00_00_07.tar.gz
An error occurred (InvalidArgument) when calling the UploadPart operation: Part number must be an integer between 1 and 10000, inclusive

AWS errors if backup destination is SSH

By default the tool copied the backups to SSH destination.
However it produces errors complaining about missing aws creds:

An error occurred (InvalidAccessKeyId) when calling the ListObjects operation: The AWS Access Key Id you provided does not exist in our records.
make_bucket failed: s3://twindb-backups/ An error occurred (InvalidAccessKeyId) when calling the CreateBucket operation: The AWS Access Key Id you provided does not exist in our records.

2016-12-05 02:30:55,823: ERROR: restore.restore_from_mysql_full():68: Failed to unarchive s3://*****/daily/mysql/mysql-2016-12-01_01_13_55.xbstream.gz: [Errno 2] No such file or directory

TwinDB Backup version: 2.4.0

Description

If xtrabackup isn't present on the system twindb-backup will exit with error 2016-12-05 02:30:55,823: ERROR: restore.restore_from_mysql_full():68: Failed to unarchive s3://****/daily/mysql/mysql-2016-12-01_01_13_55.xbstream.gz: [Errno 2] No such file or directory

What I Did

twindb-backup restore mysql s3://****/daily/mysql/mysql-2016-12-01_01_13_55.xbstream.gz

Workaround

Install Percona Xtrabackup

# yum install http://www.percona.com/downloads/percona-release/redhat/0.1-3/percona-release-0.1-3.noarch.rpm
# yum install percona-xtrabackup

Solution

List xtrabackup as a dependency

Custom SSH port

TwinDB Backup version: 2.15.2

Description

Custom SSH port is passed as port option in the ssh section of the config.
Support both options ssh_port and port for the destination, describe the in documentation.
https://twindb-backup.readthedocs.io/en/master/usage.html#ssh-settings

twindb-backup clone mysql should run innobackupex with sudo

TwinDB Backup version: 2.14.1

Description

innobackupex needs to read MySQL owned files - e.g. should be run as root.

Enhancement - Add Google Cloud Storage Support

Hi,

I need to store my backups in google cloud storage, which is also popular. I can implement it, if you point me to amazon-s3 files.

Specify hostname for status and verify subcommands

[root@storage ~]# twindb-backup status { "daily": {}, "hourly": {}, "monthly": {}, "weekly": {}, "yearly": {} }
[root@mysql ~]# twindb-backup ls { .... }

Configs of twindb-backup are cloned.

if backup job fails remove artifacts

TwinDB Backup version: 2.5.0

Description

A backup job failed because of xtrabackup error:

170103 21:39:05 Executing FLUSH NO_WRITE_TO_BINLOG TABLES...
170103 21:39:05 Executing FLUSH TABLES WITH READ LOCK...
Error: failed to execute query FLUSH TABLES WITH READ LOCK: Lock wait timeout exceeded; try restarting transaction

The tool didn't save a status record (which is expected), but it also left a backup copy in S3 which is in unknown state, I don't know if it's usable or not.
The tool should exit gracefully and do some clean-up.

Packaging issue - postinst not being called on package upgrade

TwinDB Backup version: 2.9.3-1
Python version: 2.7
Operating System: CentOS 7

Description

Upgrading the package from a prior version does not execute postinst

Status isn't updated after backup

List of backups:

[root@ip-10-0-52-101 centos]# twindb-backup ls | grep ip-10-0-52-101/hourly/mysql
s3://***/ip-10-0-52-101/hourly/mysql/mysql-2018-04-01_18_00_03.xbstream.gz
s3://***/ip-10-0-52-101/hourly/mysql/mysql-2018-04-01_19_00_05.xbstream.gz
s3://***/ip-10-0-52-101/hourly/mysql/mysql-2018-04-01_20_00_04.xbstream.gz
s3://***/ip-10-0-52-101/hourly/mysql/mysql-2018-04-01_21_00_05.xbstream.gz
s3://***/ip-10-0-52-101/hourly/mysql/mysql-2018-04-01_22_00_04.xbstream.gz
s3://***/ip-10-0-52-101/hourly/mysql/mysql-2018-04-01_23_00_03.xbstream.gz
s3://***/ip-10-0-52-101/hourly/mysql/mysql-2018-04-02_00_00_04.xbstream.gz
s3://***/ip-10-0-52-101/hourly/mysql/mysql-2018-04-02_01_17_14.xbstream.gz
s3://***/ip-10-0-52-101/hourly/mysql/mysql-2018-04-02_01_28_40.xbstream.gz
s3://***/ip-10-0-52-101/hourly/mysql/mysql-2018-04-02_02_00_03.xbstream.gz
s3://***/ip-10-0-52-101/hourly/mysql/mysql-2018-04-02_03_00_04.xbstream.gz
s3://***/ip-10-0-52-101/hourly/mysql/mysql-2018-04-02_04_00_03.xbstream.gz
s3://***/ip-10-0-52-101/hourly/mysql/mysql-2018-04-02_05_00_05.xbstream.gz
s3://***/ip-10-0-52-101/hourly/mysql/mysql-2018-04-02_06_00_03.xbstream.gz
s3://***/ip-10-0-52-101/hourly/mysql/mysql-2018-04-02_07_00_03.xbstream.gz
s3://***/ip-10-0-52-101/hourly/mysql/mysql-2018-04-02_08_00_03.xbstream.gz
s3://***/ip-10-0-52-101/hourly/mysql/mysql-2018-04-02_09_00_04.xbstream.gz
s3://***/ip-10-0-52-101/hourly/mysql/mysql-2018-04-02_10_00_03.xbstream.gz
s3://***/ip-10-0-52-101/hourly/mysql/mysql-2018-04-02_11_00_04.xbstream.gz
s3://***/ip-10-0-52-101/hourly/mysql/mysql-2018-04-02_12_00_04.xbstream.gz
s3://***/ip-10-0-52-101/hourly/mysql/mysql-2018-04-02_13_00_04.xbstream.gz
s3://***/ip-10-0-52-101/hourly/mysql/mysql-2018-04-02_14_00_04.xbstream.gz
s3://***/ip-10-0-52-101/hourly/mysql/mysql-2018-04-02_15_00_03.xbstream.gz
s3://***/ip-10-0-52-101/hourly/mysql/mysql-2018-04-02_16_00_04.xbstream.gz
s3://***/ip-10-0-52-101/hourly/mysql/mysql-2018-04-02_17_00_04.xbstream.gz

From status file:

[root@ip-10-0-52-101 centos]# twindb-backup status | grep ip-10-0-52-101/hourly/mysql
        "ip-10-0-52-101/hourly/mysql/mysql-2018-04-01_19_00_05.xbstream.gz": {
        "ip-10-0-52-101/hourly/mysql/mysql-2018-04-01_22_00_04.xbstream.gz": {
        "ip-10-0-52-101/hourly/mysql/mysql-2018-04-01_23_00_03.xbstream.gz": {
        "ip-10-0-52-101/hourly/mysql/mysql-2018-04-02_01_17_14.xbstream.gz": {
        "ip-10-0-52-101/hourly/mysql/mysql-2018-04-02_01_28_40.xbstream.gz": {
        "ip-10-0-52-101/hourly/mysql/mysql-2018-04-02_03_00_04.xbstream.gz": {
        "ip-10-0-52-101/hourly/mysql/mysql-2018-04-02_05_00_05.xbstream.gz": {
        "ip-10-0-52-101/hourly/mysql/mysql-2018-04-02_07_00_03.xbstream.gz": {
        "ip-10-0-52-101/hourly/mysql/mysql-2018-04-02_09_00_04.xbstream.gz": {
        "ip-10-0-52-101/hourly/mysql/mysql-2018-04-02_10_00_03.xbstream.gz": {
        "ip-10-0-52-101/hourly/mysql/mysql-2018-04-02_12_00_04.xbstream.gz": {
        "ip-10-0-52-101/hourly/mysql/mysql-2018-04-02_14_00_04.xbstream.gz": {
        "ip-10-0-52-101/hourly/mysql/mysql-2018-04-02_16_00_04.xbstream.gz": {

twindb-backup clone mysql should start/stop MySQL service with sudo

TwinDB Backup version: 2.14.1

Description

Start/stop MySQL with sudo.

def _mysql_service(dst, action):
    """Start or stop MySQL service

    :param dst: Destination server
    :type dst: Ssh
    :param action: string start or stop
    :type action: str
    """
    for service in ['mysqld', 'mysql']:
        try:
            return dst.execute_command(
                "service %s %s" % (service, action),
                quiet=True
            )
        except SshClientException:
            pass

    raise TwinDBBackupError('Failed to %s MySQL on %r'
                            % (action, dst))

twindb-backup clone mysql should run chown with sudo

TwinDB Backup version: 2.14.1

Description

On the destination a MySQL files should change owner to mysql.
As long as only root can do it, chown should be run with sudo

        self._ssh_client.execute("sudo chown -R mysql %s" % datadir)

OSError: [Errno 7] Argument list too long

Python: 2.7.12

I am now getting this error on one server running 2.14.1

Traceback (most recent call last):
 File "/usr/bin/twindb-backup", line 9, in <module>
   load_entry_point('twindb-backup==2.14.1', 'console_scripts', 'twindb-backup')()
 File "/opt/twindb-backup/embedded/lib/python2.7/site-packages/click-6.7-py2.7.egg/click/core.py", line 722, in __call__
   return self.main(*args, **kwargs)
 File "/opt/twindb-backup/embedded/lib/python2.7/site-packages/click-6.7-py2.7.egg/click/core.py", line 697, in main
   rv = self.invoke(ctx)
 File "/opt/twindb-backup/embedded/lib/python2.7/site-packages/click-6.7-py2.7.egg/click/core.py", line 1066, in invoke
   return _process_result(sub_ctx.command.invoke(sub_ctx))
 File "/opt/twindb-backup/embedded/lib/python2.7/site-packages/click-6.7-py2.7.egg/click/core.py", line 895, in invoke
   return ctx.invoke(self.callback, **ctx.params)
 File "/opt/twindb-backup/embedded/lib/python2.7/site-packages/click-6.7-py2.7.egg/click/core.py", line 535, in invoke
   return callback(*args, **kwargs)
 File "/opt/twindb-backup/embedded/lib/python2.7/site-packages/click-6.7-py2.7.egg/click/decorators.py", line 64, in new_func
   return ctx.invoke(f, obj, *args[1:], **kwargs)
 File "/opt/twindb-backup/embedded/lib/python2.7/site-packages/click-6.7-py2.7.egg/click/core.py", line 535, in invoke
   return callback(*args, **kwargs)
 File "/opt/twindb-backup/embedded/lib/python2.7/site-packages/twindb_backup-2.14.1-py2.7.egg/twindb_backup/cli.py", line 72, in backup
   run_backup_job(cfg, run_type)
 File "/opt/twindb-backup/embedded/lib/python2.7/site-packages/twindb_backup-2.14.1-py2.7.egg/twindb_backup/backup.py", line 254, in run_backup_job
   backup_everything(run_type, cfg)
 File "/opt/twindb-backup/embedded/lib/python2.7/site-packages/twindb_backup-2.14.1-py2.7.egg/twindb_backup/backup.py", line 207, in backup_everything
   backup_mysql(run_type, config)
 File "/opt/twindb-backup/embedded/lib/python2.7/site-packages/twindb_backup-2.14.1-py2.7.egg/twindb_backup/backup.py", line 148, in backup_mysql
   dst.status(status)
 File "/opt/twindb-backup/embedded/lib/python2.7/site-packages/twindb_backup-2.14.1-py2.7.egg/twindb_backup/destination/base_destination.py", line 122, in status
   return self._write_status(status)
 File "/opt/twindb-backup/embedded/lib/python2.7/site-packages/twindb_backup-2.14.1-py2.7.egg/twindb_backup/destination/ssh.py", line 137, in _write_status
   proc = Popen(cmd)
 File "/opt/twindb-backup/embedded/lib/python2.7/subprocess.py", line 710, in __init__
   errread, errwrite)
 File "/opt/twindb-backup/embedded/lib/python2.7/subprocess.py", line 1335, in _execute_child
   raise child_exception
OSError: [Errno 7] Argument list too long

twindb-backup RPM fails to delete cron config

TwinDB Backup version: 2.15.3

Description

During uninstall the twindb-backup RPM fails to delete a cron config.

What I Did

# yum remove twindb-backup-2.15.3-1.x86_64
...
Running transaction
  Erasing    : twindb-backup-2.15.3-1.x86_64                                                                                                                                  1/1
warning: file /opt/twindb-backup/twindb-backup.cron: remove failed: No such file or directory
twindb-backup has been uninstalled!
  Verifying  : twindb-backup-2.15.3-1.x86_64

Local retention policy is not respected

TwinDB Backup version: 2.15.0

Description

TwinDB Backup can keep few latest copies on the same server it takes backups from. The feature is enabled by keep_local_path options.

The local copies have their own retention policy managed by [retention_local] section in /etc/twindb/twindb-backup.conf.

There is a report from a user that the local retention policy in version 2.15.0 is not respected and the tool fills local disk with backup copies.

A workaround to this is to disable keep_local_path.

We need to reproduce the bug, write a unit test (possibly integration test as well) and fix the bug.

script doesn't create a host directory

TwinDB Backup version: 2.9.3

Description

When storing a local copy twindb-backup should create directory for a host it takes backup from.

Let's say keep_local_copy=/var/log/mysql/backup, hostname is ip-10-10-10-10.
If /var/log/mysql/backup/ip-10-10-10-10 doesn't exist the script will crash with traceback:

Traceback (most recent call last):
File "/usr/bin/twindb-backup", line 9, in <module>
load_entry_point('twindb-backup==2.9.3', 'console_scripts', 'twindb-backup')()
File "/opt/twindb-backup/embedded/lib/python2.7/site-packages/click-6.7-py2.7.egg/click/core.py", line 722, in _call_
return self.main(*args, **kwargs)
File "/opt/twindb-backup/embedded/lib/python2.7/site-packages/click-6.7-py2.7.egg/click/core.py", line 697, in main
rv = self.invoke(ctx)
File "/opt/twindb-backup/embedded/lib/python2.7/site-packages/click-6.7-py2.7.egg/click/core.py", line 1066, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/opt/twindb-backup/embedded/lib/python2.7/site-packages/click-6.7-py2.7.egg/click/core.py", line 895, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/opt/twindb-backup/embedded/lib/python2.7/site-packages/click-6.7-py2.7.egg/click/core.py", line 535, in invoke
return callback(*args, **kwargs)
File "/opt/twindb-backup/embedded/lib/python2.7/site-packages/click-6.7-py2.7.egg/click/decorators.py", line 64, in new_func
return ctx.invoke(f, obj, *args[1:], **kwargs)
File "/opt/twindb-backup/embedded/lib/python2.7/site-packages/click-6.7-py2.7.egg/click/core.py", line 535, in invoke
return callback(*args, **kwargs)
File "/opt/twindb-backup/embedded/lib/python2.7/site-packages/twindb_backup-2.9.3-py2.7.egg/twindb_backup/cli.py", line 53, in backup
run_backup_job(cfg, run_type)
File "/opt/twindb-backup/embedded/lib/python2.7/site-packages/twindb_backup-2.9.3-py2.7.egg/twindb_backup/backup.py", line 140, in run_backup_job
raise err
IOError: [Errno 2] No such file or directory: '/var/log/mysql/backup//ip-10-10-10-10/status'

orphaned files after uninstalling twindb-backup

TwinDB Backup version: 2.10.0
Operating System: Debian/Ubuntu

Description

During upgrade or after uninstalling the package there are orphaned files left

$ sudo dpkg -i /tmp/twindb-backup_2.10.0-1_amd64.deb
(Reading database ... 86480 files and directories currently installed.)
Preparing to unpack .../twindb-backup_2.10.0-1_amd64.deb ...
Unpacking twindb-backup (2.10.0-1) over (2.8.2-1) ...
twindb-backup has been uninstalled!
dpkg: warning: unable to delete old directory '/opt/twindb-backup/embedded/lib/python2.7/site-packages/MySQLdb/constants': Directory not empty
dpkg: warning: unable to delete old directory '/opt/twindb-backup/embedded/lib/python2.7/site-packages/MySQLdb': Directory not empty
dpkg: warning: unable to delete old directory '/opt/twindb-backup/embedded/lib/python2.7/site-packages/botocore-1.4.93-py2.7.egg/botocore/docs/bcdoc': Directory not empty
dpkg: warning: unable to delete old directory '/opt/twindb-backup/embedded/lib/python2.7/site-packages/botocore-1.4.93-py2.7.egg/botocore/docs': Directory not empty
dpkg: warning: unable to delete old directory '/opt/twindb-backup/embedded/lib/python2.7/site-packages/botocore-1.4.93-py2.7.egg/botocore/vendored/requests/packages/chardet': Directory not empty
dpkg: warning: unable to delete old directory '/opt/twindb-backup/embedded/lib/python2.7/site-packages/botocore-1.4.93-py2.7.egg/botocore/vendored/requests/packages/urllib3/packages/ssl_match_hostname': Directory not empty
dpkg: warning: unable to delete old directory '/opt/twindb-backup/embedded/lib/python2.7/site-packages/botocore-1.4.93-py2.7.egg/botocore/vendored/requests/packages/urllib3/packages': Directory not empty
dpkg: warning: unable to delete old directory '/opt/twindb-backup/embedded/lib/python2.7/site-packages/botocore-1.4.93-py2.7.egg/botocore/vendored/requests/packages/urllib3/util': Directory not empty
dpkg: warning: unable to delete old directory '/opt/twindb-backup/embedded/lib/python2.7/site-packages/botocore-1.4.93-py2.7.egg/botocore/vendored/requests/packages/urllib3/contrib': Directory not empty
dpkg: warning: unable to delete old directory '/opt/twindb-backup/embedded/lib/python2.7/site-packages/botocore-1.4.93-py2.7.egg/botocore/vendored/requests/packages/urllib3': Directory not empty
dpkg: warning: unable to delete old directory '/opt/twindb-backup/embedded/lib/python2.7/site-packages/botocore-1.4.93-py2.7.egg/botocore/vendored/requests/packages': Directory not empty
dpkg: warning: unable to delete old directory '/opt/twindb-backup/embedded/lib/python2.7/site-packages/botocore-1.4.93-py2.7.egg/botocore/vendored/requests': Directory not empty
dpkg: warning: unable to delete old directory '/opt/twindb-backup/embedded/lib/python2.7/site-packages/botocore-1.4.93-py2.7.egg/botocore/vendored': Directory not empty
dpkg: warning: unable to delete old directory '/opt/twindb-backup/embedded/lib/python2.7/site-packages/botocore-1.4.93-py2.7.egg/botocore': Directory not empty
dpkg: warning: unable to delete old directory '/opt/twindb-backup/embedded/lib/python2.7/site-packages/botocore-1.4.93-py2.7.egg': Directory not empty
dpkg: warning: unable to delete old directory '/opt/twindb-backup/embedded/lib/python2.7/site-packages/psutil-5.0.1-py2.7-linux-x86_64.egg/psutil': Directory not empty
dpkg: warning: unable to delete old directory '/opt/twindb-backup/embedded/lib/python2.7/site-packages/psutil-5.0.1-py2.7-linux-x86_64.egg': Directory not empty
dpkg: warning: unable to delete old directory '/opt/twindb-backup/embedded/lib/python2.7/site-packages/twindb_backup-2.8.2-py2.7.egg/twindb_backup/destination': Directory not empty
dpkg: warning: unable to delete old directory '/opt/twindb-backup/embedded/lib/python2.7/site-packages/twindb_backup-2.8.2-py2.7.egg/twindb_backup/source': Directory not empty
dpkg: warning: unable to delete old directory '/opt/twindb-backup/embedded/lib/python2.7/site-packages/twindb_backup-2.8.2-py2.7.egg/twindb_backup': Directory not empty
dpkg: warning: unable to delete old directory '/opt/twindb-backup/embedded/lib/python2.7/site-packages/twindb_backup-2.8.2-py2.7.egg': Directory not empty
dpkg: warning: unable to delete old directory '/opt/twindb-backup/embedded/lib/python2.7/site-packages/boto3-1.4.3-py2.7.egg/boto3/docs': Directory not empty
dpkg: warning: unable to delete old directory '/opt/twindb-backup/embedded/lib/python2.7/site-packages/boto3-1.4.3-py2.7.egg/boto3/resources': Directory not empty
dpkg: warning: unable to delete old directory '/opt/twindb-backup/embedded/lib/python2.7/site-packages/boto3-1.4.3-py2.7.egg/boto3': Directory not empty
dpkg: warning: unable to delete old directory '/opt/twindb-backup/embedded/lib/python2.7/site-packages/boto3-1.4.3-py2.7.egg': Directory not empty
Setting up twindb-backup (2.10.0-1) ...
Skipping setting up of /etc/twindb/twindb-backup.cfg as it already exists.
Skipping setting up of /etc/cron.d/twindb-backup as it already exists.
Thank you for installing twindb-backup!

Backup fails: pymysql.err.InternalError: (1381, u'You are not using binary logging')

TwinDB Backup version: 2.17.0
Python version: 2.7
Operating System: CentOS 7.6

Description

Doing a backup: twindb-backup backup daily

What I Did

# twindb-backup backup daily
Traceback (most recent call last):
  File "/bin/twindb-backup", line 9, in <module>
    load_entry_point('twindb-backup==2.17.0', 'console_scripts', 'twindb-backup')()
  File "/opt/twindb-backup/embedded/lib/python2.7/site-packages/click-6.7-py2.7.egg/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
File "/opt/twindb-backup/embedded/lib/python2.7/site-packages/click-6.7-py2.7.egg/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/opt/twindb-backup/embedded/lib/python2.7/site-packages/click-6.7-py2.7.egg/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/opt/twindb-backup/embedded/lib/python2.7/site-packages/click-6.7-py2.7.egg/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/opt/twindb-backup/embedded/lib/python2.7/site-packages/click-6.7-py2.7.egg/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/opt/twindb-backup/embedded/lib/python2.7/site-packages/click-6.7-py2.7.egg/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/opt/twindb-backup/embedded/lib/python2.7/site-packages/twindb_backup-2.17.0-py2.7.egg/twindb_backup/cli.py", line 130, in backup
    binlogs_only=binlogs_only
  File "/opt/twindb-backup/embedded/lib/python2.7/site-packages/twindb_backup-2.17.0-py2.7.egg/twindb_backup/backup.py", line 367, in run_backup_job
    binlogs_only=binlogs_only
  File "/opt/twindb-backup/embedded/lib/python2.7/site-packages/twindb_backup-2.17.0-py2.7.egg/twindb_backup/backup.py", line 307, in backup_everything
    backup_binlogs(run_type, twindb_config)
  File "/opt/twindb-backup/embedded/lib/python2.7/site-packages/twindb_backup-2.17.0-py2.7.egg/twindb_backup/backup.py", line 216, in backup_binlogs
    last_binlog=last_copy.name if last_copy else None
  File "/opt/twindb-backup/embedded/lib/python2.7/site-packages/twindb_backup-2.17.0-py2.7.egg/twindb_backup/backup.py", line 268, in binlogs_to_backup
    cursor.execute("SHOW BINARY LOGS")
  File "/opt/twindb-backup/embedded/lib/python2.7/site-packages/PyMySQL-0.9.3-py2.7.egg/pymysql/cursors.py", line 170, in execute
  File "/opt/twindb-backup/embedded/lib/python2.7/site-packages/PyMySQL-0.9.3-py2.7.egg/pymysql/cursors.py", line 328, in _query
  File "/opt/twindb-backup/embedded/lib/python2.7/site-packages/PyMySQL-0.9.3-py2.7.egg/pymysql/connections.py", line 517, in query
  File "/opt/twindb-backup/embedded/lib/python2.7/site-packages/PyMySQL-0.9.3-py2.7.egg/pymysql/connections.py", line 732, in _read_query_result
  File "/opt/twindb-backup/embedded/lib/python2.7/site-packages/PyMySQL-0.9.3-py2.7.egg/pymysql/connections.py", line 1075, in read
  File "/opt/twindb-backup/embedded/lib/python2.7/site-packages/PyMySQL-0.9.3-py2.7.egg/pymysql/connections.py", line 684, in _read_packet
  File "/opt/twindb-backup/embedded/lib/python2.7/site-packages/PyMySQL-0.9.3-py2.7.egg/pymysql/protocol.py", line 220, in check_error
  File "/opt/twindb-backup/embedded/lib/python2.7/site-packages/PyMySQL-0.9.3-py2.7.egg/pymysql/err.py", line 109, in raise_mysql_exception
pymysql.err.InternalError: (1381, u'You are not using binary logging')

fix broken link in installation.html

TwinDB Backup version: 2.15.0

Description

A documentation page http://twindb-backup.readthedocs.io/en/master/installation.html has a broken link https://github.com/twindb/twindb_backup

Error: no such option: --version

TwinDB Backup version: Unknown :)

Description

Command line --version option should be present.

What I Did

# twindb-backup --version
Error: no such option: --version

Enhancement - Implement Azure Blob as Destination

TwinDB Backup version: 2.x

Description

Please implement Azure Blob storage as an alternative to AWS S3 destination.

No RPMs available

TwinDB Backup version: n/a
Python version: 2.6.6
Operating System: Oracle Linux 6.8

Description

I was trying to install twins

What I Did

After the install process ran, I ran yum install twindb-backup (per the docs)

Here is the output:

# yum install twindb-backup
Loaded plugins: priorities, security, ulninfo
Setting up Install Process
536 packages excluded due to repository priority protections
No package twindb-backup available.
Error: Nothing to do

Is anyone having this issue? Seems that the RPM repo is empty.

Integration test test-integration-backup-ssh fails on vagrant.

TwinDB Backup version: develop branch, commit 7f89333

Description

Integration test test-integration-backup-ssh fails on vagrant.

What I Did

[root@master1 twindb_backup]# make test-integration-backup-ssh
py.test -xsv tests/integration/backup/ssh
========================================================================================== test session starts ===========================================================================================
platform linux2 -- Python 2.7.5, pytest-3.2.3, py-1.4.34, pluggy-0.4.0 -- /usr/bin/python2
cachedir: .cache
rootdir: /twindb_backup, inifile:
plugins: cov-2.5.1
collected 2 items

tests/integration/backup/ssh/test_backup.py::test_backup 2017-11-30 17:26:35,831: INFO: conftest.container_network():75: Created subnet 172.16.0.0/16
2017-11-30 17:26:35,832: DEBUG: conftest.container_network():76: {u'Warning': u'', u'Id': u'e7fb42258171763826220923b33b614247c8729dbd71020be2a8a72eb9fd6304'}
2017-11-30 17:27:05,322: DEBUG: conftest.get_container():108: {'EndpointsConfig': {'test_network': {'IPAMConfig': {'IPv4Address': '172.16.3.1'}}}}
2017-11-30 17:27:05,625: INFO: conftest.get_container():120: Created container {'ip': '172.16.3.1', u'Id': u'7839c484bd203b2ae3f43fe50cbfc7538cb003c136f53f7a3772c9a85cbf8747', u'Warnings': None}
2017-11-30 17:27:06,337: INFO: conftest.get_container():123: Started {'ip': '172.16.3.1', u'Id': u'7839c484bd203b2ae3f43fe50cbfc7538cb003c136f53f7a3772c9a85cbf8747', u'Warnings': None}
PASSED
tests/integration/backup/ssh/test_restore.py::test_restore CMD : twindb-backup --debug --config /tmp/pytest-of-root/pytest-2/test_restore0/twindb-backup.cfg ls | grep etc | grep tmp | awk -F/ '{ print $NF}' | sort | tail -1
2017-11-30 18:27:25,797: DEBUG: ls.list_available_backups():26: Running find /var/backup/local -type f
2017-11-30 18:27:25,802: DEBUG: configuration.get_destination():28: Destination in the config ssh
Traceback (most recent call last):
  File "/bin/twindb-backup", line 11, in <module>
    load_entry_point('twindb-backup', 'console_scripts', 'twindb-backup')()
  File "/usr/lib/python2.7/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/usr/lib/python2.7/site-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/lib/python2.7/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/lib/python2.7/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/click/decorators.py", line 64, in new_func
    return ctx.invoke(f, obj, *args[1:], **kwargs)
  File "/usr/lib/python2.7/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/twindb_backup/twindb_backup/cli.py", line 104, in list_backups
    list_available_backups(cfg)
  File "/twindb_backup/twindb_backup/ls.py", line 34, in list_available_backups
    for copy in dst.find_files(dst.remote_path, run_type):
  File "/twindb_backup/twindb_backup/destination/ssh.py", line 136, in find_files
    with self._ssh_client.get_remote_handlers(cmd) as (_, cout, _):
  File "/usr/lib64/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/twindb_backup/twindb_backup/ssh/client.py", line 104, in get_remote_handlers
    with self._shell() as shell:
  File "/usr/lib64/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/twindb_backup/twindb_backup/ssh/client.py", line 55, in _shell
    raise SshClientException(err)
twindb_backup.ssh.exceptions.SshClientException: [Errno None] Unable to connect to port 22 on 172.16.3.1
CMD : twindb-backup --debug --config /tmp/pytest-of-root/pytest-2/test_restore0/twindb-backup.cfg ls | grep /tmp/backup | grep  | head -1
Usage: grep [OPTION]... PATTERN [FILE]...
Try 'grep --help' for more information.
2017-11-30 18:27:28,439: DEBUG: ls.list_available_backups():26: Running find /var/backup/local -type f
2017-11-30 18:27:28,444: DEBUG: configuration.get_destination():28: Destination in the config ssh
Traceback (most recent call last):
  File "/bin/twindb-backup", line 11, in <module>
    load_entry_point('twindb-backup', 'console_scripts', 'twindb-backup')()
  File "/usr/lib/python2.7/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/usr/lib/python2.7/site-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/lib/python2.7/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/lib/python2.7/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/click/decorators.py", line 64, in new_func
    return ctx.invoke(f, obj, *args[1:], **kwargs)
  File "/usr/lib/python2.7/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/twindb_backup/twindb_backup/cli.py", line 104, in list_backups
    list_available_backups(cfg)
  File "/twindb_backup/twindb_backup/ls.py", line 34, in list_available_backups
    for copy in dst.find_files(dst.remote_path, run_type):
  File "/twindb_backup/twindb_backup/destination/ssh.py", line 136, in find_files
    with self._ssh_client.get_remote_handlers(cmd) as (_, cout, _):
  File "/usr/lib64/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/twindb_backup/twindb_backup/ssh/client.py", line 104, in get_remote_handlers
    with self._shell() as shell:
  File "/usr/lib64/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/twindb_backup/twindb_backup/ssh/client.py", line 55, in _shell
    raise SshClientException(err)
twindb_backup.ssh.exceptions.SshClientException: [Errno None] Unable to connect to port 22 on 172.16.3.1
Url:

Command output:
2017-11-30 18:27:30,024: INFO: cli.restore_file():180: No backup copy specified. Choose one from below:
2017-11-30 18:27:30,024: INFO: ls.list_available_backups():22: Local copies:
2017-11-30 18:27:30,095: INFO: ls.list_available_backups():33: hourly copies:

[Errno None] Unable to connect to port 22 on 172.16.3.1
(<class 'twindb_backup.ssh.exceptions.SshClientException'>, SshClientException(NoValidConnectionsError(None, 'Unable to connect to port 22 on 172.16.3.1'),), <traceback object at 0x3eec6c8>)
FAILED

================================================================================================ FAILURES ================================================================================================
______________________________________________________________________________________________ test_restore ______________________________________________________________________________________________

backup_server = {'Id': 'c7347718b4496d64fc38d2221956657bbc82d6bf75d0ce1e1869024acedda739', 'Warnings': None, 'ip': '172.16.3.1'}
config_content_ssh = '\n[source]\nbackup_dirs={BACKUP_DIR}\nbackup_mysql=no\n\n[destination]\nbackup_destination=ssh\nkeep_local_path=/var/...\n\n[intervals]\n\n# Run intervals\n\nrun_hourly=yes\nrun_daily=yes\nrun_weekly=yes\nrun_monthly=yes\nrun_yearly=yes\n'
tmpdir = local('/tmp/pytest-of-root/pytest-2/test_restore0')

    @pytest.mark.timeout(600)
    def test_restore(backup_server, config_content_ssh, tmpdir):

        my_cnf = tmpdir.join('.my.cnf')
        my_cnf.write("""
    [client]
    user=dba
    password=qwerty
    """)
        config = tmpdir.join('twindb-backup.cfg')
        id_rsa = tmpdir.join('id_rsa')
        id_rsa.write("""-----BEGIN RSA PRIVATE KEY-----
    MIIEoAIBAAKCAQEAyXxAjPShNGAedbaEtltFI6A7RlsyI+4evxTq6uQrgbJ6Hm+p
    HBXshXQYXDyVjvytaM+6GKF+r+6+C+6Wc5Xz4lLO/ZiSCdPbyEgqw1JoHrgPNpc6
    wmCtjJExxjzvpwSVgbZg3xOdqW1y+TyqeUkXEg/Lm4VZhN1Q/KyGCgBlWuAXoOYR
    GhaNWqcnr/Wn5YzVHAx2yJNrurtKLVYVMIkGcN/6OUaPpWqKZLaXiK/28PSZ5GdT
    DmxRg4W0pdyGEYQndpPlpLF4w5gNUEhVZM8hWVE29+DIW3XXVYGYchxmkhU7wrGx
    xZR+k5AT+7g8VspVS8zNMXM9Z27w55EQuluNMQIBIwKCAQAzz35QIaXLo7APo/Y9
    hS8JKTPQQ1YJPTsbMUO4vlRUjPrUoF6vc1oTsCOFbqoddCyXS1u9MNdvEYFThn51
    flSn6WhtGJqU0BPxrChA2q0PNqTThfkqqyVQCBQdCFrhzfqPEaPhl1RtZUlzSh01
    IWxVGgEn/bfu9xTTQk5aV9+MZQ2XKe4BGzpOZMI/B7ivRCcthEwMTx92opr52bre
    4t7DahVLN/2Wu4lxajDzCaKXpjMuL76lFov0mZZN7S8whH5xSx1tpapHqsCAwfLL
    k49lDdR8aN6oqoeK0e9w//McIaKxN2FVxD4bcuXiQTjihx+QwQOLmlHSRDKhTsYg
    4Q5bAoGBAOgVZM2eqC8hNl5UH//uuxOeBKqwz7L/FtGemNr9m0XG8N9yE/K7A5iX
    6EDvDyVI51IlIXdxfK8re5yxfbJ4YevenwdEZZ2O8YRrVByJ53PV9CcVeWL4p6f/
    I56sYyDfXcnDTEOVYY0mCfYUfUcSb1ExpuIU4RvuQJg6tvbdxD9FAoGBAN4/pVCT
    krRd6PJmt6Dbc2IF6N09OrAnLB3fivGztF5cp+RpyqZK4ve+akLoe1laTg7vNtnF
    l/PZtM9v/VT45hb70MFEHO+sKvGa5Yimxkb6YCriJOcLxTysSgFHKz7v+8BqqoHi
    qY4fORGwPVDv28I8jKRvcuNHendV/Rdcuk79AoGAd1t1q5NscAJzu3u4r4IXEWc1
    mZzClpHROJq1AujTgviZInUu1JqxZGthgHrx2KkmggR3nIOB86/2bdefut7TRhq4
    L5+Et24VzxKgSTD6sJnrR0zfV3iQvMxbdizFRBsaSoGyMWLEdHn2fo4xzMem9o6Q
    VwNsdMOsMQhA1rsxuiMCgYBr8wcnIxte68jqxC1OIXKOsmnKi3RG7nSDidXF2vE1
    JbCiJMGD+Hzeu5KyyLDw4rgzI7uOWKjkJ+obnMuBCy3t6AZPPlcylXPxsaKwFn2Q
    MHfaUJWUyzPqRQ4AnukekdINAJv18cAR1Kaw0fHle9Ej1ERP3lxfw6HiMRSHsLJD
    nwKBgCIXVhXCDaXOOn8M4ky6k27bnGJrTkrRjHaq4qWiQhzizOBTb+7MjCrJIV28
    8knW8+YtEOfl5R053SKQgVsmRjjDfvCirGgqC4kSAN4A6MD+GNVXZVUUjAUBVUbU
    8Wt4BxW6kFA7+Su7n8o4DxCqhZYmK9ZUhNjE+uUhxJCJaGr4
    -----END RSA PRIVATE KEY-----
    """)

        content = config_content_ssh.format(
            PRIVATE_KEY=str(id_rsa),
            BACKUP_DIR="/etc",
            HOST_IP=backup_server['ip']
        )
        config.write(content)
        runner = CliRunner()
        result = runner.invoke(main, [
            '--debug',
            '--config', str(config),
            'backup', 'daily'
        ])

        if result.exit_code != 0:
            print('Command output:')
            print(result.output)
            print(result.exception)
            print(result.exc_info)
        assert result.exit_code == 0

        cmd = 'twindb-backup --debug --config %s ls | grep etc | grep tmp ' \
              '| awk -F/ \'{ print $NF}\' | sort | tail -1' % str(config)
        print('CMD : %s' % cmd)
        basename = check_output(args=cmd, shell=True).rstrip()
        cmd = 'twindb-backup --debug --config %s ls ' \
              '| grep /tmp/backup | grep %s | head -1' % (str(config), basename)
        print('CMD : %s' %cmd)
        url = check_output(cmd, shell=True).rstrip()
        print('Url:')
        print(url)
        restore_dir = tmpdir.mkdir('restore')
        runner = CliRunner()
        result = runner.invoke(main,
                               ['--config', str(config),
                                'restore', 'file', url,
                                "--dst", str(restore_dir)]
                               )
        if result.exit_code != 0:
            print('Command output:')
            print(result.output)
            print(result.exception)
            print(result.exc_info)
>       assert result.exit_code == 0
E       AssertionError: assert -1 == 0
E        +  where -1 = <Result SshClientException(NoValidConnectionsError(None, 'Unable to connect to port 22 on 172.16.3.1'),)>.exit_code

tests/integration/backup/ssh/test_restore.py:91: AssertionError
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
================================================================================= 1 failed, 1 passed in 3659.69 seconds ==================================================================================
make: *** [test-integration-backup-ssh] Error 2
[root@master1 twindb_backup]#

twindb-backup clone mysql shouldn't stop MySQL on destination

TwinDB Backup version: 2.14.1

Description

clone mysql attempts to stop MySQL on the destination. It shouldn't.
Instead it should check if MySQL is running and raise an exception if it is.

Enhancement: Allow local-only (on the filesystem) backup destination

I would like to use this software locally only, but am not able to deploy SSH keys for connecting back to localhost in a particular scenario.

Since it's already implemented to have a local copy saved, would it be possible to enable a local destination as a primary location (rather than s3 or ssh)?

I did start hacking on this, and hope to submit a PR, but wanted to open it up with the off-chance that a better coder might tackle it before I can.

Integration test fails due to intermittent S3 error

TwinDB Backup version: 50fe201

Description

Inetrgration test in travis failed bc S3 raised an exception. See log

https://travis-ci.org/twindb/backup/jobs/504154396

AttributeError: 'int' object has no attribute 'lower'

TwinDB Backup version: 2.14.0
Python version: 2.7.12
Operating System: Ubnutu Server 16.04.3 LTS

Description

There is a small bug in the source:

File "/opt/twindb-backup/embedded/lib/python2.7/site-packages/twindb_backup-2.14.0-py2.7.egg/twindb_backup/source/mysql_source.py", line 447

I get a python error:

AttributeError: 'int' object has no attribute 'lower'

What I Did

Lines 446 & 447:

                   return (str(row['wsrep_on']).lower() == "1" or
                            row['wsrep_on'].lower() == 'on')

Should be fixed by doing this:

                   return (str(row['wsrep_on']).lower() == "1" or
                            str(row['wsrep_on']).lower() == 'on')

Can you please update the code with this change? Otherwise, I have to edit this file everytime it updates.

Where/How should I set if backup type should be full or incremental?

TwinDB Backup version: latest
Python version: -
Operating System: -

Description

Hi guys,

I'm reading the docs here: https://twindb-backup.readthedocs.io/
I see that full/incremental types of backups are supported, but I can't find in the docs where and how should I set such type.

Thanks.
Best, Michael Rikmas

mysql clone should understand hostnames

TwinDB Backup version: 2.14.1

Description

twindb-backup clone mysql doesn't work with hostnames.

What I Did

twindb-backup --config ~/.twindb-backup.cfg clone mysql hostname1 hostname2
2017-10-13 11:56:28,003: ERROR: clone.clone_mysql():85: Failed to stop MySQL on <twindb_backup.destination.ssh.Ssh object at 0x7f5ae5a30d50>
Traceback (most recent call last):
  File "/home/okuzminskyi/.virtualenvs/backup/bin/twindb-backup", line 11, in <module>
    load_entry_point('twindb-backup', 'console_scripts', 'twindb-backup')()
  File "/home/okuzminskyi/.virtualenvs/backup/lib/python2.7/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/home/okuzminskyi/.virtualenvs/backup/lib/python2.7/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/home/okuzminskyi/.virtualenvs/backup/lib/python2.7/site-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/okuzminskyi/.virtualenvs/backup/lib/python2.7/site-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/okuzminskyi/.virtualenvs/backup/lib/python2.7/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/okuzminskyi/.virtualenvs/backup/lib/python2.7/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/home/okuzminskyi/.virtualenvs/backup/lib/python2.7/site-packages/click/decorators.py", line 64, in new_func
    return ctx.invoke(f, obj, *args[1:], **kwargs)
  File "/home/okuzminskyi/.virtualenvs/backup/lib/python2.7/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/home/okuzminskyi/src/backup/twindb_backup/cli.py", line 231, in clone_mysql_backup
    netcat_port=netcat_port)
  File "/home/okuzminskyi/src/backup/twindb_backup/clone.py", line 107, in clone_mysql
    src.clone_config(dst)
  File "/home/okuzminskyi/src/backup/twindb_backup/source/remote_mysql_source.py", line 66, in clone_config
    self._save_cfg(dst, cfg_path)
  File "/home/okuzminskyi/src/backup/twindb_backup/source/remote_mysql_source.py", line 71, in _save_cfg
    server_id = struct.unpack("!I", socket.inet_aton(dst.host))[0]
socket.error: illegal IP address string passed to inet_aton

get_stream works invalid via ssh

[root@stoage ~]# twindb-backup --debug verify mysql /backup_path/hostname/hourly/mysql/mysql-2018-09-02_13_00_03.xbstream.gz --dst /tmp/verify
2018-09-02 11:46:06,165: DEBUG: cli.verify():235: Restore: <ConfigParser.ConfigParser instance at 0x7f873e8ffc68>
2018-09-02 11:46:06,165: DEBUG: cli.verify_mysql():250: mysql: <ConfigParser.ConfigParser instance at 0x7f873e8ffc68>
2018-09-02 11:46:06,166: INFO: restore.restore_from_mysql():307: Restoring /backup_path/hostname/hourly/mysql/mysql-2018-09-02_13_00_03.xbstream.gz in /tmp/verify
2018-09-02 11:46:06,167: DEBUG: configuration.get_destination():29: Destination in the config ssh
2018-09-02 11:46:06,299: DEBUG: client.execute():110: Executing command: bash -c 'if test -s /backupbgpercona/nbxp003.nbxp.local/status; then echo exists; else echo not_exists; fi'
2018-09-02 11:46:06,403: DEBUG: client.get_remote_handlers():163: Try to get remote handlers: cat /backupbgpercona/nbxp003.nbxp.local/status
2018-09-02 11:46:06,411: DEBUG: mysql_status.candidate_parent():127: Looking a parent candidate for hourly run
2018-09-02 11:46:06,411: DEBUG: mysql_status.candidate_parent():133: Checking 17 hourly copies
2018-09-02 11:46:06,411: DEBUG: mysql_status.candidate_parent():133: Checking 2 daily copies
2018-09-02 11:46:06,411: DEBUG: mysql_status.candidate_parent():138: Found parent MySQLCopy(hostname/daily/mysql/mysql-2018-09-02_00_00_03.xbstream.gz)
2018-09-02 11:46:06,411: DEBUG: restore.restore_from_mysql():367: Full parent copy is hostname/daily/mysql/mysql-2018-09-02_00_00_03.xbstream.gz
2018-09-02 11:46:06,412: DEBUG: restore.restore_from_mysql_full():73: Not decrypting the stream
2018-09-02 11:46:06,413: DEBUG: base._revert_stream():50: Running gunzip -c
2018-09-02 11:46:06,415: DEBUG: restore._extract_xbstream():117: Running /opt/twindb-backup/embedded/bin/xbstream -x
2018-09-02 11:46:06,415: DEBUG: restore._extract_xbstream():118: Working directory: /tmp/verify
2018-09-02 11:46:06,416: DEBUG: restore._extract_xbstream():119: Xbstream binary: /opt/twindb-backup/embedded/bin/xbstream
2018-09-02 11:46:06,549: DEBUG: ssh._write_to_pipe():131: Executing cat hostname/daily/mysql/mysql-2018-09-02_00_00_03.xbstream.gz
2018-09-02 11:46:06,553: DEBUG: ssh._write_to_pipe():137: closing channel
2018-09-02 11:46:06,556: ERROR: base._revert_stream():59: gunzip -c exited with non-zero code.
2018-09-02 11:46:06,556: ERROR: base._revert_stream():60:
gzip: stdin: unexpected end of file

Line:
2018-09-02 11:46:06,549: DEBUG: ssh._write_to_pipe():131: Executing cat hostname/daily/mysql/mysql-2018-09-02_00_00_03.xbstream.gz

Expected:
/backup_path/hostname/daily/mysql/mysql-2018-09-02_00_00_03.xbstream.gz

2016-11-02 15:43:47,981: DEBUG: backup.disable_wsrep_desync():110: Timeout expired. Disabling wsrep_desync

TwinDB Backup version: 2.1.4

Description

the script waits for wsrep_local_recv_queue go to zero, but even though it touches zero the script doesn't exit until it times out.

What I Did

# twindb-backup --debug backup hourly
...
2016-11-02 15:43:47,981: DEBUG: backup.disable_wsrep_desync():110: Timeout expired. Disabling wsrep_desync
...

s3.delete() mistakenly joining duplicated bucket name from mysql_source.apply_retention_policy()

On a MySQL 5.7.23 host, while performing the hourly backup to an s3 bucket, twindb-backup hs been reporting "WARNING: mysql_source.apply_retention_policy():232: Copy **redacted host**/hourly/mysql/.... not found

Looking into this further, it appears as though this was introduced in the 2.15.8 release, and continues to exhibit the behavior through tonight's 2.15.10 release.

Looking at the debug logs, it looks as though mutliple S3 bucket URLs are being concatenated together, instead of just one into the URL string for the s3.delete call:

from twindb-backup backup --debug hourly output:

deleting s3://bucketname/s3://bucketname

I've verified the presence of the files on the S3 bucket, but the call to s3.delete is mangled:

2018-09-04 06:49:28,081: DEBUG: mysql_source.apply_retention_policy():227: Deleting remote file s3://mysql-backups-bucket/mysql001/hourly/mysql/mysql-2018-09-01_13_00_05.xbstream.gz
2018-09-04 06:49:28,092: DEBUG: s3.delete():271: deleting s3://mysql-backups-bucket/s3://mysql-backups-bucket/mysql001/hourly/mysql/mysql-2018-09-01_13_00_05.xbstream.gz

List ssh client as dependency

TwinDB Backup version:
Python version: 2.8.1
Operating System: centos

Description

Yes, some systems don't have ssh client. As long as twindb-backup depends on it, let's list it as a dependency.

What I Did

# twindb-backup --debug ls
2017-01-14 18:43:23,388: INFO: ls.list_available_backups():11: Local copies:
2017-01-14 18:43:23,388: INFO: ls.list_available_backups():22: hourly copies:
2017-01-14 18:43:23,388: DEBUG: util.get_destination():14: Destination in the config ssh
2017-01-14 18:43:23,389: DEBUG: ssh.find_files():87: Running ssh -l root -o StrictHostKeyChecking=no -o PasswordAuthentication=no -p 22 -i /root/.ssh/id_rsa 127.0.0.1 find /tmp/backup/*/hourly -type f
Traceback (most recent call last):
  File "/usr/bin/twindb-backup", line 9, in <module>
    load_entry_point('twindb-backup==2.8.1', 'console_scripts', 'twindb-backup')()
  File "/opt/twindb-backup/embedded/lib/python2.7/site-packages/click-6.7-py2.7.egg/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/opt/twindb-backup/embedded/lib/python2.7/site-packages/click-6.7-py2.7.egg/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/opt/twindb-backup/embedded/lib/python2.7/site-packages/click-6.7-py2.7.egg/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/opt/twindb-backup/embedded/lib/python2.7/site-packages/click-6.7-py2.7.egg/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/opt/twindb-backup/embedded/lib/python2.7/site-packages/click-6.7-py2.7.egg/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/opt/twindb-backup/embedded/lib/python2.7/site-packages/click-6.7-py2.7.egg/click/decorators.py", line 64, in new_func
    return ctx.invoke(f, obj, *args[1:], **kwargs)
  File "/opt/twindb-backup/embedded/lib/python2.7/site-packages/click-6.7-py2.7.egg/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/opt/twindb-backup/embedded/lib/python2.7/site-packages/twindb_backup-2.8.1-py2.7.egg/twindb_backup/cli.py", line 60, in ls
    list_available_backups(cfg)
  File "/opt/twindb-backup/embedded/lib/python2.7/site-packages/twindb_backup-2.8.1-py2.7.egg/twindb_backup/ls.py", line 25, in list_available_backups
    for copy in dst.find_files(dst.remote_path, run_type):
  File "/opt/twindb-backup/embedded/lib/python2.7/site-packages/twindb_backup-2.8.1-py2.7.egg/twindb_backup/destination/ssh.py", line 88, in find_files
    proc = Popen(cmd, stdout=PIPE, stderr=PIPE)
  File "/opt/twindb-backup/embedded/lib/python2.7/subprocess.py", line 710, in __init__
    errread, errwrite)
  File "/opt/twindb-backup/embedded/lib/python2.7/subprocess.py", line 1335, in _execute_child
    raise child_exception
OSError: [Errno 2] No such file or directory
[root@9bf0f4a6a870 ~]# ssh
bash: ssh: command not found

verification job leaves files in /tmp

TwinDB Backup version: 2.15.1

Description

After twindb-backup runs a verification job it leaves temporary files in /tmp

What I Did

# twindb-backup verify mysql ${backup_copy} --dst /mnt/data/verify
...
[root@ip-10-0-1-151 tmp]# ll
total 8
-rw-r--r--.  1 root root   57 Jan 12 09:00 percona-version-check
drwx------. 11 root root 4096 Jan 12 19:08 tmpas6x1T

Could not find distribution named centos/6

TwinDB Backup version: 2.8.0
Operating System: centos

Description

When travis builds packages from a master branch it uploads them to package cloud.
centos/6 is wrong name. it should be el/6

Verify not working

[root@xxx ~]# twindb-backup verify mysql /path/to/hourly/mysql/mysql-2018-08-29_19_00_03.xbstream.gz --dst /tmp/verifw
2018-08-29 19:22:47,119: INFO: restore.restore_from_mysql():305: Restoring /path/to/hourly/mysql/mysql-2018-08-29_19_00_03.xbstream.gz in /tmp/verifw
Traceback (most recent call last):
  File "/usr/bin/twindb-backup", line 9, in <module>
    load_entry_point('twindb-backup==2.15.7', 'console_scripts', 'twindb-backup')()
  File "/opt/twindb-backup/embedded/lib/python2.7/site-packages/click-6.7-py2.7.egg/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/opt/twindb-backup/embedded/lib/python2.7/site-packages/click-6.7-py2.7.egg/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/opt/twindb-backup/embedded/lib/python2.7/site-packages/click-6.7-py2.7.egg/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/opt/twindb-backup/embedded/lib/python2.7/site-packages/click-6.7-py2.7.egg/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/opt/twindb-backup/embedded/lib/python2.7/site-packages/click-6.7-py2.7.egg/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/opt/twindb-backup/embedded/lib/python2.7/site-packages/click-6.7-py2.7.egg/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/opt/twindb-backup/embedded/lib/python2.7/site-packages/click-6.7-py2.7.egg/click/decorators.py", line 64, in new_func
    return ctx.invoke(f, obj, *args[1:], **kwargs)
  File "/opt/twindb-backup/embedded/lib/python2.7/site-packages/click-6.7-py2.7.egg/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/opt/twindb-backup/embedded/lib/python2.7/site-packages/twindb_backup-2.15.7-py2.7.egg/twindb_backup/cli.py", line 239, in verify_mysql
    print(verify_mysql_backup(cfg, dst, backup_copy, hostname))
  File "/opt/twindb-backup/embedded/lib/python2.7/site-packages/twindb_backup-2.15.7-py2.7.egg/twindb_backup/verify.py", line 48, in verify_mysql_backup
    restore_from_mysql(config, url, dst_path, tmp_dir)
  File "/opt/twindb-backup/embedded/lib/python2.7/site-packages/twindb_backup-2.15.7-py2.7.egg/twindb_backup/restore.py", line 340, in restore_from_mysql
    if status[key].type == "full":
AttributeError: 'NoneType' object has no attribute 'type'

Restore from s3 endpoint fails to restore with "cli.restore_mysql() . ... not found"

Looks like a regression in behavior, where taking the output of twindb-backup ls s3 URLs results in a failed restore:

TwinDB-backup version: 2.15.10
Python version: 2.7.5
Kernel/OS: 3.10.0-862.2.3.el7.x86_64 / CentOS 7.3

root@**redactedhostname**:/tmp/mysql# twindb-backup --debug restore mysql s3://s3-bucket/**redactedhostname**/daily/mysql/mysql-2018-08-30_00_00_08.xbstream.gz
2018-09-06 02:17:09,295: DEBUG: cli.restore():177: restore: <ConfigParser.ConfigParser instance at 0x7f6707a865f0>
2018-09-06 02:17:09,296: DEBUG: cli.restore_mysql():189: mysql: <ConfigParser.ConfigParser instance at 0x7f6707a865f0>
2018-09-06 02:17:09,296: DEBUG: configuration.get_destination():29: Destination in the config s3
2018-09-06 02:17:10,977: ERROR: cli.restore_mysql():210: Copy **redactedhostname**/daily/mysql/mysql-2018-08-30_00_00_08.xbstream.gz not found

root@**redactedhostname**:/tmp/mysql# twindb-backup ls | grep s3://s3-bucket/**redactedhostname**/daily/mysql/mysql-2018-08-30_00_00_08.xbstream.gz
s3://s3-bucket/**redactedhostname**/daily/mysql/mysql-2018-08-30_00_00_08.xbstream.gz

Possibly related to #137 ?

Last known working 2.15.7

twindb / backup Goto Github PK

backup's People

Contributors

Stargazers

Watchers

Forkers

backup's Issues

Description

Description

Description

Description

What I Did

Description

Description

Description

Description

Description

Description

What I Did

Description

Description

Description

What I Did

Workaround

Solution

Description

Description

Description

Description

Description

Description

Description

What I Did

Description

Description

Description

Description

What I Did

Description

Description

What I Did

Description

Description

What I Did

Description

What I Did

Description

Description

Description

What I Did

Description

Description

What I Did

Description

What I Did

Description

What I Did

Description

What I Did

Description

Recommend Projects

Recommend Topics

Recommend Org