qingstor / qsctl Goto Github PK

View Code? Open in Web Editor NEW

22.0 10.0 13.0 1.05 MB

Advanced command line tool for QingStor Object Storage.

License: Apache License 2.0

Makefile 1.95% Go 98.05%

qingstor qsctl cli

qsctl's People

Contributors

Stargazers

Watchers

Forkers

ksellesk hashdatainc fiile dongxifu precisource frostyplanet prnyself gitter-badger abyss-w owarai tianhaoc

qsctl's Issues

qsctl stat should support format like "%Y"

qsutil cp support wildcard *, like bash cp

hope qsutil cp support wildcard *, like bash cp
such as:

qsutil cp *.jpg qs://${mybucket}/images/

test failed with http 400

the error reason is max_requests_exceeded
max_requests_exceeded 当请求创建和删除 bucket 过于频繁时，返回该错误 400

Please reuse the bucket instead of recreate bucket for every test cases.

rsync is a popular utility that is used for synchronizing files.
https://en.wikipedia.org/wiki/Rsync
I'm wondering if we can extend qsctl to synchronize files from a remote server which supports rsync protocol.

One scenario would be: linux repository synchronization

Dependabot can't resolve your Go dependency files

Dependabot can't resolve your Go dependency files.

As a result, Dependabot couldn't update your dependencies.

The error Dependabot encountered was:

go: github.com/Xuanwo/[email protected]: invalid version: unknown revision 25eae5601e66

If you think the above is an error on Dependabot's side please don't hesitate to get in touch - we'll do whatever we can to fix it.

View the update logs.

Add progress bar support

qsctl sync support read include and exclude list from file

It will be used like --exclude-from=FILE and --include-from=FILE in rsync.

test failed due to incorrect import

wangzw:qsctl wangzw$ tox -e py27
GLOB sdist-make: /hashdata/qsctl/setup.py
py27 inst-nodeps: /hashdata/qsctl/.tox/dist/qsctl-1.1.0.zip
py27 installed: You are using pip version 8.1.2, however version 9.0.1 is available.,You should consider upgrading via the 'pip install --upgrade pip' command.,docutils==0.12,funcsigs==1.0.2,future==0.15.2,mock==2.0.0,pbr==1.10.0,py==1.4.31,pytest==3.0.3,PyYAML==3.12,qingcloud-sdk==1.0.11,qsctl==1.1.0,six==1.10.0
py27 runtests: PYTHONHASHSEED='1675402229'
py27 runtests: commands[0] | py.test
======================================== test session starts ========================================
platform darwin -- Python 2.7.10, pytest-3.0.3, py-1.4.31, pluggy-0.4.0
rootdir: /hashdata/qsctl, inifile:
collected 21 items / 9 errors

============================================== ERRORS ===============================================
___________________________ ERROR collecting tests/commands/test_base.py ____________________________
ImportError while importing test module '/hashdata/qsctl/tests/commands/test_base.py'.
Original error message:
'No module named test_data'
Make sure your test modules/packages have valid Python names.
____________________________ ERROR collecting tests/commands/test_cp.py _____________________________
ImportError while importing test module '/hashdata/qsctl/tests/commands/test_cp.py'.
Original error message:
'No module named test_data'
Make sure your test modules/packages have valid Python names.
____________________________ ERROR collecting tests/commands/test_ls.py _____________________________
ImportError while importing test module '/hashdata/qsctl/tests/commands/test_ls.py'.
Original error message:
'No module named test_data'
Make sure your test modules/packages have valid Python names.
____________________________ ERROR collecting tests/commands/test_mb.py _____________________________
ImportError while importing test module '/hashdata/qsctl/tests/commands/test_mb.py'.
Original error message:
'No module named test_data'
Make sure your test modules/packages have valid Python names.
____________________________ ERROR collecting tests/commands/test_mv.py _____________________________
ImportError while importing test module '/hashdata/qsctl/tests/commands/test_mv.py'.
Original error message:
'No module named test_data'
Make sure your test modules/packages have valid Python names.
____________________________ ERROR collecting tests/commands/test_rb.py _____________________________
ImportError while importing test module '/hashdata/qsctl/tests/commands/test_rb.py'.
Original error message:
'No module named test_data'
Make sure your test modules/packages have valid Python names.
____________________________ ERROR collecting tests/commands/test_rm.py _____________________________
ImportError while importing test module '/hashdata/qsctl/tests/commands/test_rm.py'.
Original error message:
'No module named test_data'
Make sure your test modules/packages have valid Python names.
___________________________ ERROR collecting tests/commands/test_sync.py ____________________________
ImportError while importing test module '/hashdata/qsctl/tests/commands/test_sync.py'.
Original error message:
'No module named test_data'
Make sure your test modules/packages have valid Python names.
_________________________ ERROR collecting tests/commands/test_transfer.py __________________________
ImportError while importing test module '/hashdata/qsctl/tests/commands/test_transfer.py'.
Original error message:
'No module named test_data'
Make sure your test modules/packages have valid Python names.
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 9 errors during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
====================================== 9 error in 0.20 seconds ======================================
ERROR: InvocationError: '/hashdata/qsctl/.tox/py27/bin/py.test'
______________________________________________ summary ______________________________________________
ERROR:   py27: commands failed

where is the logfile of qsctl

After installed qsctl and setup the config in the config.yaml as follow：

Then try to use "qsctl ls qs://bucket-name", but failed. The error message is "Error: Please check if bucket exists. If you are trying to create a bucket, please specify the zone name.". Where is the logfile and how can I do the debug?

sys.exit will only exit child thread

https://github.com/yunify/qsctl/blob/master/qingstor/qsctl/commands/base.py#L166

This will only exit child thread, and will lead to main thread exit with 0, which will make user confused.

qsctl stat return object checksum information

It‘s useful for debugging and verifying object integrity

Dependabot can't resolve your Go dependency files

Dependabot can't resolve your Go dependency files.

As a result, Dependabot couldn't update your dependencies.

The error Dependabot encountered was:

go: github.com/Xuanwo/[email protected]: invalid version: unknown revision 25eae5601e66

If you think the above is an error on Dependabot's side please don't hesitate to get in touch - we'll do whatever we can to fix it.

View the update logs.

Add task unittests

Package for linux distribution

deb
rpm
pacman

qsctl sync repeatedly uploads existing objects

Reproduce:

run the same qsctl sync command repeatly

2.0.0-rc.1 Roadmap

Translation
Docs
Tests

Add confirmation before remove bucket forcely

本次操作将会删除该 Bucket 内包括分段在内的所有数据，删除后将无法恢复，请输入 Bucket 名以确认
BucketName: <等待用户输入>

sync --delete

When sync two directories with sync, and the delete flag was set, like qsctl sync qs://bucket/path/ /local/path/ --delete, I intended to delete the folder which was only exists in destination, but not in source. However, qsctl only deletes the files in the folder, leaves a blank folder behind.

failed to download object but exit code is 0

(py27) wangzw:qsctl wangzw$ qsctl cp  qs://hashdata-test/not-exist /tmp/
{"url": "http://docs.qingcloud.com/qingstor/api/common/error_code.html", "message": "The object you are accessing does not exist.", "code": "object_not_exists", "request_id": "fc6b0c3687ea11e6b41a52540443ed8a"}
(py27) wangzw:qsctl wangzw$ echo $?
0

AttributeError: 'int' object has no attribute 'encode'

[root@i-0ygo7ve6 x86_64]# qsctl ls qs://hashdata-public
[2017-01-20 15:14:26,615] qingstor-sdk parse DEBUG:
parsed_uri: https://qingstor.com:443/
[2017-01-20 15:14:26,616] qingstor-sdk get_authorization DEBUG:
GET

application/octet-stream
Fri, 20 Jan 2017 07:14:26 GMT
/
[2017-01-20 15:14:27,421] qingstor-sdk __init__ DEBUG:
status_code: 200
[2017-01-20 15:14:27,421] qingstor-sdk unpack_response_body DEBUG:
count: 2
[2017-01-20 15:14:27,422] qingstor-sdk unpack_response_body DEBUG:
buckets: [{u'url': u'http://hashdata-public.pek3a.qingstor.com', u'location': u'pek3a', u'name': u'hashdata-public', u'created': u'2016-04-12T03:26:47Z'}, {u'url': u'http://hashdata-test.pek3a.qingstor.com', u'location': u'pek3a', u'name': u'hashdata-test', u'created': u'2016-06-01T09:33:44Z'}]
[2017-01-20 15:14:27,422] qingstor-sdk parse DEBUG:
parsed_uri: https://pek3a.qingstor.com:443/hashdata-public
[2017-01-20 15:14:27,422] qingstor-sdk get_authorization DEBUG:
HEAD

application/octet-stream
Fri, 20 Jan 2017 07:14:27 GMT
/hashdata-public
[2017-01-20 15:14:27,540] qingstor-sdk __init__ DEBUG:
status_code: 200
Traceback (most recent call last):
  File "/usr/bin/qsctl", line 26, in <module>
    sys.exit(main())
  File "/usr/bin/qsctl", line 23, in main
    return qingstor.qsctl.driver.main()
  File "/usr/lib/python2.7/site-packages/qingstor/qsctl/driver.py", line 91, in main
    command.main(args[2:])
  File "/usr/lib/python2.7/site-packages/qingstor/qsctl/commands/base.py", line 225, in main
    return cls.send_request(options)
  File "/usr/lib/python2.7/site-packages/qingstor/qsctl/commands/ls.py", line 128, in send_request
    cls.list_keys(options)
  File "/usr/lib/python2.7/site-packages/qingstor/qsctl/commands/ls.py", line 117, in list_keys
    bucket, prefix, delimiter, marker, limit
  File "/usr/lib/python2.7/site-packages/qingstor/qsctl/commands/base.py", line 202, in list_multiple_keys
    marker=marker, prefix=prefix, delimiter=delimiter, limit=limit
  File "/usr/lib/python2.7/site-packages/qingstor/sdk/service/bucket.py", line 364, in list_objects
    delimiter=delimiter, limit=limit, marker=marker, prefix=prefix
  File "/usr/lib/python2.7/site-packages/qingstor/sdk/service/bucket.py", line 360, in list_objects_request
    return Request(self.config, operation)
  File "/usr/lib/python2.7/site-packages/qingstor/sdk/request.py", line 34, in __init__
    self.req = Builder(config, operation).parse()
  File "/usr/lib/python2.7/site-packages/qingstor/sdk/build.py", line 49, in parse
    parsed_operation["URI"] = self.parse_request_uri()
  File "/usr/lib/python2.7/site-packages/qingstor/sdk/build.py", line 171, in parse_request_uri
    params_parts.append("%s=%s" % (k, quote(v.encode("utf-8"))))
AttributeError: 'int' object has no attribute 'encode'
[root@i-0ygo7ve6 x86_64]# pip freeze | grep qing
qingstor-sdk==2.0.2
[root@i-0ygo7ve6 x86_64]# pip freeze | grep qs
qsctl==1.2.1

Failed in sync while path not exists

When executing qsctl sync to a directory not exists, it would not be created automatically if I did not set the suffix /. Instead, I got an error "both source and destination should be directories".

qsctl mb should set flag "zone" required

Executing qsctl mb command with only bucket-name will get 403 permission denied.
qsctl mb test-bucket
And adding "zone" flag to specify which zone to create bucket works.
qsctl mb test-bucket -z sh1a

support upload data to QingStor from stdin

it is a convenient feature to upload data to QingStor from stdin. We can pipeline data from upstream command and upload data to QingStor without write to local disk.

some_command_generate_lot_data | gzip | qsctl cp - qs://some_bucket/some_key

Set config if user do not have a default one

Add cp -r support

qsctl --rate-limit bug

An error occurred when using qsctl sync juniper qs://juniper/1234 --rate-limit 30 uploaded the pictures .

Failed to upload larger file

The file is about 60GB

error when uploading artifacts: No module named concurrent.futures

I got this error when uploading files to bucket
Traceback (most recent call last):
File "/usr/local/bin/qsctl", line 21, in
import qingstor.qsctl.driver
File "/usr/local/lib/python2.7/dist-packages/qingstor/qsctl/driver.py", line 28, in
from qingstor.qsctl.commands.ls import LsCommand
File "/usr/local/lib/python2.7/dist-packages/qingstor/qsctl/commands/ls.py", line 23, in
from .base import BaseCommand
File "/usr/local/lib/python2.7/dist-packages/qingstor/qsctl/commands/base.py", line 25, in
from concurrent.futures import ThreadPoolExecutor
ImportError: No module named concurrent.futures

Full build log can be found here : https://travis-ci.org/QingCloudAppcenter/user-guide/builds/276486639

qsctl throws UnicodeDecodeError while use os.walk

Step to reproduce

qsctl 1.7.0 with python 2.x on linux

>>> # create a file with bad name:
>>> touch $(echo -e "\x8b\x8bThis")
>>> qsctl sync ./ qs://xxxxxx

Refactor all task to meet storage changes

Old task were written with local and remote concept, but after posixfs storage replace, all tasks are operations between two Storagers.

So all task should be refactored to fit these changes.

Support mv

install error

Ubuntu 16.04, pip 18.0, install qsctl use "sudo pip install qsctl", the system report error "Command "/usr/bin/python -u -c "import setuptools, tokenize;file='/tmp/pip-install-vn14xwtx/pycrypto/setup.py';f=getattr(tokenize, 'open', open)(file);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, file, 'exec'))" install --record /tmp/pip-record-xficvb6x/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-install-vn14xwtx/pycrypto/"

how to fix ?

Add l10n support

qsctl cp exit with 0 on error

It should exit with a non-zero code on error.

Transferring: 100%|#########9| 75.5M/75.7M [00:00 ETA, 241kB/s][2018-10-12 10:49:55,021] urllib3.connectionpool urlopen WARNING:
Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProtocolError('Connection aborted.', error("(32, 'EPIPE')",))': /hashdata-docs/v2.1.1/020-admin-guide/020-admin-guide.pdf
Transferring: 100%|#########9| 75.7M/75.7M [00:00 ETA, 78.4kB/s][2018-10-12 10:50:50,463] urllib3.connectionpool urlopen WARNING:
Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProtocolError('Connection aborted.', BadStatusLine("''",))': /hashdata-docs/v2.1.1/search_index.json
Transferring: 100%|#########9| 75.7M/75.7M [00:00 ETA, 78.4kB/s][2018-10-12 10:50:58,237] urllib3.connectionpool urlopen WARNING:
Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProtocolError('Connection aborted.', BadStatusLine("''",))': /hashdata-docs/v2.1.1/020-admin-guide/020-admin-guide.pdf
[2018-10-12 10:51:55,728] urllib3.connectionpool urlopen WARNING:
Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProtocolError('Connection aborted.', BadStatusLine("''",))': /hashdata-docs/v2.1.1/search_index.json
[2018-10-12 10:52:06,266] urllib3.connectionpool urlopen WARNING:
Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProtocolError('Connection aborted.', BadStatusLine("''",))': /hashdata-docs/v2.1.1/020-admin-guide/020-admin-guide.pdf
[2018-10-12 10:53:07,487] urllib3.connectionpool _put_conn WARNING:
Connection pool is full, discarding connection: pek3a.qingstor.com

Support tab completion for remote/local path

Support tab completion for remote and local path should be easier to use.

# Example

$ qsctl ls qs://bucket-name/na

# Press Tab

$ qsctl ls qs://bucket-name/name/

# Press Tab Tab

$ qsctl ls qs://bucket-name/name/
LiLei  HanMeimei

Except for ls, other commands like cp, mv and sync could support this behavior.

rm -r panic

I tried to remove all objects from my bucket by rm -r, but got panic:

goroutine 1909 [sync.Cond.Wait]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:310
sync.runtime_notifyListWait(0xc0002ea5d0, 0x534)
/usr/local/go/src/runtime/sema.go:510 +0xf8
sync.(*Cond).Wait(0xc0002ea5c0)
/usr/local/go/src/sync/cond.go:56 +0x9d
github.com/Xuanwo/navvy.(*Pool).retrieveWorker(0xc00010bf80, 0x1)
/users/lance/desktop/qingcloud/projects/go/pkg/mod/github.com/!xuanwo/[email protected]/pool.go:155 +0x201
github.com/Xuanwo/navvy.(*Pool).Submit(0xc00010bf80, 0x16cc080, 0xc00021a5d0)
/users/lance/desktop/qingcloud/projects/go/pkg/mod/github.com/!xuanwo/[email protected]/pool.go:61 +0x52
created by github.com/yunify/qsctl/v2/utils.SubmitNextTask
/Users/lance/Desktop/QingCloud/projects/modgo/qsctl/utils/utils.go:44 +0x159

goroutine 1908 [sync.Cond.Wait]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:310
sync.runtime_notifyListWait(0xc0002ea5d0, 0x532)
/usr/local/go/src/runtime/sema.go:510 +0xf8
sync.(*Cond).Wait(0xc0002ea5c0)
/usr/local/go/src/sync/cond.go:56 +0x9d
github.com/Xuanwo/navvy.(*Pool).retrieveWorker(0xc00010bf80, 0x1)
/users/lance/desktop/qingcloud/projects/go/pkg/mod/github.com/!xuanwo/[email protected]/pool.go:155 +0x201
github.com/Xuanwo/navvy.(*Pool).Submit(0xc00010bf80, 0x16cc1a0, 0xc00021a150)
/users/lance/desktop/qingcloud/projects/go/pkg/mod/github.com/!xuanwo/[email protected]/pool.go:61 +0x52
created by github.com/yunify/qsctl/v2/utils.SubmitNextTask
/Users/lance/Desktop/QingCloud/projects/modgo/qsctl/utils/utils.go:44 +0x159

goroutine 1279 [IO wait]:
internal/poll.runtime_pollWait(0x1f5a158, 0x72, 0xffffffffffffffff)
/usr/local/go/src/runtime/netpoll.go:184 +0x55
internal/poll.(*pollDesc).wait(0xc0007c1218, 0x72, 0xf00, 0xf3d, 0xffffffffffffffff)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc0007c1200, 0xc0009b6000, 0xf3d, 0xf3d, 0x0, 0x0, 0x0)
/usr/local/go/src/internal/poll/fd_unix.go:169 +0x22b
net.(*netFD).Read(0xc0007c1200, 0xc0009b6000, 0xf3d, 0xf3d, 0x1a92e40, 0xe9, 0x0)
/usr/local/go/src/net/fd_unix.go:202 +0x4f
net.(*conn).Read(0xc000010090, 0xc0009b6000, 0xf3d, 0xf3d, 0x0, 0x0, 0x0)
/usr/local/go/src/net/net.go:184 +0x68
github.com/yunify/qingstor-sdk-go/v3/utils.Conn.Read(0x4f07240, 0xc000010090, 0x6fc23ac00, 0x6fc23ac00, 0x0, 0xc0009b6000, 0xf3d, 0xf3d, 0x203000, 0x203000, ...)
/users/lance/desktop/qingcloud/projects/go/pkg/mod/github.com/yunify/qingstor-sdk-go/[email protected]/utils/conn.go:91 +0x68
crypto/tls.(*atLeastReader).Read(0xc0001980e0, 0xc0009b6000, 0xf3d, 0xf3d, 0x0, 0x0, 0xc0006248a0)
/usr/local/go/src/crypto/tls/conn.go:780 +0x60
bytes.(*Buffer).ReadFrom(0xc0006b2258, 0x16cb800, 0xc0001980e0, 0x100b855, 0x1571dc0, 0x15c2740)
/usr/local/go/src/bytes/buffer.go:204 +0xb4
crypto/tls.(*Conn).readFromUntil(0xc0006b2000, 0x4f07298, 0xc0008708a0, 0x5, 0xc0008708a0, 0x0)
/usr/local/go/src/crypto/tls/conn.go:802 +0xec
crypto/tls.(*Conn).readRecordOrCCS(0xc0006b2000, 0x0, 0x0, 0x0)
/usr/local/go/src/crypto/tls/conn.go:609 +0x124
crypto/tls.(*Conn).readRecord(...)
/usr/local/go/src/crypto/tls/conn.go:577
crypto/tls.(*Conn).Read(0xc0006b2000, 0xc000568000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/usr/local/go/src/crypto/tls/conn.go:1255 +0x161
net/http.(*persistConn).Read(0xc0002326c0, 0xc000568000, 0x1000, 0x1000, 0x10067ad, 0x60, 0x0)
/usr/local/go/src/net/http/transport.go:1744 +0x75
bufio.(*Reader).fill(0xc0008204e0)
/usr/local/go/src/bufio/bufio.go:100 +0x103
bufio.(*Reader).Peek(0xc0008204e0, 0x1, 0xc000102ea0, 0x0, 0x1386c79, 0xc0005a8c98, 0x1048757)
/usr/local/go/src/bufio/bufio.go:138 +0x4f
net/http.(*persistConn).readLoop(0xc0002326c0)
/usr/local/go/src/net/http/transport.go:1897 +0x1d6
created by net/http.(*Transport).dialConn
/usr/local/go/src/net/http/transport.go:1566 +0xafe

goroutine 1299 [select]:
net.(*Resolver).lookupIPAddr(0x1a92660, 0x16d7880, 0xc0009361e0, 0x16062df, 0x3, 0xc0001b0c20, 0x12, 0x1bb, 0x0, 0x0, ...)
/usr/local/go/src/net/lookup.go:275 +0x664
net.(*Resolver).internetAddrList(0x1a92660, 0x16d7880, 0xc0009361e0, 0x16062df, 0x3, 0xc0001b0c20, 0x16, 0x0, 0x0, 0x0, ...)
/usr/local/go/src/net/ipsock.go:280 +0x605
net.(*Resolver).resolveAddrList(0x1a92660, 0x16d7880, 0xc0009361e0, 0x16065d9, 0x4, 0x16062df, 0x3, 0xc0001b0c20, 0x16, 0x0, ...)
/usr/local/go/src/net/dial.go:220 +0x506
net.(*Dialer).DialContext(0xc000127c20, 0x16d7840, 0xc0000ae008, 0x16062df, 0x3, 0xc0001b0c20, 0x16, 0x0, 0x0, 0x0, ...)
/usr/local/go/src/net/dial.go:402 +0x1fe
github.com/yunify/qingstor-sdk-go/v3/utils.(*Dialer).DialContext(0xc0002becc0, 0x16d7840, 0xc0000ae008, 0x16062df, 0x3, 0xc0001b0c20, 0x16, 0xc00054c480, 0x118, 0x118, ...)
/users/lance/desktop/qingcloud/projects/go/pkg/mod/github.com/yunify/qingstor-sdk-go/[email protected]/utils/conn.go:47 +0x80
net/http.(*Transport).dial(0xc0002cab40, 0x16d7840, 0xc0000ae008, 0x16062df, 0x3, 0xc0001b0c20, 0x16, 0x0, 0x1326ff0, 0xc0003f1c00, ...)
/usr/local/go/src/net/http/transport.go:1034 +0x1f5
net/http.(*Transport).dialConn(0xc0002cab40, 0x16d7840, 0xc0000ae008, 0x0, 0xc00054c180, 0x5, 0xc0001b0c20, 0x16, 0x0, 0x16314d0, ...)
/usr/local/go/src/net/http/transport.go:1470 +0x193e
net/http.(*Transport).dialConnFor(0xc0002cab40, 0xc00050ee70)
/usr/local/go/src/net/http/transport.go:1300 +0xdc
created by net/http.(*Transport).queueForDial
/usr/local/go/src/net/http/transport.go:1269 +0x41d

Failed to find module packages.urllib3.util.retry

The command "gitbook build . $FILENAME" exited with 0.
0.18s$ if [ "$TRAVIS_BRANCH" = "master" -a "$TRAVIS_PULL_REQUEST" = "false" ]; then openssl aes-256-cbc -K $encrypted_7127281a9724_key -iv $encrypted_7127281a9724_iv -in config.yaml.enc -out config.yaml -d; qsctl sync _book/qingcloud-apps-user-guide/ qs://appcenter-docs/user-guide/apps/ --config config.yaml; fi
Traceback (most recent call last):
File "/usr/local/bin/qsctl", line 21, in
import qingstor.qsctl.driver
File "/usr/local/lib/python2.7/dist-packages/qingstor/qsctl/driver.py", line 28, in
from qingstor.qsctl.commands.ls import LsCommand
File "/usr/local/lib/python2.7/dist-packages/qingstor/qsctl/commands/ls.py", line 23, in
from .base import BaseCommand
File "/usr/local/lib/python2.7/dist-packages/qingstor/qsctl/commands/base.py", line 25, in
from qingstor.sdk.service.qingstor import QingStor
File "/usr/local/lib/python2.7/dist-packages/qingstor/sdk/service/qingstor.py", line 22, in
from requests.packages.urllib3.util.retry import Retry
ImportError: No module named packages.urllib3.util.retry

The complete log can be found here https://api.travis-ci.org/jobs/256741370/log.txt?deansi=true

Failed to init config

Error: setup config failed [open /root/.qingstor/config.yaml: no such file or directory], please try again

failed use cp command on Ubuntu Server 18.04.1 LTS 64bit

debug log

❯ qsctl cp Death.Wish.2018.1080p.BluRay.H264.AAC-RARBG.mp4 qs://mark-movie
DEBU[2019-11-08T17:45:47+08:00] Started CopyFileTask {DestinationPath: Death.Wish.2018.1080p.BluRay.H264.AAC-RARBG.mp4, DestinationStorage: qingstor Storager {WorkDir /}, SourcePath: Death.Wish.2018.1080p.BluRay.H264.AAC-RARBG.mp4, SourceStorage: posixfs Storager {WorkDir /mnt/plex/Death.Wish.2018.1080p.BluRay.H264.AAC-RARBG}}
DEBU[2019-11-08T17:45:47+08:00] Started CopyLargeFileTask {DestinationPath: Death.Wish.2018.1080p.BluRay.H264.AAC-RARBG.mp4, DestinationStorage: qingstor Storager {WorkDir /}, SourcePath: Death.Wish.2018.1080p.BluRay.H264.AAC-RARBG.mp4, SourceStorage: posixfs Storager {WorkDir /mnt/plex/Death.Wish.2018.1080p.BluRay.H264.AAC-RARBG}, TotalSize: 2185046542}
DEBU[2019-11-08T17:45:47+08:00] Started SegmentInitTask {PartSize: 134217728, Path: Death.Wish.2018.1080p.BluRay.H264.AAC-RARBG.mp4, Storage: qingstor Storager {WorkDir /}}
DEBU[2019-11-08T17:45:47+08:00] Finished SegmentInitTask {PartSize: 134217728, Path: Death.Wish.2018.1080p.BluRay.H264.AAC-RARBG.mp4, Storage: qingstor Storager {WorkDir /}}
panic: SegmentID value is not valid

goroutine 14 [running]:
github.com/yunify/qsctl/v2/pkg/types.(*SegmentID).GetSegmentID(0xc0001c51b8, 0x0, 0x0)
        github.com/yunify/qsctl/v2@/pkg/types/types.go:2039 +0xea
github.com/yunify/qsctl/v2/task.(*CopyLargeFileTask).run(0xc000092000)
        github.com/yunify/qsctl/v2@/task/copy.go:71 +0x18c
github.com/yunify/qsctl/v2/task.(*CopyLargeFileTask).Run(0xc000092000)
        github.com/yunify/qsctl/v2@/task/generated.go:251 +0xa4
github.com/yunify/qsctl/v2/pkg/schedule.(*task).Run(0xc000486100)
        github.com/yunify/qsctl/v2@/pkg/schedule/scheduler.go:63 +0x64
created by github.com/yunify/qsctl/v2/pkg/schedule.(*RealScheduler).Sync
        github.com/yunify/qsctl/v2@/pkg/schedule/scheduler.go:86 +0xff

qsctl doesn't handle Ctrl+C on windows

Run qsctl cp xxx qs://xxx, while click Ctrl + C, qsctl should exit but not.

Handle error correctly

Return helpful error messages for users.

qsctl log rules

When we use logger, and when we print to stdout
Should we need to hide the logger when user not enable the debug?

Roadmap to qsctl 2.0

qsctl's first public commit is 69a5258 at Jul 5, 2016. It's a long time ago, and everything has been changed. It's time to move forward.

Reasons

Yep, we still love python, but it's not suitable for qsctl now.

Python GIL sucks.
Python 2 will be retired, but our users' still use python 2.7 (even 2.6) #73
qsctl code structure is hard to maintain and harder to add new features.
It's hard to distribute python version's qsctl in binary.

Promise

We will try our best to keep qsctl bug free and do not have known vulnerability.
We will try our best to keep qsctl stable and do not break any old features.
We will try our best to make qsctl faster.
We will add more user-friendly features in qsctl, such as cat, tee, initialization wizard.

Roadmap

Current Stage: Beta

Alpha Stage

In alpha stage, we will implement all features that qsctl used to support and add features that we promised.

Old features:

New features:

Fix bug:

met error when install qsctl #73
qsctl cp exit with 0 on error #71
sys.exit will only exit child thread #64
qsctl doesn't handle Ctrl+C on windows #61

Beta Stage

In beta stage, we will invite external users to test the qsctl 2.0, and fix every bug we found.

During this stage, we are accepting community's feature request. Some of them will be implemented in beta stage, and others will added in stable stage.

RC Stage

In RC stage, we will prepare for the stable release: fix bug, refactor, tests ...

Stable Stage

qsctl 2.0 is ready!

No breaking changes will be introduced until qsctl 3.0

Unify required and dependent tasks

Currently we have two methods to schedule a task:

required tasks can be scheduled via AddTodo and NextTodo
dependent tasks can be scheduled via scheduler

These make it hard to schedule ListObjects in a scheduler, we need to unify them instead.

cp output is not correct

:) bin/qsctl cp . qs://xuanwo-qsctl-test-1/test_file -r
Dir <qsctl> copied to <>.

Output is not correct.

qsctl copy get wrong object content

I copy 12254 small files from a bucket to local filesystem. The total data is 534.05 MB.

qsctl cp --no-progress -r qs://xxx/ /local/path/

The command exit with exit code 0.

I got wrong content for some files, which is not consistent with the object content in qingstor bucket.

[root@i-ds1xio2d ~]# python --version
Python 2.7.5
[root@i-ds1xio2d ~]# qsctl -v
qsctl 1.7.2
[root@i-ds1xio2d ~]# uname -a
Linux i-ds1xio2d 3.10.0-693.2.2.el7.x86_64 #1 SMP Tue Sep 12 22:26:13 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
[root@i-ds1xio2d ~]# cat /etc/redhat-release
CentOS Linux release 7.4.1708 (Core)

Add sync support

Works like rsync.

Presign not work as intended

I set expire to 30, got an URL as intended. However, I can still download object from this pre-signed URL after 5 minutes.

met error when install qsctl

ubuntu 16.04.
First install, root user, pip 8.1.2, error message:
Command "/usr/bin/python2 -u -c "import setuptools, tokenize;file='/tmp/pip-build-6iS1nI/pycrypto/setup.py';exec(compile(getattr(tokenize, 'open', open)(file).read().replace('\r\n', '\n'), file, 'exec'))" install --record /tmp/pip-0dESnP-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-build-6iS1nI/pycrypto/

upgrade pip to 19.1.1
Second installation, same server, root user, error message:
Running setup.py install for pycrypto ... error
ERROR: Complete output from command /usr/bin/python2 -u -c 'import setuptools, tokenize;file='"'"'/tmp/pip-install-oiuhcr/pycrypto/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(file);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' install --record /tmp/pip-record-NfQ6_6/install-record.txt --single-version-externally-managed --compile:
ERROR: running install
running build
running build_py
creating build
creating build/lib.linux-x86_64-2.7
creating build/lib.linux-x86_64-2.7/Crypto
copying lib/Crypto/init.py -> build/lib.linux-x86_64-2.7/Crypto
copying lib/Crypto/pct_warnings.py -> build/lib.linux-x86_64-2.7/Crypto
creating build/lib.linux-x86_64-2.7/Crypto/Hash
copying lib/Crypto/Hash/SHA224.py -> build/lib.linux-x86_64-2.7/Crypto/Hash
copying lib/Crypto/Hash/SHA384.py -> build/lib.linux-x86_64-2.7/Crypto/Hash
copying lib/Crypto/Hash/MD5.py -> build/lib.linux-x86_64-2.7/Crypto/Hash
copying lib/Crypto/Hash/RIPEMD.py -> build/lib.linux-x86_64-2.7/Crypto/Hash
copying lib/Crypto/Hash/MD2.py -> build/lib.linux-x86_64-2.7/Crypto/Hash
copying lib/Crypto/Hash/init.py -> build/lib.linux-x86_64-2.7/Crypto/Hash
copying lib/Crypto/Hash/HMAC.py -> build/lib.linux-x86_64-2.7/Crypto/Hash
copying lib/Crypto/Hash/SHA512.py -> build/lib.linux-x86_64-2.7/Crypto/Hash
.......

maybe the actual error was from "pip install pycrypto"

can qsctl cp support more than 2G file?

Can use ”qsctl cp“ to copy more then 2G file into bucket directly?