aliyun / ossutil Goto Github PK

OSSUTIL VERSION: 1.4.0

When using the copy command to recursively import a specific directory, it adds full prefix to local_dir:

Example:
I have a directory with images files oss://bucket/dir1/dir2/dir3/*.png
Using:
ossutil cp oss://bucket/dir1/dir2/dir3 /home/ubuntu/dir3 -r -u -f
will copy all the png files in oss://bucket/dir1/dir2/dir3 to /home/ubuntu/dir3/dir1/dir2/dir3/*.png
when I would like to copy to /home/ubuntu/dir3/*.png

The binary release from here doesn't work in docker ubuntu 18.04 image, when building this dockerfile:

FROM ubuntu:18.04

RUN apt-get update && apt-get install -y curl
RUN curl -o ossutil http://gosspublic.alicdn.com/ossutil/1.4.2/ossutil64 && chmod +x ossutil
RUN ./ossutil -v

Output:

Step 4/4 : RUN ./ossutil -v
 ---> Running in 872483d5f935
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x40 pc=0x6b4981]

goroutine 1 [running]:
github.com/aliyun/ossutil/lib.DecideConfigFile(0x0, 0x0, 0x7a8017, 0x7)
        /Users/fengyu/go/src/github.com/aliyun/ossutil/lib/config_helper.go:57 +0x51
The command '/bin/sh -c ./ossutil -v' returned a non-zero code: 2

But it works fine if I build it from scratch:

FROM ubuntu:18.04

WORKDIR /root

RUN apt-get update && apt-get install -y curl git
RUN curl -O https://dl.google.com/go/go1.10.3.linux-amd64.tar.gz \
 && tar xvf go1.10.3.linux-amd64.tar.gz

ENV GOROOT=/root/go
ENV GOPATH=/root/work
ENV PATH=$PATH:$GOROOT/bin:$GOPATH/bin

RUN go get github.com/aliyun/ossutil \
 && cd $GOPATH/src \
 && go build github.com/aliyun/ossutil/ossutil.go

RUN ossutil -v

Output:

Step 9/9 : RUN ossutil -v
 ---> Running in 24c75ef99721
ossutil version: 1.4.2

我想把ossutil打包，放到ppa上，这样在添加了ppa源的ubuntu系统中安装、更新、uninstall etc 都比较方便，不知道 aliyun有没有开 自己的ppa，或者可以让用户自己弄。

上传能不能加个限速功能?直接打满不合适吧

in alpine
bash-4.4# ./ossutil64
bash: ./ossutil64: No such file or directory

I think this should be supported cuz MC does

Hi, I'm mosuke5. I'm using this tool usually.
By the way why don't you publish binary pkg about this tool?
I think you should publish it.

https://github.com/aliyun/ossutil/releases
目前这里只有源码，而构建后的在 https://help.aliyun.com/document_detail/50452.html 这里。

文档那个页面上的链接，在国外访问非常不顺畅，在 Travis CI 需要构建后部署的场景无法顺利的下载下来，像这样。。

$ curl --location --retry 5 --output ossutil 'http://docs-aliyun.cn-hangzhou.oss.aliyun-inc.com/assets/attach/50452/intl_zh/1488523315888/ossutil'
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  8 8150k    8  668k    0     0   2371      0  0:58:39  0:04:48  0:53:51  8344
curl: (18) transfer closed with 7660849 bytes remaining to read
The command "curl --location --retry 5 --output ossutil 'http://docs-aliyun.cn-hangzhou.oss.aliyun-inc.com/assets/attach/50452/intl_zh/1488523315888/ossutil'" failed and exited with 18 during .
Your build has been stopped.

如果在 github release 上提供一个构件好的版本，在国外访问会顺畅很多

通过 ossutil 1.4（Linux x86_64版，Ubuntu 17.10 64位） 版无法上传超过4GB的文件。同样的环境，使用 osscmd 刚正常。错误信息如下：

Total num: 1, size: 4,639,948,800. Dealed num: 0, OK size: 0, Progress: 0%
Total num: 1, size: 4,639,948,800. Dealed num: 0, Transfer size: 0. When error happens.
Error: Put http://XXXX.oss-cn-shenzhen.aliyuncs.com/V834394-01-7.3.iso?partNumber=3&uploadId=XXXXDB06B4D64B4B9DD9BA359FD8XXXX: EOF, File=/PATH/OracleLinux/V834394-01-7.3.iso!

比如--temp_dir=TEMP_DIR

原因：
我的场景是这样，使用ossutil下载oss的大量文件到本地目录，这个目录由logstash的file插件监控着，并且logstash配置为不允许读取.temp结尾的文件，ossutil产生.temp文件时，logstash是会扫描到并记录这个文件路径和inode信息以及偏移量，当ossutil将.temp后缀改名后，文件inode是不会改变的，logstash再读取这个文件时，偏移量就不是从0开始的，所以导致了丢失数据

Win10 1809版本无法使用

Possible cause : at line 87 lib/const.go
Version string = "1.0.0"
This misleads the update checker.

./ossutil ls oss://
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x30 pc=0x6463fb]

goroutine 1 [running]:
github.com/aliyun/aliyun-oss-go-sdk/oss.Conn.doRequest(0xc4201760f0, 0xc4200118c0, 0xc4200118f0, 0x7b59ba, 0x3, 0x0, 0xc420016e4e, 0x1, 0x0, 0x0, ...)
/home/zubo.yzb/go/src/github.com/aliyun/aliyun-oss-go-sdk/oss/conn.go:192 +0x17b
github.com/aliyun/aliyun-oss-go-sdk/oss.Conn.Do(0xc4201760f0, 0xc4200118c0, 0xc4200118f0, 0x7b59ba, 0x3, 0x0, 0x0, 0x0, 0x0, 0xc4200119b0, ...)
/home/zubo.yzb/go/src/github.com/aliyun/aliyun-oss-go-sdk/oss/conn.go:59 +0x276
github.com/aliyun/aliyun-oss-go-sdk/oss.Client.do(0xc4201760f0, 0xc4200d2fa0, 0x7b59ba, 0x3, 0x0, 0x0, 0xc4200119b0, 0x0, 0x0, 0x0, ...)
/home/zubo.yzb/go/src/github.com/aliyun/aliyun-oss-go-sdk/oss/client.go:800 +0xea
github.com/aliyun/aliyun-oss-go-sdk/oss.Client.ListBuckets(0xc4201760f0, 0xc4200d2fa0, 0xc42004f8a0, 0x2, 0x2, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/home/zubo.yzb/go/src/github.com/aliyun/aliyun-oss-go-sdk/oss/client.go:143 +0x12c
github.com/aliyun/ossutil/lib.(*ListCommand).ossListBucketsRetry(0x9b62e0, 0xc420041cb0, 0xc42004f8a0, 0x2, 0x2, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/home/zubo.yzb/go/src/github.com/aliyun/ossutil/lib/ls.go:461 +0x11f
github.com/aliyun/ossutil/lib.(*ListCommand).listBuckets(0x9b62e0, 0x0, 0x0, 0x0, 0x7ffd1db8d6fc)
/home/zubo.yzb/go/src/github.com/aliyun/ossutil/lib/ls.go:411 +0x67d
github.com/aliyun/ossutil/lib.(*ListCommand).RunCommand(0x9b62e0, 0x79cde0, 0x9b62e0)
/home/zubo.yzb/go/src/github.com/aliyun/ossutil/lib/ls.go:380 +0x156
github.com/aliyun/ossutil/lib.(*CommandManager).RunCommand(0xc42004fe18, 0x7ffd1db8d6f9, 0x2, 0xc42005a590, 0x1, 0x3, 0xc420011380, 0x29, 0xc4200185d0, 0x23)
/home/zubo.yzb/go/src/github.com/aliyun/ossutil/lib/command_manager.go:89 +0x12b
github.com/aliyun/ossutil/lib.RunCommand(0xc42005a580, 0x2, 0x4, 0xc420011380, 0x0, 0x0, 0x7d3da8)
/home/zubo.yzb/go/src/github.com/aliyun/ossutil/lib/command_manager.go:63 +0x106
github.com/aliyun/ossutil/lib.ParseAndRunCommand(0xc420014ba0, 0x14)
/home/zubo.yzb/go/src/github.com/aliyun/ossutil/lib/command_manager.go:27 +0x101
main.main()
/home/zubo.yzb/go/src/github.com/aliyun/ossutil/ossutil.go:12 +0x26

运行ossutil ls报错：

Error: Get http:/?marker=&prefix=: dial tcp: lookup oss-cn-beijing.aliyuncs.com on 10.10.0.1:53: read udp 10.10.5.76:52088->10.10.0.1:53: i/o timeout!

环境CentOS 7，iptables全ACCEPT，selinux已关。

When trying to ossutil rm a large number of files, I get the following error:
Total 1002 objects. Removed 100 objects. when error happens.
Error: oss: service returned error: StatusCode=400, ErrorCode=MalformedXML, ErrorMessage="The XML you provided was not well-formed or did not validate against our published schema.", RequestId=5D3F0D47C1E6C9CDCBBD3B8F

What is the suggested way to delete a large number of objects ?
Thanks

ls 能不能增加一个--max-depth=1 这样的参数
我希望可以直接 ossutil ls bukeck -d 的时候列出所有文件夹对象
然后可以用--max-depth=1 这样的参数自己定义层数

Hi, would it be possible to fetch credentials from environment variables ALICLOUD_ACCESS_KEY and ALICLOUD_SECRET_KEY.

That would make it easy to run from within Docker.

Thanks !

~ # ls -ltr
total 9820
-rwxr-xr-x 1 root root 10053808 Aug 14 08:32 ossutil64
~ # cat /etc/issue
Welcome to Alpine Linux 3.10
Kernel \r on an \m (\l)

~ # bash ./ossutil64
./ossutil64: ./ossutil64: cannot execute binary file

I don't see any help or example of using ossutil on an ECS instance that has a RAM role attached to.

When I try running ossutil64 on such an instance where the attached RAM role has AliyunOSSFullAccess permission, I still see the following error message :-

.ossutilconfig: no such file or directory

Can an example be added to the documentation stating how to use ossutil on ECS instance with valid RAM role ?

shellus@raspberrypi ~ $ ossutil cp "oss://wawa-disk/Website Backup/shuoxingba/db-2016-10-05.sql.tgz" ./
Total num: 1, size: 229,522,629. Dealed num: 0, OK size: 0, Progress: 0%

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x4 pc=0x11580]

goroutine 1 [running]:
sync/atomic.addUint64(0x49903c, 0x271668, 0x0, 0x76f6e708, 0x0)
C:/Go/src/sync/atomic/64bit_arm.go:31 +0x4c
github.com/aliyun/ossutil/lib.(*CPMonitor).updateTransferSize(0x49901c, 0x271668, 0x0)
C:/Users/shellus/go/src/github.com/aliyun/ossutil/lib/monitor.go:428 +0x34
github.com/aliyun/ossutil/lib.(*OssProgressListener).ProgressChanged(0x10868440, 0x1093dd40)
C:/Users/shellus/go/src/github.com/aliyun/ossutil/lib/cp.go:88 +0x6c
github.com/aliyun/aliyun-oss-go-sdk/oss.publishProgress(0x470ca0, 0x10868440, 0x1093dd40)
C:/Users/shellus/go/src/github.com/aliyun/aliyun-oss-go-sdk/oss/progress.go:43 +0x40
github.com/aliyun/aliyun-oss-go-sdk/oss.Bucket.downloadFileWithCp(0x1080a1b0, 0x10800760, 0x7e9018b1, 0x9, 0x7e9018bb, 0x2f, 0x10746ec0, 0x31, 0x271668, 0x0, ...)
C:/Users/shellus/go/src/github.com/aliyun/aliyun-oss-go-sdk/oss/download.go:474 +0x608
github.com/aliyun/aliyun-oss-go-sdk/oss.Bucket.DownloadFile(0x1080a1b0, 0x10800760, 0x7e9018b1, 0x9, 0x7e9018bb, 0x2f, 0x10746ec0, 0x31, 0x271668, 0x0, ...)
C:/Users/shellus/go/src/github.com/aliyun/aliyun-oss-go-sdk/oss/download.go:42 +0x1f8
github.com/aliyun/ossutil/lib.(*CopyCommand).ossResumeDownloadRetry(0x498f48, 0x10800780, 0x7e9018bb, 0x2f, 0x10746ec0, 0x31, 0xdae3cc5, 0x0, 0x271668, 0x0, ...)
C:/Users/shellus/go/src/github.com/aliyun/ossutil/lib/cp.go:1831 +0xb8
github.com/aliyun/ossutil/lib.(*CopyCommand).downloadSingleFile(0x498f48, 0x10800780, 0x7e9018bb, 0x2f, 0xffffffff, 0xffffffff, 0xd0ff7ce3, 0xe, 0x195f6c23, 0x49aa38, ...)
C:/Users/shellus/go/src/github.com/aliyun/ossutil/lib/cp.go:1776 +0x67c
github.com/aliyun/ossutil/lib.(*CopyCommand).downloadSingleFileWithReport(0x498f48, 0x10800780, 0x7e9018bb, 0x2f, 0xffffffff, 0xffffffff, 0xd0ff7ce3, 0xe, 0x195f6c23, 0x49aa38, ...)
C:/Users/shellus/go/src/github.com/aliyun/ossutil/lib/cp.go:1717 +0x40
github.com/aliyun/ossutil/lib.(*CopyCommand).downloadFiles(0x498f48, 0x7e9018ab, 0x3f, 0x7e9018b1, 0x9, 0x7e9018bb, 0x2f, 0x7e9018eb, 0x2, 0xf, ...)
C:/Users/shellus/go/src/github.com/aliyun/ossutil/lib/cp.go:1680 +0x1dc
github.com/aliyun/ossutil/lib.(*CopyCommand).RunCommand(0x498f48, 0x0, 0x0)
C:/Users/shellus/go/src/github.com/aliyun/ossutil/lib/cp.go:1077 +0x6e4
github.com/aliyun/ossutil/lib.(*CommandManager).RunCommand(0x1072ff28, 0x7e9018a8, 0x2, 0x1070f8c8, 0x2, 0x3, 0x1070f8a0, 0xc4, 0x3204e7, 0xf7)
C:/Users/shellus/go/src/github.com/aliyun/ossutil/lib/command_manager.go:89 +0xe0
github.com/aliyun/ossutil/lib.RunCommand(0x1070f8c0, 0x3, 0x4, 0x1070f8a0, 0x0, 0x0, 0x1)
C:/Users/shellus/go/src/github.com/aliyun/ossutil/lib/command_manager.go:63 +0xbc
github.com/aliyun/ossutil/lib.ParseAndRunCommand(0x0, 0x1070f640)
C:/Users/shellus/go/src/github.com/aliyun/ossutil/lib/command_manager.go:27 +0xec
main.main()
C:/Users/shellus/go/src/github.com/aliyun/ossutil/ossutil.go:11 +0x14

文件是在windows7上使用go get 下载源码，然后使用GOOS=linux GOARCH=arm go build来编译到的。
然后在树莓派上运行。ls 之类的命令正常。但是cp时，空指针异常。

Error: oss: service returned error: StatusCode=403, ErrorCode=AccessDenied, ErrorMessage=The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint., RequestId=5B8E81775CC26B5BBA097D54, Bucket=test!

I met problem when I tried to upload local folder(a git repo) to OSS bucket.
I just want to upload the normal files only, but couldn't find an option to exclude the '.git' dir.

软连接能不能支持多层
或者加个-L的参数来支持多层软连接

当object name是 oss://mingdi-hz-2/y9n/g/%E9%98%B4%E9%98%B3%E5%B8%88-%E9%A3%9F%E6%A2%A6%E8%B2%98.jpg时，显示ls命令显示如下：
2016-12-16 14:54:18 +0800 CST 11903 75209F561F61BADFA0F2B112D61BE7EE oss://mingdi-hz-2/y9n/g/%!E(MISSING)9%B4%!E(MISSING)9%B3%!E(MISSING)5%!B(MISSING)8%!-(MISSING)%!E(MISSING)9%!A(MISSING)3%!F(MISSING)%!E(MISSING)6%!A(MISSING)2%!A(MISSING)6%!E(MISSING)8%!B(MISSING)2%!j(MISSING)pg

Hi,
Would it please be possible to direct error messages to stderr, and not stdout ?

Such as:
b'Error: oss: service returned error: StatusCode=404, ErrorCode=NoSuchBucket, ErrorMessage=The specified bucket does not exist., RequestId=5B4C9536C7E7649D66BC1CE0, Bucket=notexisting-123!\n'

access_key & access_secret 是写在环境变量里的，ossutil -i $ACCESS_KEY -k $ACCESS_SECRET cp xxx 这样；
生成 report file 会把敏感信息也直接保存在里面，非常不安全，可不可以禁止生成 report file？

ossutil cp -r -j 10 -u oss://zblz/imgs/ ./

Error occurs, message: oss: service returned error: StatusCode=404, ErrorCode=NoSuchStyle, ErrorMessage=No Such Image Style., RequestId=5D104B9408F4BE32886E0689, Bucket=zblz, Object=imgs/1502568338814.jpg@!300x. See more information in file: ossutil_output/ossutil_report_20190624_120316.report
Scanned 54300 objects. Dealed num: 32453(download 645 objects, skip 31808 objects), OK size: 20,498,633,633(transfer: 354,494,013, skip: 20,144,139,620). When error happens.
Error: oss: service returned error: StatusCode=403, ErrorCode=AccessDenied, ErrorMessage=You do not have read permission on this object., RequestId=5D104BA21E7FB62C9F7EDCB5, Bucket=zblz, Object=imgs/1502890551777.jpg!

这些异常文件不能复制就算了，希望能有个参数可以直接忽略这些错误，跳过文件后继续复制

Ossutil use none vendoring management tools such as godep / govendor / etc, it is using raw Go lang repo management.

Will ossutil use go vendoring system in future?

related Issue: Homebrew/homebrew-core#43857

We need upload/download through proxy.

Doesn't matter how the ossutil command with ls switch is used it doesn't work.

./ossutilmac64 ls oss://macos-backup --include=*.jpg
Error: invalid usage of "ls" command, reason: the command does not support option: "include", please try "help ls" for more information!

./ossutilmac64 ls oss://macos-backup --include= *.jpg
Error: invalid usage of "ls" command, reason: the command needs at most 1 argument, please try "help ls" for more information!

./ossutilmac64 ls oss://macos-backup --include = *.jpg
Error: invalid usage of "ls" command, reason: the command needs at most 1 argument, please try "help ls" for more information!

./ossutilmac64 ls oss://macos-backup --include =*.jpg
Error: invalid usage of "ls" command, reason: the command does not support option: "include", please try "help ls" for more information!

./ossutilmac64 ls oss://macos-backup --include="*.jpg"
Error: invalid usage of "ls" command, reason: the command does not support option: "include", please try "help ls" for more information!

./ossutilmac64 ls oss://macos-backup --include= "*.jpg"
Error: invalid usage of "ls" command, reason: the command needs at most 1 argument, please try "help ls" for more information!

/ossutilmac64 ls oss://macos-backup --include = "*.jpg"
Error: invalid usage of "ls" command, reason: the command needs at most 1 argument, please try "help ls" for more information!

ossutile version:1.6.3

cp 上传进度错误

size,progree,speed都显示错误，speed还会出现负值。

官网和go get下的版本都是1.0.0版本，为什么文档里面都说当前版本是：1.2.0

osscmd 可以设置retry count，
而ossutil传输大文件不方便。
./ossutil64 cp传输大文件不方便。
retry count:9:upload file:data/StereoRecons-5d6fbf86d4c2b807c939fa0f/tutorial_out/reconstruction_sequential/sfm_data.json.

Can we also have the binary release, it's much easier to have a binary release for download it in script, not installing it from scratch.

Like aliyun cli does. https://github.com/aliyun/aliyun-cli/releases/download/v3.0.2/aliyun-cli-linux-amd64.tar.gz

Thanks.

能不能增加一个传输的时候排除某个文件或者文件夹选项
或者传输文件的时候支持通配符或者正则

之前官网帮助页面有arm版本的，现在又没有了。

For performance testing it would be helpful if copy supported writing to stdout or at least /dev/null. For example, aws s3 supports writing to stdout using - as the local path.

执行下面这操作时提示错误：

ali@ljp:/var/backups$ ossutil64 cp /var/backups/alihome.tgz oss://alicode/`date +%Y%m%d`/ -f
Error: mkdir .ossutil_checkpoint: permission denied!

上传文件要使用到mkdir?
在ali用户的home目录下执行这个命令是正常的。

In our company I needs socks5 proxy or http proxy to access internet, does ossutil support socks5 proxy or http connect proxy?

举例如下：
a目录下有
a-2018-06-17.bak (2018-06-17创建）
a-2018-06-18.bak (2018-06-18创建）
a-2018-06-19.bak (2018-06-19创建）

当我在2018-06-25号使用ossutil cp命令将这些文件上传到oss上时，文件的更新时间都变成上传时刻的时间2018-06-25，请问ossutil是否有选项，在上传或下载时可以保留原来文件的创建时间，以方便在批量备份时实现oss的生命周期控制。

你好，我们在CI环境中使用 ossutil 上传文件，上传过程的日志可以有办法隐藏吗？

我们只需要知道是否上传成功就可以了

比如windows 磁盘的路径\.\PhysicalDrive1 不支持上传。。
ossutil32.exe cp \.\PhysicalDrive1 oss://evidence-image-shanghai/hello
并且我看了sdk。。这种路径skd也不支持。。只有append的方法适合这种情形，然后这种方法还限制最大上传5个G。。没考虑过支持上传整个磁盘么？。。取证界可是刚需。。现在我该咋搞呢，貌似只能自己修改源码了？

issue 19 的fix如下

Pull Request: #23
Update atomic.AddInt64() to RWMutex to avoid atomic.AddInt64() crash
Local mutex不能避免并发访问。需要根据go官方advise重新fix

我本地没有go编译环境，不知能否方便下使用者，直接在release之后上传各个对应的操作系统编译好的版本呢？

Is it possible to download files in CDN, not in OSS?

ossutil cp -r oss://mycard/mobile oss://mycard/mobile3

Error: "oss://mycard/mobile3" include "oss://mycard/mobile", it's not allowed, recursivlly copy should be avoided!

显然mobile3和mobile是两个不同的目录，并没有包含关系