Code Monkey home page Code Monkey logo

aliyun-oss-go-sdk's Introduction

Alibaba Cloud OSS SDK for Go

GitHub Version Build Status Coverage Status

About

  • This Go SDK is based on the official APIs of Alibaba Cloud OSS.
  • Alibaba Cloud Object Storage Service (OSS) is a cloud storage service provided by Alibaba Cloud, featuring massive capacity, security, a low cost, and high reliability.
  • The OSS can store any type of files and therefore applies to various websites, development enterprises and developers.
  • With this SDK, you can upload, download and manage data on any app anytime and anywhere conveniently.

Version

  • Current version: v3.0.2

Running Environment

  • Go 1.5 or above.

Installing

Install the SDK through GitHub

  • Run the 'go get github.com/aliyun/aliyun-oss-go-sdk' command to get the remote code package.
  • Use 'import "github.com/aliyun/aliyun-oss-go-sdk/oss"' in your code to introduce OSS Go SDK package.

Getting Started

List Bucket

    client, err := oss.New("Endpoint", "AccessKeyId", "AccessKeySecret")
    if err != nil {
        // HandleError(err)
    }
    
    lsRes, err := client.ListBuckets()
    if err != nil {
        // HandleError(err)
    }
    
    for _, bucket := range lsRes.Buckets {
        fmt.Println("Buckets:", bucket.Name)
    }

Create Bucket

    client, err := oss.New("Endpoint", "AccessKeyId", "AccessKeySecret")
    if err != nil {
        // HandleError(err)
    }
    
    err = client.CreateBucket("my-bucket")
    if err != nil {
        // HandleError(err)
    }

Delete Bucket

    client, err := oss.New("Endpoint", "AccessKeyId", "AccessKeySecret")
    if err != nil {
        // HandleError(err)
    }
    
    err = client.DeleteBucket("my-bucket")
    if err != nil {
        // HandleError(err)
    }

Put Object

    client, err := oss.New("Endpoint", "AccessKeyId", "AccessKeySecret")
    if err != nil {
        // HandleError(err)
    }
    
    bucket, err := client.Bucket("my-bucket")
    if err != nil {
        // HandleError(err)
    }
    
    err = bucket.PutObjectFromFile("my-object", "LocalFile")
    if err != nil {
        // HandleError(err)
    }

Get Object

    client, err := oss.New("Endpoint", "AccessKeyId", "AccessKeySecret")
    if err != nil {
        // HandleError(err)
    }
    
    bucket, err := client.Bucket("my-bucket")
    if err != nil {
        // HandleError(err)
    }
    
    err = bucket.GetObjectToFile("my-object", "LocalFile")
    if err != nil {
        // HandleError(err)
    }

List Objects

    client, err := oss.New("Endpoint", "AccessKeyId", "AccessKeySecret")
    if err != nil {
        // HandleError(err)
    }
    
    bucket, err := client.Bucket("my-bucket")
    if err != nil {
        // HandleError(err)
    }
    
    lsRes, err := bucket.ListObjects()
    if err != nil {
        // HandleError(err)
    }
    
    for _, object := range lsRes.Objects {
        fmt.Println("Objects:", object.Key)
    }

Delete Object

    client, err := oss.New("Endpoint", "AccessKeyId", "AccessKeySecret")
    if err != nil {
        // HandleError(err)
    }
    
    bucket, err := client.Bucket("my-bucket")
    if err != nil {
        // HandleError(err)
    }
    
    err = bucket.DeleteObject("my-object")
    if err != nil {
        // HandleError(err)
    }

Complete Example

More example projects can be found at 'src\github.com\aliyun\aliyun-oss-go-sdk\sample' under the installation path of the OSS Go SDK (the first path of the GOPATH variable). The directory contains example projects. Or you can refer to the example objects in the sample directory under 'https://github.com/aliyun/aliyun-oss-go-sdk'.

Running Example

  • Copy the example file. Go to the installation path of OSS Go SDK (the first path of the GOPATH variable), enter the code directory of the OSS Go SDK, namely 'src\github.com\aliyun\aliyun-oss-go-sdk', and copy the sample directory and sample.go to the src directory of your test project.
  • Modify the endpoint, AccessKeyId, AccessKeySecret and BucketName configuration settings in sample/config.go.
  • Run 'go run src/sample.go' under your project directory.

Contacting us

Author

License

aliyun-oss-go-sdk's People

Contributors

baiyubin avatar baiyubin2020 avatar dengwu12 avatar franklouwers avatar hangzws avatar huiguangjun avatar kkuai avatar langwan avatar lgyaxx avatar liyanzhang505 avatar mars-coder avatar mzmuer avatar yangzong18 avatar yankunsam avatar yizhihouzi avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

aliyun-oss-go-sdk's Issues

当有大量请求时,会出现大量的ESTABLISHED连接

由于向OSS发起的请求都是新建一个client和transport,导致当向OSS发起大量请求的时候,会出现大量的ESTABLISHED连接。
当我并发40个GO程的时候,很快就产生了3万多个ESTABLISHED连接,导致其后请求大量失败。
我的解决方法,是将transport的DisbaleKeepAlives置为true,连接就只维持在了50个左右。
建议:
1、可以共用一个client和transport;
2、开启transport的DisableKeepAlive,让transport不再缓存persistConn

Can't change an (expired) STS token inside an OSS client

There is a problem with STS tokens we encounter. Our use case is similar to the one documented here https://www.alibabacloud.com/help/doc-detail/54579.htm?spm=a2c63.p38356.a3.2.173c3461fmtaEf but with Golang instead of Python.

The issue is with accessing the OSS API from Golang code with STS tokens, this is certainly possible, but the STS tokens are expiring, and there is no way to change the tokens inside an existing OSS client, although in the https://github.com/aliyun/alibaba-cloud-sdk-go the tokens are refreshed on-demand.

Are there any plans to change this behavior our this is not the correct way to use the API? Currently, we have to throw away the OSS client every time the token expires, and users of the API have to implement the same pattern at every use case.

I open this issue here but been advised to move it here.

english godoc

Hi,

Thanks for the great library.
Is there a plan to add english docs?

English docs will broaden the user base of this library

Thanks

PutObject失败,返回错误里字段都为空

从美西连美西的endpoint,一百个go routine并发调用同一个oss client,调用bucket.PutObject方法,有很多失败,而且失败返回的err信息是:
oss: service returned error: StatusCode=0, ErrorCode=, ErrorMessage=, RequestId=
无法进一步判断问题,是否是错误信息不对?

使用断点分片下载有时候会出现the process cannot access the file because it is being used by another process错误

使用断点分片下载,如下

err:=bucket.DownloadFile(src, dst, 5*1024*1024, oss.Routines(20), oss.Checkpoint(true, ""))

当捕捉err错误重试下载时,有时候又会遇到

rename xxx.temp xxx:the process cannot access the file because it is being used by another process.

另外有时候还会出现err错误

unexpected EOF

是文件描述符没有关闭吗?

看了下代码,rename是在完成下载后进行的
https://github.com/aliyun/aliyun-oss-go-sdk/blob/master/oss/download.go#L413

func (cp *downloadCheckpoint) complete(cpFilePath, downFilepath string) error {
	os.Remove(cpFilePath)
	return os.Rename(downFilepath, cp.FilePath)
}

How do we get the content length of file ?

Sorry, I am new to golang. I have some questions.

Now I have tried to get GetObject from the bucket method(oss) and it now only returns the io.ReadCloser. And I can get the file but fail to get the file size, because i will act like proxy server to return the file from my http server and thus need to set the content-length in header.

Do you know how to get content length of file or file size ?

I saw in the bucket.go, it will process like in method DoGetObject
resp.Body = ioutil.NopCloser(TeeReader(resp.Body, crcCalc, contentLen, listener, nil))

And return Response.Body to GetObject method.

Is it possible to expose the contentLen too ?

Or it needs us to have further process to read/parse it in order to get the file size ?

Thanks

上传后回调通知

请问Go的sdk中如何实现上传后回调通知?

其他语言都有demo,Golang这边sdk里没有写这部分内容。

Delete objects doesn't work

the object key is : /product/1234-A-ABC.jpg

        bucket, err := this.AliyunOSSClient.Bucket(aliyun.BUCKET_IMG)
	if err != nil {
		return err
	}
	objectKeys := strings.Split(params.Key, ",")

	deleteObjectsResult, err :=  bucket.DeleteObjects(objectKeys, oss.DeleteObjectsQuiet(false))
	if err != nil {
		return err
	}
	if len(deleteObjectsResult.DeletedObjects) <= 0 {
		return errors.New("Nothing deleted")
	}
	jcllog.Info.Printf("Delete file success %s\n", strings.Join(deleteObjectsResult.DeletedObjects, ","))
	return nil

It doesn't return any error, but nothing deleted from Bucket

urlMaker访问权限不足

conn_test.go里面有个测试方法

func (s *OssConnSuite) TestAuth(c *C) {
	endpoint := "https://github.com/"
	cfg := getDefaultOssConfig()
	um := urlMaker{}
	um.Init(endpoint, false, false)
	conn := Conn{cfg, &um, nil}
	uri := um.getURL("bucket", "object", "")
	req := &http.Request{
		Method:     "PUT",
		URL:        uri,
		Proto:      "HTTP/1.1",
		ProtoMajor: 1,
		ProtoMinor: 1,
		Header:     make(http.Header),
		Host:       uri.Host,
	}

	req.Header.Set("Content-Type", "text/html")
	req.Header.Set("Date", "Thu, 17 Nov 2005 18:49:58 GMT")
	req.Header.Set("Host", endpoint)
	req.Header.Set("X-OSS-Meta-Your", "your")
	req.Header.Set("X-OSS-Meta-Author", "[email protected]")
	req.Header.Set("X-OSS-Magic", "abracadabra")
	req.Header.Set("Content-Md5", "ODBGOERFMDMzQTczRUY3NUE3NzA5QzdFNUYzMDQxNEM=")

	conn.signHeader(req, um.getResource("bucket", "object", ""))
	testLogger.Println("AUTHORIZATION:", req.Header.Get(HTTPHeaderAuthorization))
}

注意

uri := um.getURL("bucket", "object", "")

以及

	conn.signHeader(req, um.getResource("bucket", "object", ""))

这2行代码

urlMaker无法在外部使用,也就导致了我们需要自己构造url以及resource.能否修改一下urlMaker的可访问性

支持context.Context

现在主流都是一个参数传context.Context, 用于取消请求或者设置超时

是否考虑支持一下?

通过 SignURL() 生成的 url 使用其他工具或语言无法上传文件

通过SignURL() 生成 PUT 类型的 url, 在Postman 或是java的client中都无法上传文件。
返回结果如下:

<?xml version="1.0" encoding="UTF-8"?>
<Error>
    <Code>SignatureDoesNotMatch</Code>
    <Message>The request signature we calculated does not match the signature you provided. Check your key and signing method.</Message>
    <RequestId>5B965349863BEA2476DF97B5</RequestId>
    <HostId>llvision.oss-cn-beijing.aliyuncs.com</HostId>
    <OSSAccessKeyId>LTAI5QcNcJYoGMGN</OSSAccessKeyId>
    <SignatureProvided>1I3rvvG27u/Wb5ZkHtSj5IdCLIM=</SignatureProvided>
    <StringToSign>PUT

text/plain
1536581960
/llvision/sample3.png</StringToSign>
    <StringToSignBytes>50 55 54 0A 0A 74 65 78 74 2F 70 6C 61 69 6E 0A 31 35 33 36 35 38 31 39 36 30 0A 2F 6C 6C 76 69 73 69 6F 6E 2F 73 61 6D 70 6C 65 33 2E 70 6E 67 </StringToSignBytes>
</Error>

The specified key does not exist.

  • endpoint: oss-cn-shanghai.aliyuncs.com。

  • accessKeyId: 阿里云超级管理员key。

  • accessKeySecret: 对应的secret。

  • error message: oss: service returned error: StatusCode=404, ErrorCode=NoSuchKey, ErrorMessage=The specified key does not exist., RequestId=5A44D439888183B3BCB9F742

如何设置 http 的超时时间?

我看到 conf.go 里有默认的超时时间。但新建 client 时没办法传入自己的 config。

请问有什么办法控制 http 建立连接的超时时间?

如何获取下载进度

我期望获在分片下载方法里取到文件的大小与当前已经传输的字节数。现在的 SDK 推荐怎么做呢?

Why are the author & license information removed?

Why do you copy the code from aliyun-beta and import them under you account, instead of transferring it?

Why do you remove the original README.md together with the author information and LICENSE file from the repository?

This is very unprofessional! Please correct the mistake as soon as possible: remove the repository from github.com/aliyun and transfer from github.com/aliyun-beta and retain the author and license information, otherwise I will submit the issue to Github officials.

upload progress API

希望有接口能获取实时的上传进度,或者添加一个接口可以从外界传入md5值而不用内部ReadAll一遍

多出来的q

.../sample/bucket_referer.go:43:20:client.GetBucketReferqer undefined

Config里的RetryTimes实际没有生效

oss.Client的Config里面有RetryTimes并默认5,但在其余代码中未见使用,实际上是没有重试?

实际生产中暂时只能依靠设置小的Config.Timeout并在外部重试解决

go run sample.go no such host exit status 255

Endpoint、AccessKeyId、AccessKeySecret、BucketName已经设置。这些值,我同事用java是可以用的。现在,我需要用golang来进行设置。但是,运行示例代码的时候,出现了no such host exit status 255的错误。

翻看 OSS错误响应 没有错误代码255的解释。

运行示例代码

运行示例
下载代码文件。从GitHub下载OSS Go SDK的代码。
修改sample/config.go里的Endpoint、AccessKeyId、AccessKeySecret、BucketName配置。
执行go run sample.go。

➜  aliyun-oss-go-sdk git:(master) ✗ go run sample.go 
occurred error: Put http://<my-bucket>.<endpoint>/: dial tcp: lookup <my-bucket>.<endpoint>: no such host
exit status 255

相关信息

➜  aliyun-oss-go-sdk git:(master) ✗ lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 16.04.3 LTS
Release:	16.04
Codename:	xenial
➜  aliyun-oss-go-sdk git:(master) ✗ go version
go version go1.9.1 linux/amd64

oss.New有bug

client, err := oss.New(endpoint, ...)

如果endpoint 无法访问,也不会返回err 。

可否支持STS token生成?

看了官方STS SDK文档, 其他语言都有SDK, 而go语言无相关SDK或使用说明?
可否在go sdk中加上此功能?

Error on ListObjects: Bucket must be addressed using the specified endpoint

When calling oss.Bucket.ListObjects, it got the following error:

oss: service returned error: StatusCode=403, ErrorCode=AccessDenied, ErrorMessage=The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint., RequestId=5A408B4B42BD0241CE407E2B

And calling codes are as following:

bucket, err := factory.GetOssClient().Bucket(name)
......
objectMarker := ""
for {
objects, err := bucket.ListObjects(oss.Marker(objectMarker))
......

Contexts:

  1. This error occurs only if regionId = "eu-central-1".
  2. The following API can be called successfully (for all regions):

factory.GetOssClient().ListBuckets(oss.Marker(bucketMarker))

DownloadFile 的中止

我在制作客户端的下载工具,Bucket.DownloadFile 解决了断点续传,是很方便的方法。

但现在的 Bucket.DownloadFile 一旦调用后会一直下载完成,我期望在某些时刻(比如网速低时)停止 Bucket.DownloadFile

这种情况如何通过 SDK 实现呢?

分片下载大文件200多MB,出现 “i/o timeout”

参数设置connectTimeoutSec=30,readWriteTimeout=120,分片大小:5M,Checkpoint=false
错误信息如下:
read tcp 198.9.4.207:53567->101.201.182.67:80: i/o timeout

另:通过“OSS控制台客户端(Windows)”下载正常

Option.go的一个参数key的问题

我在调用Option.go文件中的下述方法(line:282——284)
func Process(value string) Option {
    return addParam("X-Oss-Process",value)
}
生成一个option对象时,作为参数执行
signURL,_ :=bucket.SignUrl("objectKey","GET",300,option),
发现生成的SignUrl不可使用。
经过与官方帮助文档的对比发现,
将"X-Oss-Process"改为"x-oss-process",
生成的SignUrl可以使用。
但是这种方式,使用时会比较麻烦,
我们的团队开发者在使用工具编译之前,都要在Option.go中更改这个字符串。
请问,是否可以提供此问题的解决方法,非常感谢!

Seperate the tests which rely on network

As we are going to package this library in Debian[1], we find this package's tests is hard to run since network access is not allowed in our build server, and we don't have the access key.
Although we can skip the tests(what we currently do[2]), it's not good for package quality in Debian. Thus I hope you can provide tests that don't need network.

[1] https://bugs.debian.org/869863
[2] https://anonscm.debian.org/cgit/pkg-go/packages/golang-github-aliyun-aliyun-oss-go-sdk.git/tree/debian/rules#n6

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.