ncw / swift Goto Github PK
View Code? Open in Web Editor NEWGo language interface to Swift / Openstack Object Storage / Rackspace cloud files (golang)
License: MIT License
Go language interface to Swift / Openstack Object Storage / Rackspace cloud files (golang)
License: MIT License
Openstack Swift API will return HTTP 404 not only when trying to access a nonexistent object but also when trying to put an object in a nonexistent container. Hence, the description for 404 "Object Not Found" will be bizarre when returned by an ObjectPut()
"Container or Object Not Found" may be a proper one.
This allows a Content-Length header to be set in responses from middleware using this library as a data source.
(Pull request forthcoming)
This is not an issue but more of a question. I am new to using this library and just started reading it. If there is another place I should be asking questions, please let me know.
Question - What does "Timeout time.Duration // Data channel timeout (default 60s)" field in Connection type get used for?
I'm getting this sometimes on ContainerCreate:
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xb code=0x1 addr=0x0 pc=0x566047]
goroutine 50 [running]:
runtime.panic(0x74cf80, 0x96f3d3)
/usr/lib/golang/src/pkg/runtime/panic.c:279 +0xf5
net/url.(*URL).ResolveReference(0x0, 0xc208029650, 0xc208029650)
/usr/lib/golang/src/pkg/net/url/url.go:648 +0x207
net/url.(*URL).Parse(0x0, 0x0, 0x0, 0xc20813ab70, 0x0, 0x0)
/usr/lib/golang/src/pkg/net/url/url.go:636 +0x8d
github.com/ncw/swift.(*Connection).Call(0xc20811c000, 0x0, 0x0, 0xc208137b0c, 0x4, 0x0, 0x0, 0x78feb0, 0x3, 0x0, ...)
/.../src/github.com/ncw/swift/swift.go:437 +0x1d0
github.com/ncw/swift.(*Connection).storage(0xc20811c000, 0xc208137b0c, 0x4, 0x0, 0x0, 0x78feb0, 0x3, 0x0, 0x0, 0xc208022ae0, ...)
/.../src/github.com/ncw/swift/swift.go:521 +0x142
github.com/ncw/swift.(*Connection).ContainerCreate(0xc20811c000, 0xc208137b0c, 0x4, 0x0, 0x0, 0x0)
/.../src/github.com/ncw/swift/swift.go:975 +0xbd
$MYPROGRAM/resource.swiftResource.Upload(0xc20811c000, 0xc208137b0c, 0x4, 0xc208137b11, 0x1c, 0xc208043e00, 0x3a, 0xc2080caa20, 0x0, 0x0)
/.../src/$MYPROGRAM/resource/swift.go:130 +0xf4
$MYPROGRAM/resource.(*swiftResource).Upload(0xc208022f60, 0xc208043e00, 0x3a, 0xc2080caa20, 0x0, 0x0)
...unrelated stack frames...
From the rest of the stack traces, no other goroutines are running in any code in the swift library.
Using go 1.3.3 with OpenStack Swift Grizzly (and OpenStack Keystone Grizzly for auth), ncw/swift as of 71da36c.
Isn't the url
package being shadowed by the url
variable declared on line 436? This seems to universally panic when targetUrl does not contain a scheme. That said, I don't know why it wouldn't contain a scheme in the first place.
Unlike #26, I have no other interesting log lines around this panic.
The memcached for Keystone was not up during the time that this failed; it was probably not returning useful data.
I see the documentation has various authentication examples, but none for openstack?
I am confused over the configuration variables and methods to get these variables, and a doc entry would be awesome.
Thanks
The md5 is actually the md5 sum of the md5sums of the parts. This isn't very helpful to detect a corrupted download, so don't check md5 for objects with X-Object-Manifest
set.
At the moment the md5 check always fails for an object with X-Object-Manifest
.
Hi!
Do you have any plans to support bulk extract-archive and bulk delete operations?
If not, I am willing to add them to swift.
Thanks.
Not so much an issue, but more of a question/request:
Have you ever considered making this into a full Go Swift Client, as opposed to "just" a library?
Feel free to close this "issue" if you feel it's outside of scope or not of interest pursuing.
Do you have plans to "export " the Authenticator interface?
I'm working on Hubic(OVH) API but i have to do a little hack to get this "working"
here-> dadc254
Thanks
i have old version of swift and can't get ObjectGet because of error:
parsing time "" as "Mon, 02 Jan 2006 15:04:05 GMT": cannot parse "" as "Mon"
as i understand default TimeFormat does not equal of my output, how can i deal with this?
For restic, we'll switch to dep
shortly, and it would be very helpful if you could tag a release. Thanks!
https://dave.cheney.net/2016/06/24/gophers-please-tag-your-releases
Hello,
I'm new very to go and I just installed it on ubuntu 14.04 using apt-get.
Here is my go version go version xgcc (Ubuntu 4.9.3-0ubuntu4) 4.9.3 linux/amd64
my test script:
package main
import (
"fmt"
"bytes"
"github.com/ncw/swift"
)
func main() {
// Create a v1 auth connection
c := &swift.Connection{
// This should be your username
UserName: "username",
// This should be your api key
ApiKey: "foobar",
// This should be a v1 auth url, eg
// Rackspace US https://auth.api.rackspacecloud.com/v1.0
// Rackspace UK https://lon.auth.api.rackspacecloud.com/v1.0
// Memset Memstore UK https://auth.storage.memset.com/v1.0
AuthUrl: "https://sao01.objectstorage.softlayer.net/auth/v1.0/",
}
// Authenticate
err := c.Authenticate()
if err != nil {
panic(err)
}
/*
// List all the containers
containers, err := c.ContainerNames(nil)
fmt.Println(containers)
// etc...
*/
// c.ObjectPut("bnovo-201608", "testgox", "XABC", "")
headers := swift.Headers{}
// Now set content size correctly
contents := bytes.NewBufferString("ABCDE")
//headers["Content-Length"] = strconv.FormatInt(CONTENT_SIZE, 10)
h, err := c.ObjectPut("saopaulo", "testgo", contents, false, "", "text/plain", headers)
if err != nil {
fmt.Println(err)
}
fmt.Println(h)
// as above...
}
When I run this, sometimes it work, sometimes not:
renato@renato-Inspiron-5420:~/projetos/go/src/github.com/renatocron/hello-world-swift$ go run hello.go
map[X-Trans-Id:tx4c759a66c5924032b88fb-0057c2a657 Date:Sun, 28 Aug 2016 08:52:39 GMT Last-Modified:Sun, 28 Aug 2016 08:52:40 GMT Content-Type:text/html; charset=UTF-8 Content-Length:0 Etag:2ecdde3959051d913f61b14579ea136d]
renato@renato-Inspiron-5420:~/projetos/go/src/github.com/renatocron/hello-world-swift$ go run hello.go
signal: Segmentation fault (core dumped)
renato@renato-Inspiron-5420:~/projetos/go/src/github.com/renatocron/hello-world-swift$ go run hello.go
signal: Segmentation fault (core dumped)
renato@renato-Inspiron-5420:~/projetos/go/src/github.com/renatocron/hello-world-swift$ go run hello.go
signal: Segmentation fault (core dumped)
renato@renato-Inspiron-5420:~/projetos/go/src/github.com/renatocron/hello-world-swift$ go run hello.go
signal: Segmentation fault (core dumped)
renato@renato-Inspiron-5420:~/projetos/go/src/github.com/renatocron/hello-world-swift$ go run hello.go
^[[Amap[X-Trans-Id:tx6725b0e88b854f1488f73-0057c2a65b Date:Sun, 28 Aug 2016 08:52:43 GMT Last-Modified:Sun, 28 Aug 2016 08:52:44 GMT Content-Type:text/html; charset=UTF-8 Content-Length:0 Etag:2ecdde3959051d913f61b14579ea136d]
renato@renato-Inspiron-5420:~/projetos/go/src/github.com/renatocron/hello-world-swift$ go run hello.go
signal: Segmentation fault (core dumped)
renato@renato-Inspiron-5420:~/projetos/go/src/github.com/renatocron/hello-world-swift$ go run hello.go
signal: Segmentation fault (core dumped)
renato@renato-Inspiron-5420:~/projetos/go/src/github.com/renatocron/hello-world-swift$ go run hello.go
signal: Segmentation fault (core dumped)
I'm using sao01.objectstorage.softlayer.net and I'm running mtr to see if I have any package loss, but that do not seem to happen too often (none in a few minutes)
Hi,
The documentation for ObjectDelete() states that it may return ObjectDoesNotExist.
However, looking at the errObjectMap, it seems that that error does not exist.
Maybe the comment should read:
// May return ObjectNotFound if the object isn't found
Inspecting the 'curl' results for deleting a non-existing object name seems to, in fact, return a 404. So a ObjectNotFound seems right.
Thanks!
Livio
Checking MD5SUMs when downloading a static large object doesn't work.
The library should be checking for X-Static-Large-Object: True
and disabling the check - swift doesn't seem to be putting the X-Object-Manifest
on like it used to for dynamic large objects.
I have a very simple script where I try to BulkUpload a tar.gz unfortunately rackspace return a BadRequest. I am wondering if it is that rackspace do not support this feature or it is because I am not using the api as it should ?
package main [1/39]
import (
"compress/gzip"
"fmt"
"github.com/ncw/swift"
"os"
)
func main() {
fmt.Println("Connecting to rackspace:", os.Getenv("UserName"))
// Create a connection
c := swift.Connection{
UserName: os.Getenv("UserName"),
ApiKey: os.Getenv("ApiKey"),
AuthUrl: "https://identity.api.rackspacecloud.com/v1.0",
}
// Authenticate
err := c.Authenticate()
if err != nil {
panic(err)
}
// List all the containers
containers, err := c.ContainerNames(nil)
fmt.Println(containers)
file, err := os.Open("/home/yml/golang/cloud_importer/testNew.tar.gz")
if err != nil {
fmt.Println("An error occured while opening the tar.gz file")
}
defer file.Close()
dataStream, err := gzip.NewReader(file)
if err != nil {
fmt.Println("An error occured while creating the data stream")
}
// Bulk upload to rackspace
bulkUploadResult, err := c.BulkUpload("myContainer", dataStream, swift.UploadTarGzip, nil)
if err != nil {
fmt.Println("An error occured while bulk uploading the tar.gz\n", err)
}
fmt.Println("Uploaded: ", bulkUploadResult)
}
For some reason rackspace return a BadRequest
yml@smithsonian$ go run src/cloud_importer.go
Connecting to rackspace: MyUsername
[myContainer, ... all my other containers]
An error occured while bulk uploading the tar.gz
Bad Request
Uploaded: {0 map[] map[Content-Type:application/json X-Trans-Id:txf46e68078c12414f8d2ea-005245590cord1 Date:Fri, 27 Sep 2013 10:08:12 GMT]}
I've been writing a lot of code to build SLO manifest files from smaller chunks of data. I need the MD5 hash of each chunk for its manifest entry. Right now, even though the connection.ObjectCreate
can check the MD5 hash for me, I can't get its value, so I have to redundantly compute it. I know that I can pull it down after the upload completes with connection.Object
, but I'd rather not make the HTTP request. Could a method be added to the ObjectCreateFile
struct that exposes the computed hash? Perhaps something like:
// Etag returns the MD5 hash of a successful upload. If it is called before
// an upload is closed or on an unsuccessful upload, it will return an error
// and an undefined byte slice.
func (file *ObjectCreateFile) Etag() ([]byte, error)
Keystone v3 has introduced trust. It is necessary to add support for it.
As far as I can see a Connection is not thread safe. Is that a correct assertion?
fr, _, err := storage.ObjectOpen(foo, bar, false, swift.Headers{})
_, err = fr.Seek(0, os.SEEK_END)
if err != nil {
fmt.Println(err.Error())
}
Expected:
err == nil
Actual:
416 Requested Range Not Satisfiable
Comment: this situation is not covered by tests as well. Maybe I'm missing something?
Hello,
Thank for your work. I've setup an openstack swift with keystone identity and kerberos. My usual workflow to access swift is an 3 step process.
Using the lib, I just have to do step 1 and 2 and then provide AuthToken, StorageUrl and bypass authenticate call to get a working example.
It works fine
Is this intentional, or is it working by accident ?
I've done some small update to rclone to get it working too.
working example.
package main
import (
"fmt"
"github.com/ncw/swift"
)
func main() {
auth_token := "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
storage_url := "https://api.example.net/v1/AUTH_project1"
c := &swift.Connection{
AuthVersion: 0,
AuthToken: auth_token,
StorageUrl: storage_url,
}
// Authenticate XXX
//err := c.Authenticate()
//if err != nil {
// panic(err)
//}
info, headers, err := c.Account()
if err != nil {
panic(err)
}
fmt.Println(info)
fmt.Println(headers)
}
I'm currently working on this CLI used to interact with SoftLayer Object Storage, specifically on IBM Bluemix.
To get a Connection object, I need to authenticate with each command. This requires a couple of requests to bluemix.net unless I have my Object Storage service credentials saved locally. If I'm not caching creds, it takes ~5-8 seconds for each command. And I often find myself having to string commands together, so those seconds add up fast.
I'm not thrilled about caching service credentials that don't change, and would feel a lot better about storing the expiring Auth Token and taking the longer time to get a new one as needed. Is there currently a way for me to get an active Connection object with just my Auth Token and Storage URL or can I only obtain one with the service credentials?
Thanks!
Is there any plan to add storage polocy support ?
logs:
It's the same error repeated in each test:
+ go test -compiler gc -ldflags '' github.com/ncw/swift
Waiting for server to start . Started
2017/07/28 14:16:45 http: panic serving 127.0.0.1:60710: runtime error: invalid memory address or nil pointer dereference
goroutine 156 [running]:
net/http.(*conn).serve.func1(0x11a18180)
/usr/lib/golang/src/net/http/server.go:1693 +0x9c
panic(0x2d2768, 0x49d178)
/usr/lib/golang/src/runtime/panic.go:491 +0x204
sync/atomic.loadUint64(0x11a7410c, 0x119f6380, 0x11a68144)
/usr/lib/golang/src/sync/atomic/64bit_arm.go:10 +0x3c
github.com/ncw/swift/swifttest.(*SwiftServer).serveHTTP(0x11a740f0, 0x472fd8, 0x11a20360, 0x119e6200)
/builddir/build/BUILDROOT/golang-github-ncw-swift-0-0.8.gite3042b2.fc27.arm/usr/share/gocode/src/github.com/ncw/swift/swifttest/server.go:768 +0x90
github.com/ncw/swift/swifttest.NewSwiftServer.func1(0x472fd8, 0x11a20360, 0x119e6200)
/builddir/build/BUILDROOT/golang-github-ncw-swift-0-0.8.gite3042b2.fc27.arm/usr/share/gocode/src/github.com/ncw/swift/swifttest/server.go:1093 +0x34
net/http.HandlerFunc.ServeHTTP(0x119ec050, 0x472fd8, 0x11a20360, 0x119e6200)
/usr/lib/golang/src/net/http/server.go:1914 +0x34
net/http.serverHandler.ServeHTTP(0x11a04000, 0x472fd8, 0x11a20360, 0x119e6200)
/usr/lib/golang/src/net/http/server.go:2610 +0x74
net/http.(*conn).serve(0x11a18180, 0x473528, 0x119f66c0)
/usr/lib/golang/src/net/http/server.go:1797 +0x57c
created by net/http.(*Server).Serve
/usr/lib/golang/src/net/http/server.go:2711 +0x208
--- FAIL: TestTransport (0.00s)
swift_test.go:321: Auth failed Get http://127.0.0.1:33877/v1.0: EOF
The tests work fine in other arches like ppc64le or x86_64.
Any idea of what could be causing this?
I get this line printed multiple times per second:
swift: received multiple values for header "Accept-Ranges"
from: https://github.com/ncw/swift/blob/master/swift.go#L202
(while using rclone). This makes the output completely unusable, and there is nothing that I can do to fix it, so it is quite a useless warning to me. Would there be any way to silence it?
Also run the Internal tests
Here is an outline
I have the following code:
f, err := os.Create(fmt.Sprintf("downloads/%s", obj_path))
if err != nil {
return err
}
defer f.Close()
h, err := conn.ObjectGet(*bucket, obj_path, f, true, nil)
if err != nil {
fmt.Printf("\nERROR: Problem downloading %s\n", obj_path)
fmt.Println(h)
fmt.Println(err)
return err
}
fmt.Printf(" downloaded: %s\n", obj_path)
fmt.Println(h)
This code works just fine when we have compression turned off on the NetScaler (which is between the client and the swift proxy).
Example Output:
downloaded: 100.txt
map[X-Timestamp:1434554271.52524 Content-Type:text/plain; charset=utf-8 X-Trans-Id:txc301ab2320a64791b978e-0055818fab Date:Wed, 17 Jun 2015 15:18:03 GMT Content-Length:102400 Accept-Ranges:bytes Last-Modified:Wed, 17 Jun 2015 15:17:52 GMT Etag:4c6426ac7ef186464ecbb0d81cbfcb1e]
This code fails when we turn on compression on the NetScaler.
Example Output:
ERROR: Problem downloading 100.txt
map[Content-Type:text/plain; charset=utf-8 X-Trans-Id:txca5e91027dce4491820f5-0055819411 Last-Modified:Wed, 17 Jun 2015 15:36:45 GMT X-Timestamp:1434555404.78653 Etag:4c6426ac7ef186464ecbb0d81cbfcb1e Date:Wed, 17 Jun 2015 15:36:49 GMT Cache-Control:private Cteonnt-Length:102400 Accept-Ranges:bytes]
Bad Header 'Content-Length': '': strconv.ParseInt: parsing "": invalid syntax
Notice that when the code works, we have Content-Length:102400
Notice that when the code fails, we have Cteonnt-Length:102400
and Content-Length
is not set.
I have tested other libraries in Java and Python and they do not fail in this same case. In doing some research, this seems to be a standard way to specify if the request was compressed or not. The Content-Length
header is returned jumbled if the request was compressed to signify that it is reporting the compressed length.
Do you have any ideas how to address this? Right now I can upload files, but I can't download any files when compression is turned on.
Hi, with the 2.2 (Juno) release of Swift it finally supports Identity v3 support. Would it be possible to extend the existing AUTH system to support this?
V3 support was recently added to Gophercloud as well, so pointers on how to implement this should be easily accessible.
https://github.com/rackspace/gophercloud/tree/master/openstack/identity/v3
Connection.ObjectCopy
(and, therefore, Connection.ObjectMove
) issues only a single COPY
request to copy an object. However, this leads to large objects being concatenated into a single object. The copy operation is therefore needlessly expensive, or may even fail if the object becomes too large because of the concatenation.
ObjectCopy
and ObjectMove
should (either automatically or upon explicit choice of the caller) detect large objects and copy only the manifest instead of the entire object.
This issue is especially severe because, at least for SLOs, the caller cannot work around this problem because of #101.
As requested I opened an issue because the Go tests fail on my OpenStack Mitaka backend with v2.0 authentication.
~/src/github.com/ncw/swift$ source openrc.sh
~/src/github.com/ncw/swift$ export SWIFT_TENANT_ID=$OS_TENANT_ID
~/src/github.com/ncw/swift$ export SWIFT_API_KEY=$OS_PASSWORD
~/src/github.com/ncw/swift$ export SWIFT_AUTH_URL=$OS_AUTH_URL
~/src/github.com/ncw/swift$ export SWIFT_TENANT=$OS_TENANT
~/src/github.com/ncw/swift$ export SWIFT_API_USER=$OS_USERNAME
~/src/github.com/ncw/swift$ go version
go version go1.8.3 linux/amd64
~/src/github.com/ncw/swift$ go test
Waiting for server to start . Started
--- FAIL: TestAuthenticateRace (1.10s)
swift_test.go:456: Auth failed Response didn't have storage url and auth token
swift_test.go:456: Auth failed Response didn't have storage url and auth token
swift_test.go:456: Auth failed Response didn't have storage url and auth token
swift_test.go:456: Auth failed Response didn't have storage url and auth token
swift_test.go:456: Auth failed Response didn't have storage url and auth token
swift_test.go:456: Auth failed Response didn't have storage url and auth token
swift_test.go:456: Auth failed Response didn't have storage url and auth token
swift_test.go:456: Auth failed Response didn't have storage url and auth token
swift_test.go:456: Auth failed Response didn't have storage url and auth token
swift_test.go:456: Auth failed Response didn't have storage url and auth token
--- FAIL: TestSerializeConnectionJson (0.11s)
swift_test.go:134: Auth failed Response didn't have storage url and auth token
--- FAIL: TestSerializeConnectionXml (0.10s)
swift_test.go:134: Auth failed Response didn't have storage url and auth token
--- FAIL: TestOnReAuth (0.10s)
swift_test.go:134: Auth failed Response didn't have storage url and auth token
--- FAIL: TestAccount (0.11s)
swift_test.go:134: Auth failed Response didn't have storage url and auth token
--- FAIL: TestAccountUpdate (0.10s)
swift_test.go:134: Auth failed Response didn't have storage url and auth token
--- FAIL: TestContainerCreate (0.10s)
swift_test.go:134: Auth failed Response didn't have storage url and auth token
--- FAIL: TestContainer (0.10s)
swift_test.go:134: Auth failed Response didn't have storage url and auth token
--- FAIL: TestContainersAll (0.10s)
swift_test.go:134: Auth failed Response didn't have storage url and auth token
--- FAIL: TestContainersAllWithLimit (0.09s)
swift_test.go:134: Auth failed Response didn't have storage url and auth token
--- FAIL: TestContainerUpdate (0.09s)
swift_test.go:134: Auth failed Response didn't have storage url and auth token
--- FAIL: TestContainerNames (0.09s)
swift_test.go:134: Auth failed Response didn't have storage url and auth token
--- FAIL: TestContainerNamesAll (0.10s)
swift_test.go:134: Auth failed Response didn't have storage url and auth token
--- FAIL: TestContainerNamesAllWithLimit (0.10s)
swift_test.go:134: Auth failed Response didn't have storage url and auth token
--- FAIL: TestObjectPutString (0.11s)
swift_test.go:134: Auth failed Response didn't have storage url and auth token
--- FAIL: TestObjectPut (0.11s)
swift_test.go:134: Auth failed Response didn't have storage url and auth token
--- FAIL: TestObjectEmpty (0.10s)
swift_test.go:134: Auth failed Response didn't have storage url and auth token
--- FAIL: TestObjectPutBytes (0.11s)
swift_test.go:134: Auth failed Response didn't have storage url and auth token
--- FAIL: TestObjectPutMimeType (0.11s)
swift_test.go:134: Auth failed Response didn't have storage url and auth token
--- FAIL: TestObjectCreate (0.13s)
swift_test.go:134: Auth failed Response didn't have storage url and auth token
--- FAIL: TestObjectGetString (0.09s)
swift_test.go:134: Auth failed Response didn't have storage url and auth token
--- FAIL: TestObjectGetBytes (0.10s)
swift_test.go:134: Auth failed Response didn't have storage url and auth token
--- FAIL: TestObjectOpen (0.11s)
swift_test.go:134: Auth failed Response didn't have storage url and auth token
^Csignal: interrupt
FAIL github.com/ncw/swift 4.409s
Just like ObjectCreate, BulkUpload should accept custom headers.
This enables users to add CORS headers to all uploaded files, for example.
I will change BulkUpload's signature and break backward-compatibility because this is a youngest API.
--- a/go/src/github.com/ncw/swift/swift.go
+++ b/go/src/github.com/ncw/swift/swift.go
@@ -1443,16 +1443,15 @@ type BulkUploadResult struct {
// See also:
// * http://docs.openstack.org/trunk/openstack-object-storage/admin/content/object-storage-extract-archive.html
// * http://docs.rackspace.com/files/api/v1/cf-devguide/content/Extract_Archive-d1e2338.html
-func (c *Connection) BulkUpload(uploadPath string, dataStream io.Reader, format string) (result BulkUploadResult, err error) {
+func (c *Connection) BulkUpload(uploadPath string, dataStream io.Reader, format string, h Headers) (result BulkUploadResult, err error) {
I have some empty files, when i try to put it to swift i get errors like:
Put http://TTTTT:80/v1/XXXXXXXXX/public/file18: EOF
this is expected like EOF error, becasue file body is empty, but how can i deal with error in function?
I can't check file size before put, because i have only io.Reader.
When a Call()
is retried because the existing token is expired, the next call with the new token will have an empty body because the reader
instance in
Line 491 in ce444d6
Reading SLO manifests requires a url.Values
of ?multipart-manifest=get
, and writing SLO manifests requires a url.Values
of ?multipart-manifest=put
. However, the only public method which accepts url.Values
is Connection.Call
which is otherwise very ugly to work with.
I had a deadlock issue in a virtualized testing environment (and thus with random latency spikes for both IO and CPU) while creating objects. By the time I realize it's locked up, the only goroutine running anything of note has this as its head of stack trace:
2 @ 0x41f639 0x41f6bb 0x433700 0x47f8bd 0x4f271f 0x4f2b8f 0x575c4c 0x4f16eb 0x4bb8e3 0x4bded4 0x4bbb38 0x4b9ee3 0x4b9d78 0x4bcc12 0x4bbb38 0x4ba660 0x4ba413 0x4bd201 0x409f62 0x40a3c6 0x409a6e 0x40998c 0x409659 0x406ae8 0x405ef5 0x40164f 0x40119b 0x40a947 0x41f8d0
# 0x433700 sync.runtime_Syncsemacquire+0xc0 /home/croll/local/go-1.3.3/src/pkg/runtime/sema.goc:257
# 0x47f8bd sync.(*Cond).Wait+0x9d /home/croll/local/go-1.3.3/src/pkg/sync/cond.go:62
# 0x4f271f io.(*pipe).write+0x21f /home/croll/local/go-1.3.3/src/pkg/io/pipe.go:94
# 0x4f2b8f io.(*PipeWriter).Write+0x5f /home/croll/local/go-1.3.3/src/pkg/io/pipe.go:161
# 0x575c4c github.com/ncw/swift.(*ObjectCreateFile).Write+0x1cc /$GOPATH/github.com/ncw/swift/swift.go:1062
# 0x4f16eb io.Copy+0x27b /home/croll/local/go-1.3.3/src/pkg/io/io.go:355
# 0x4bb8e3 mycompany/myproject/resource.swiftResource.Upload+0x503 /mycompany/myproject/resource/swift.go:160
...
I think this deadlock is caused by this race:
If (*ObjectCreateFile).Write()
fails its nonblocking read on file.done
(and thus passes through), it sends the byte string over to the pipeWriter half and to the hasher.
If during the same time, the request finishes unexpectedly (for example, due to a timeout), there's a time period in between that and the helper goroutine of (*Connection).ObjectCreate()
's call to close(file.done)
. The Write()
call has already made its check on file.done
, and thus blocks forever waiting for a read on file.pipeReader
.
This patch changes (*Connection).ObjectCreate()
's helper goroutine to close pipeReader when the request finishes in addition to closing file.done
, and (*ObjectCreateFile).Write()
is changed to use the io.Pipe
's signaling instead of file.done
's signaling. file.done
is still neccessary to wait for completion of the request in (*ObjectCreateFile).Close()
.
Object
does not provide for a way to set the X-Newest: true
header. Refer to the API reference for details.
I'm aware that I can give extra headers to ObjectOpen
, ObjectGet
etc., but I'm dealing with large objects and therefore cannot afford downloading the entire file.
Would it be possible to add OVH's authentication endpoint (https://auth.cloud.ovh.net/v2.0
) in the set of URLs proposed when generating a remote's configuration for Swift ?
Problem: we are using the package to integrate with a swift server that returns the Last-Modified
header with UTC
TZ instead of GMT
.
This causes time.Parse(http.TimeFormat, res.Header.Get("Last-Modified"))
to fail, because the TZ should be GMT
instead of UTC
.
We have rolled-out a temporary with a custom http.RoundTripper
that fixes the headers. However, it will be nicer to be able to configure the time format for cases like this.
Would you be open to receive a PR that makes the time format configurable?
I understand is diverging from the specs, but the reality is sometimes implementors diverge from the spec and we have to adapt.
Thanks!
Ernesto
runtime.panic(0x817500, 0x101bfc8)
#011/usr/local/go/src/pkg/runtime/panic.c:266 +0xb6
net/url.(*URL).ResolveReference(0x0, 0xc217711150, 0xc217711150)
#011/usr/local/go/src/pkg/net/url/url.go:648 +0x20c
net/url.(*URL).Parse(0x0, 0x8a1cd0, 0x0, 0x0, 0x1044104, ...)
#011/usr/local/go/src/pkg/net/url/url.go:636 +0x71
github.com/ncw/swift.(*Connection).Call(0xc2103828c0, 0x8a1cd0, 0x0, 0xc211bd5ce0, 0x1d, ...)
#011/home/goab/gocode/src/github.com/ncw/swift/swift.go:387 +0x1e0
github.com/ncw/swift.(*Connection).storage(0xc2103828c0, 0xc211bd5ce0, 0x1d, 0xc211bd5d80, 0x19, ...)
#011/home/goab/gocode/src/github.com/ncw/swift/swift.go:468 +0xd6
#011/home/goab/gocode/src/github.com/ncw/swift/swift.go:1340 +0xf8
github.com/ncw/swift.(*Connection).ObjectGet(0xc2103828c0, 0xc211bd5ce0, 0x1d, 0xc211bd5d80, 0x19, ...)
#011/home/goab/gocode/src/github.com/ncw/swift/swift.go:1373 +0x98
Looks like concurrent authentication is not thread-safe.
Hi Nick,
Can you please provide any hints how to use this library with Rackspace ORD (Chicago) region? Official docs are not very helpful: http://docs.rackspace.com/files/api/v1/cf-devguide/content/Multi_region-dl2200.html
When we try to serve a resumable object (like an audio file), ObjectOpenFile cannot Seek directly to the end.
Because in http.ServeContent size is determined that way:
size, err := content.Seek(0, io.SeekEnd)
We just add one return as a quickfix in swift.go
func (file *ObjectOpenFile) Seek(offset int64, whence int) (newPos int64, err error) {
switch whence {
case 0: // relative to start
newPos = offset
case 1: // relative to current
newPos = file.pos + offset
case 2: // relative to end
if !file.lengthOk {
return file.pos, newError(0, "Length of file unknown so can't seek from end")
}
newPos = file.length + offset
**return**
default:
The internal struct implementing the LargeObjectFile
interface keeps track of which segments it has uploaded since {Dynamic,Static}LargeObjectCreate
. When the upload fails (either during PUT of another segment or PUT of the manifest), it is useful to know which segments were already uploaded in order to perform cleanup.
Since LargeObjectFile does not have a method to expose its segments, one can only perform this cleanup by listing the uploaded segments, i.e.
names, err := conn.ObjectNameAll(segmentContainer,
&swift.ObjectsOpts{Prefix: segmentPrefix},
)
... handle err ...
result, err := conn.BulkDelete(segmentContainer, names)
However, this is prone to eventual consistency issues: The segments might not yet be visible in the container listing. I see two possible options:
SegmentNames() []string
to the LargeObjectFile
interface.Cleanup() error
or Reset() error
to the LargeObjectFile
interface that deletes all segments which were uploaded since {Dynamic,Static}LargeObjectCreate
.vendor/github.com/ncw/swift/swift.go:237: unreachable code vendor/github.com/ncw/swift/timeout_reader.go:48: unreachable code exit status 1
When no region is specified in the connection to Rackspace (with the usual connection, not the special CDN management submodule), the container list returned is always empty, although the connection is successful.
Ideally the getting started example would include this information here:
http://godoc.org/github.com/ncw/swift#ex-package
While debugging some clients during a keystone outage I noticed that goswift clients always try to authenticate twice before failing. This is due to the changes to fix #21 specifically commit f7b19fe.
I'm using swift with a v3 keystone endpoint and in that case the retry is totally unnecessary as the request does not change and is simply replayed. In general I don't like the idea that the client always retries authentication because one specific implementation (Rackspace) has a funny way of doing things.
I have been thinking about how this could be improved so that the retry is not static and depended on the specific authenticator.
Without changing too much I can think of three ways to tackle this:
Connection
. Easy to implement but would be difficult to default to the current behaviour of 1
with 0
being the default value ( I don't like tossing around -1
's)Retries(c *Connection) bool
function to the Authenticator
interface. That way the Authenticator can declare if its has alternative ways of authentication based on all config options.Authenticator
interface to Request(*Connection) (*http.Request, retry_on_failure, bool, error)
, which is a variation of 2.Any feedback on the proposal would this be something worth considering?
Connection.storage()
makes a call to Connection.Call()
passing Connection.StorageUrl
. Connection.Call()
might enter a re-authentication state, calling to Connection.Authenticate()
which does overwrite the Connection.StorageUrl.
However, Connection.Call()
will continue using the initially passed in storageUrl instead of the reading the value of Connection.StorageUrl.
The easiest way to reproduce this issue is to make any API request on a connection without first calling Authenticate().
Hi.
I am working on an OpenStack landscape in which there are many projects in the same domain.
I am using v3 auth endpoint.
While creating connection, there is no field to specify project/tenant (for v3 auth).
// Create a connection
c := swift.Connection {
UserName: username,
ApiKey: apikey,
AuthUrl: authurl,
Domain: domain, // Name of the domain (v3 auth only)
AuthVersion: 3,
//Tenant: tenant, // Name of the tenant (v2 auth only)
//ProjectId: projectid,
}
By default, its connecting to the first project in the domain.
However, I am not able to connect to my desired project in the domain.
Hello !
Per the swift docs, copying a Large Object requires the query string ?multipart-manifest=get
to be appended to the COPY
request URI. If not provided the LO copy will fail for files > 5GB or convert the source to a standard object otherwise.
Similarly, actually deleting a LO with its segments requires ?multipart-manifest=delete
to be appended to the DELETE
request URI. If not, only the manifest object will be deleted leaving orphan segments in place.
Moving a LO is a combination of the first operation with a standard delete operation (first copy the manifest file to a new destination then delete the old manifest file but not the segments).
These features does not seem supported for now, did I miss something in the lib ? If not, how do you think a good implementation of this could be added to your package ?
At the moment the v3 API returns the first storage endpoint it finds. It would be nice to be able to specify by region too.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.