Hi!

Do you have any plans to support bulk extract-archive and bulk delete operations?
If not, I am willing to add them to swift.

Thanks.

• Extract archive
• Bulk delete
• Rackspace implementations

I'm getting this sometimes on ContainerCreate:

panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xb code=0x1 addr=0x0 pc=0x566047]

goroutine 50 [running]:
runtime.panic(0x74cf80, 0x96f3d3)
        /usr/lib/golang/src/pkg/runtime/panic.c:279 +0xf5
net/url.(*URL).ResolveReference(0x0, 0xc208029650, 0xc208029650)
        /usr/lib/golang/src/pkg/net/url/url.go:648 +0x207
net/url.(*URL).Parse(0x0, 0x0, 0x0, 0xc20813ab70, 0x0, 0x0)
        /usr/lib/golang/src/pkg/net/url/url.go:636 +0x8d
github.com/ncw/swift.(*Connection).Call(0xc20811c000, 0x0, 0x0, 0xc208137b0c, 0x4, 0x0, 0x0, 0x78feb0, 0x3, 0x0, ...)
        /.../src/github.com/ncw/swift/swift.go:437 +0x1d0
github.com/ncw/swift.(*Connection).storage(0xc20811c000, 0xc208137b0c, 0x4, 0x0, 0x0, 0x78feb0, 0x3, 0x0, 0x0, 0xc208022ae0, ...)
        /.../src/github.com/ncw/swift/swift.go:521 +0x142
github.com/ncw/swift.(*Connection).ContainerCreate(0xc20811c000, 0xc208137b0c, 0x4, 0x0, 0x0, 0x0)
        /.../src/github.com/ncw/swift/swift.go:975 +0xbd
$MYPROGRAM/resource.swiftResource.Upload(0xc20811c000, 0xc208137b0c, 0x4, 0xc208137b11, 0x1c, 0xc208043e00, 0x3a, 0xc2080caa20, 0x0, 0x0)
        /.../src/$MYPROGRAM/resource/swift.go:130 +0xf4
$MYPROGRAM/resource.(*swiftResource).Upload(0xc208022f60, 0xc208043e00, 0x3a, 0xc2080caa20, 0x0, 0x0)
...unrelated stack frames...

From the rest of the stack traces, no other goroutines are running in any code in the swift library.

Using go 1.3.3 with OpenStack Swift Grizzly (and OpenStack Keystone Grizzly for auth), ncw/swift as of 71da36c.

Isn't the url package being shadowed by the url variable declared on line 436? This seems to universally panic when targetUrl does not contain a scheme. That said, I don't know why it wouldn't contain a scheme in the first place.

Unlike #26, I have no other interesting log lines around this panic.

The memcached for Keystone was not up during the time that this failed; it was probably not returning useful data.

As requested I opened an issue because the Go tests fail on my OpenStack Mitaka backend with v2.0 authentication.

restic/restic#1039 (comment)

~/src/github.com/ncw/swift$ source openrc.sh
~/src/github.com/ncw/swift$ export SWIFT_TENANT_ID=$OS_TENANT_ID
~/src/github.com/ncw/swift$ export SWIFT_API_KEY=$OS_PASSWORD
~/src/github.com/ncw/swift$ export SWIFT_AUTH_URL=$OS_AUTH_URL
~/src/github.com/ncw/swift$ export SWIFT_TENANT=$OS_TENANT
~/src/github.com/ncw/swift$ export SWIFT_API_USER=$OS_USERNAME
~/src/github.com/ncw/swift$ go version
go version go1.8.3 linux/amd64
~/src/github.com/ncw/swift$ go test
Waiting for server to start . Started
--- FAIL: TestAuthenticateRace (1.10s)
	swift_test.go:456: Auth failed Response didn't have storage url and auth token
	swift_test.go:456: Auth failed Response didn't have storage url and auth token
	swift_test.go:456: Auth failed Response didn't have storage url and auth token
	swift_test.go:456: Auth failed Response didn't have storage url and auth token
	swift_test.go:456: Auth failed Response didn't have storage url and auth token
	swift_test.go:456: Auth failed Response didn't have storage url and auth token
	swift_test.go:456: Auth failed Response didn't have storage url and auth token
	swift_test.go:456: Auth failed Response didn't have storage url and auth token
	swift_test.go:456: Auth failed Response didn't have storage url and auth token
	swift_test.go:456: Auth failed Response didn't have storage url and auth token
--- FAIL: TestSerializeConnectionJson (0.11s)
	swift_test.go:134: Auth failed Response didn't have storage url and auth token
--- FAIL: TestSerializeConnectionXml (0.10s)
	swift_test.go:134: Auth failed Response didn't have storage url and auth token
--- FAIL: TestOnReAuth (0.10s)
	swift_test.go:134: Auth failed Response didn't have storage url and auth token
--- FAIL: TestAccount (0.11s)
	swift_test.go:134: Auth failed Response didn't have storage url and auth token
--- FAIL: TestAccountUpdate (0.10s)
	swift_test.go:134: Auth failed Response didn't have storage url and auth token
--- FAIL: TestContainerCreate (0.10s)
	swift_test.go:134: Auth failed Response didn't have storage url and auth token
--- FAIL: TestContainer (0.10s)
	swift_test.go:134: Auth failed Response didn't have storage url and auth token
--- FAIL: TestContainersAll (0.10s)
	swift_test.go:134: Auth failed Response didn't have storage url and auth token
--- FAIL: TestContainersAllWithLimit (0.09s)
	swift_test.go:134: Auth failed Response didn't have storage url and auth token
--- FAIL: TestContainerUpdate (0.09s)
	swift_test.go:134: Auth failed Response didn't have storage url and auth token
--- FAIL: TestContainerNames (0.09s)
	swift_test.go:134: Auth failed Response didn't have storage url and auth token
--- FAIL: TestContainerNamesAll (0.10s)
	swift_test.go:134: Auth failed Response didn't have storage url and auth token
--- FAIL: TestContainerNamesAllWithLimit (0.10s)
	swift_test.go:134: Auth failed Response didn't have storage url and auth token
--- FAIL: TestObjectPutString (0.11s)
	swift_test.go:134: Auth failed Response didn't have storage url and auth token
--- FAIL: TestObjectPut (0.11s)
	swift_test.go:134: Auth failed Response didn't have storage url and auth token
--- FAIL: TestObjectEmpty (0.10s)
	swift_test.go:134: Auth failed Response didn't have storage url and auth token
--- FAIL: TestObjectPutBytes (0.11s)
	swift_test.go:134: Auth failed Response didn't have storage url and auth token
--- FAIL: TestObjectPutMimeType (0.11s)
	swift_test.go:134: Auth failed Response didn't have storage url and auth token
--- FAIL: TestObjectCreate (0.13s)
	swift_test.go:134: Auth failed Response didn't have storage url and auth token
--- FAIL: TestObjectGetString (0.09s)
	swift_test.go:134: Auth failed Response didn't have storage url and auth token
--- FAIL: TestObjectGetBytes (0.10s)
	swift_test.go:134: Auth failed Response didn't have storage url and auth token
--- FAIL: TestObjectOpen (0.11s)
	swift_test.go:134: Auth failed Response didn't have storage url and auth token
^Csignal: interrupt
FAIL	github.com/ncw/swift	4.409s

I have the following code:

f, err := os.Create(fmt.Sprintf("downloads/%s", obj_path))
if err != nil {
    return err
}
defer f.Close()

h, err := conn.ObjectGet(*bucket, obj_path, f, true, nil)
if err != nil {
    fmt.Printf("\nERROR: Problem downloading %s\n", obj_path)
    fmt.Println(h)
    fmt.Println(err)
    return err
}
fmt.Printf(" downloaded: %s\n", obj_path)
fmt.Println(h)

This code works just fine when we have compression turned off on the NetScaler (which is between the client and the swift proxy).

Example Output:

 downloaded: 100.txt
map[X-Timestamp:1434554271.52524 Content-Type:text/plain; charset=utf-8 X-Trans-Id:txc301ab2320a64791b978e-0055818fab Date:Wed, 17 Jun 2015 15:18:03 GMT Content-Length:102400 Accept-Ranges:bytes Last-Modified:Wed, 17 Jun 2015 15:17:52 GMT Etag:4c6426ac7ef186464ecbb0d81cbfcb1e]

This code fails when we turn on compression on the NetScaler.

Example Output:

ERROR: Problem downloading 100.txt
map[Content-Type:text/plain; charset=utf-8 X-Trans-Id:txca5e91027dce4491820f5-0055819411 Last-Modified:Wed, 17 Jun 2015 15:36:45 GMT X-Timestamp:1434555404.78653 Etag:4c6426ac7ef186464ecbb0d81cbfcb1e Date:Wed, 17 Jun 2015 15:36:49 GMT Cache-Control:private Cteonnt-Length:102400 Accept-Ranges:bytes]
Bad Header 'Content-Length': '': strconv.ParseInt: parsing "": invalid syntax

Notice that when the code works, we have Content-Length:102400

Notice that when the code fails, we have Cteonnt-Length:102400 and Content-Length is not set.

I have tested other libraries in Java and Python and they do not fail in this same case. In doing some research, this seems to be a standard way to specify if the request was compressed or not. The Content-Length header is returned jumbled if the request was compressed to signify that it is reporting the compressed length.

Do you have any ideas how to address this? Right now I can upload files, but I can't download any files when compression is turned on.

Also run the Internal tests

This is not an issue but more of a question. I am new to using this library and just started reading it. If there is another place I should be asking questions, please let me know.

Question - What does "Timeout time.Duration // Data channel timeout (default 60s)" field in Connection type get used for?

vendor/github.com/ncw/swift/swift.go:237: unreachable code vendor/github.com/ncw/swift/timeout_reader.go:48: unreachable code exit status 1

When a Call() is retried because the existing token is expired, the next call with the new token will have an empty body because the reader instance in

I have some empty files, when i try to put it to swift i get errors like:
Put http://TTTTT:80/v1/XXXXXXXXX/public/file18: EOF

this is expected like EOF error, becasue file body is empty, but how can i deal with error in function?
I can't check file size before put, because i have only io.Reader.

Problem: we are using the package to integrate with a swift server that returns the Last-Modified header with UTC TZ instead of GMT.

This causes time.Parse(http.TimeFormat, res.Header.Get("Last-Modified")) to fail, because the TZ should be GMT instead of UTC.

We have rolled-out a temporary with a custom http.RoundTripper that fixes the headers. However, it will be nicer to be able to configure the time format for cases like this.

Would you be open to receive a PR that makes the time format configurable?

I understand is diverging from the specs, but the reality is sometimes implementors diverge from the spec and we have to adapt.

Thanks!
Ernesto

I'm currently working on this CLI used to interact with SoftLayer Object Storage, specifically on IBM Bluemix.

To get a Connection object, I need to authenticate with each command. This requires a couple of requests to bluemix.net unless I have my Object Storage service credentials saved locally. If I'm not caching creds, it takes ~5-8 seconds for each command. And I often find myself having to string commands together, so those seconds add up fast.

I'm not thrilled about caching service credentials that don't change, and would feel a lot better about storing the expiring Auth Token and taking the longer time to get a new one as needed. Is there currently a way for me to get an active Connection object with just my Auth Token and Storage URL or can I only obtain one with the service credentials?

Thanks!

Not so much an issue, but more of a question/request:

Have you ever considered making this into a full Go Swift Client, as opposed to "just" a library?

Feel free to close this "issue" if you feel it's outside of scope or not of interest pursuing.

Hi.
I am working on an OpenStack landscape in which there are many projects in the same domain.
I am using v3 auth endpoint.
While creating connection, there is no field to specify project/tenant (for v3 auth).

// Create a connection
c := swift.Connection {
UserName: username,
ApiKey: apikey,
AuthUrl: authurl,
Domain: domain, // Name of the domain (v3 auth only)
AuthVersion: 3,
//Tenant: tenant, // Name of the tenant (v2 auth only)
//ProjectId: projectid,
}

By default, its connecting to the first project in the domain.
However, I am not able to connect to my desired project in the domain.

When we try to serve a resumable object (like an audio file), ObjectOpenFile cannot Seek directly to the end.
Because in http.ServeContent size is determined that way:
size, err := content.Seek(0, io.SeekEnd)

We just add one return as a quickfix in swift.go

func (file *ObjectOpenFile) Seek(offset int64, whence int) (newPos int64, err error) {
	switch whence {
	case 0: // relative to start
		newPos = offset
	case 1: // relative to current
		newPos = file.pos + offset
	case 2: // relative to end
		if !file.lengthOk {
			return file.pos, newError(0, "Length of file unknown so can't seek from end")
		}
		newPos = file.length + offset
                **return**
	default:

I've been writing a lot of code to build SLO manifest files from smaller chunks of data. I need the MD5 hash of each chunk for its manifest entry. Right now, even though the connection.ObjectCreate can check the MD5 hash for me, I can't get its value, so I have to redundantly compute it. I know that I can pull it down after the upload completes with connection.Object, but I'd rather not make the HTTP request. Could a method be added to the ObjectCreateFile struct that exposes the computed hash? Perhaps something like:

// Etag returns the MD5 hash of a successful upload. If it is called before
// an upload is closed or on an unsuccessful upload, it will return an error
// and an undefined byte slice.
func (file *ObjectCreateFile) Etag() ([]byte, error)

runtime.panic(0x817500, 0x101bfc8)
#011/usr/local/go/src/pkg/runtime/panic.c:266 +0xb6
net/url.(*URL).ResolveReference(0x0, 0xc217711150, 0xc217711150)
#011/usr/local/go/src/pkg/net/url/url.go:648 +0x20c
net/url.(*URL).Parse(0x0, 0x8a1cd0, 0x0, 0x0, 0x1044104, ...)
#011/usr/local/go/src/pkg/net/url/url.go:636 +0x71
github.com/ncw/swift.(*Connection).Call(0xc2103828c0, 0x8a1cd0, 0x0, 0xc211bd5ce0, 0x1d, ...)
#011/home/goab/gocode/src/github.com/ncw/swift/swift.go:387 +0x1e0
github.com/ncw/swift.(*Connection).storage(0xc2103828c0, 0xc211bd5ce0, 0x1d, 0xc211bd5d80, 0x19, ...)
#011/home/goab/gocode/src/github.com/ncw/swift/swift.go:468 +0xd6
#011/home/goab/gocode/src/github.com/ncw/swift/swift.go:1340 +0xf8
github.com/ncw/swift.(*Connection).ObjectGet(0xc2103828c0, 0xc211bd5ce0, 0x1d, 0xc211bd5d80, 0x19, ...)
#011/home/goab/gocode/src/github.com/ncw/swift/swift.go:1373 +0x98

Looks like concurrent authentication is not thread-safe.

This is similar to #102: Connection.ObjectDelete should offer a way to not only delete the manifest of a large object, but also its segments, especially because callers cannot do it themselves because of #101.

Here is an outline

As far as I can see a Connection is not thread safe. Is that a correct assertion?

Hi, with the 2.2 (Juno) release of Swift it finally supports Identity v3 support. Would it be possible to extend the existing AUTH system to support this?

V3 support was recently added to Gophercloud as well, so pointers on how to implement this should be easily accessible.

https://github.com/rackspace/gophercloud/tree/master/openstack/identity/v3

logs:

• i686: https://kojipkgs.fedoraproject.org//work/tasks/7956/20857956/build.log
• armv7: https://kojipkgs.fedoraproject.org//work/tasks/7958/20857958/build.log

It's the same error repeated in each test:

+ go test -compiler gc -ldflags '' github.com/ncw/swift
Waiting for server to start . Started
2017/07/28 14:16:45 http: panic serving 127.0.0.1:60710: runtime error: invalid memory address or nil pointer dereference
goroutine 156 [running]:
net/http.(*conn).serve.func1(0x11a18180)
	/usr/lib/golang/src/net/http/server.go:1693 +0x9c
panic(0x2d2768, 0x49d178)
	/usr/lib/golang/src/runtime/panic.go:491 +0x204
sync/atomic.loadUint64(0x11a7410c, 0x119f6380, 0x11a68144)
	/usr/lib/golang/src/sync/atomic/64bit_arm.go:10 +0x3c
github.com/ncw/swift/swifttest.(*SwiftServer).serveHTTP(0x11a740f0, 0x472fd8, 0x11a20360, 0x119e6200)
	/builddir/build/BUILDROOT/golang-github-ncw-swift-0-0.8.gite3042b2.fc27.arm/usr/share/gocode/src/github.com/ncw/swift/swifttest/server.go:768 +0x90
github.com/ncw/swift/swifttest.NewSwiftServer.func1(0x472fd8, 0x11a20360, 0x119e6200)
	/builddir/build/BUILDROOT/golang-github-ncw-swift-0-0.8.gite3042b2.fc27.arm/usr/share/gocode/src/github.com/ncw/swift/swifttest/server.go:1093 +0x34
net/http.HandlerFunc.ServeHTTP(0x119ec050, 0x472fd8, 0x11a20360, 0x119e6200)
	/usr/lib/golang/src/net/http/server.go:1914 +0x34
net/http.serverHandler.ServeHTTP(0x11a04000, 0x472fd8, 0x11a20360, 0x119e6200)
	/usr/lib/golang/src/net/http/server.go:2610 +0x74
net/http.(*conn).serve(0x11a18180, 0x473528, 0x119f66c0)
	/usr/lib/golang/src/net/http/server.go:1797 +0x57c
created by net/http.(*Server).Serve
	/usr/lib/golang/src/net/http/server.go:2711 +0x208
--- FAIL: TestTransport (0.00s)
	swift_test.go:321: Auth failed Get http://127.0.0.1:33877/v1.0: EOF

The tests work fine in other arches like ppc64le or x86_64.

Any idea of what could be causing this?

Would it be possible to add OVH's authentication endpoint (https://auth.cloud.ovh.net/v2.0) in the set of URLs proposed when generating a remote's configuration for Swift ?

At the moment the v3 API returns the first storage endpoint it finds. It would be nice to be able to specify by region too.

This allows a Content-Length header to be set in responses from middleware using this library as a data source.

(Pull request forthcoming)

When no region is specified in the connection to Rackspace (with the usual connection, not the special CDN management submodule), the container list returned is always empty, although the connection is successful.

Ideally the getting started example would include this information here:
http://godoc.org/github.com/ncw/swift#ex-package

Connection.ObjectCopy (and, therefore, Connection.ObjectMove) issues only a single COPY request to copy an object. However, this leads to large objects being concatenated into a single object. The copy operation is therefore needlessly expensive, or may even fail if the object becomes too large because of the concatenation.

ObjectCopy and ObjectMove should (either automatically or upon explicit choice of the caller) detect large objects and copy only the manifest instead of the entire object.

This issue is especially severe because, at least for SLOs, the caller cannot work around this problem because of #101.

Reading SLO manifests requires a url.Values of ?multipart-manifest=get, and writing SLO manifests requires a url.Values of ?multipart-manifest=put. However, the only public method which accepts url.Values is Connection.Call which is otherwise very ugly to work with.

Just like ObjectCreate, BulkUpload should accept custom headers.
This enables users to add CORS headers to all uploaded files, for example.

I will change BulkUpload's signature and break backward-compatibility because this is a youngest API.

--- a/go/src/github.com/ncw/swift/swift.go
+++ b/go/src/github.com/ncw/swift/swift.go
@@ -1443,16 +1443,15 @@ type BulkUploadResult struct {
 // See also:
 // * http://docs.openstack.org/trunk/openstack-object-storage/admin/content/object-storage-extract-archive.html
 // * http://docs.rackspace.com/files/api/v1/cf-devguide/content/Extract_Archive-d1e2338.html
-func (c *Connection) BulkUpload(uploadPath string, dataStream io.Reader, format string) (result BulkUploadResult, err error) {
+func (c *Connection) BulkUpload(uploadPath string, dataStream io.Reader, format string, h Headers) (result BulkUploadResult, err error) {

I see the documentation has various authentication examples, but none for openstack?

I am confused over the configuration variables and methods to get these variables, and a doc entry would be awesome.

Thanks

Is there any plan to add storage polocy support ?

fr, _, err := storage.ObjectOpen(foo, bar, false, swift.Headers{})
_, err = fr.Seek(0, os.SEEK_END)
	if err != nil {
		fmt.Println(err.Error())
	}

Expected:
err == nil
Actual:
416 Requested Range Not Satisfiable

Comment: this situation is not covered by tests as well. Maybe I'm missing something?

I get this line printed multiple times per second:

swift: received multiple values for header "Accept-Ranges"

from: https://github.com/ncw/swift/blob/master/swift.go#L202

(while using rclone). This makes the output completely unusable, and there is nothing that I can do to fix it, so it is quite a useless warning to me. Would there be any way to silence it?

While debugging some clients during a keystone outage I noticed that goswift clients always try to authenticate twice before failing. This is due to the changes to fix #21 specifically commit f7b19fe.

I'm using swift with a v3 keystone endpoint and in that case the retry is totally unnecessary as the request does not change and is simply replayed. In general I don't like the idea that the client always retries authentication because one specific implementation (Rackspace) has a funny way of doing things.

I have been thinking about how this could be improved so that the retry is not static and depended on the specific authenticator.

Without changing too much I can think of three ways to tackle this:

1. Make the authentication retry configurable in Connection. Easy to implement but would be difficult to default to the current behaviour of 1 with 0 being the default value ( I don't like tossing around -1's)
2. Add a Retries(c *Connection) bool function to the Authenticator interface. That way the Authenticator can declare if its has alternative ways of authentication based on all config options.
3. Change the Authenticator interface to Request(*Connection) (*http.Request, retry_on_failure, bool, error), which is a variation of 2.

Any feedback on the proposal would this be something worth considering?

The md5 is actually the md5 sum of the md5sums of the parts. This isn't very helpful to detect a corrupted download, so don't check md5 for objects with X-Object-Manifest set.

At the moment the md5 check always fails for an object with X-Object-Manifest.

Connection.storage() makes a call to Connection.Call() passing Connection.StorageUrl. Connection.Call() might enter a re-authentication state, calling to Connection.Authenticate() which does overwrite the Connection.StorageUrl.
However, Connection.Call() will continue using the initially passed in storageUrl instead of the reading the value of Connection.StorageUrl.

The easiest way to reproduce this issue is to make any API request on a connection without first calling Authenticate().

I had a deadlock issue in a virtualized testing environment (and thus with random latency spikes for both IO and CPU) while creating objects. By the time I realize it's locked up, the only goroutine running anything of note has this as its head of stack trace:

2 @ 0x41f639 0x41f6bb 0x433700 0x47f8bd 0x4f271f 0x4f2b8f 0x575c4c 0x4f16eb 0x4bb8e3 0x4bded4 0x4bbb38 0x4b9ee3 0x4b9d78 0x4bcc12 0x4bbb38 0x4ba660 0x4ba413 0x4bd201 0x409f62 0x40a3c6 0x409a6e 0x40998c 0x409659 0x406ae8 0x405ef5 0x40164f 0x40119b 0x40a947 0x41f8d0
#       0x433700        sync.runtime_Syncsemacquire+0xc0                                                        /home/croll/local/go-1.3.3/src/pkg/runtime/sema.goc:257
#       0x47f8bd        sync.(*Cond).Wait+0x9d                                                                  /home/croll/local/go-1.3.3/src/pkg/sync/cond.go:62
#       0x4f271f        io.(*pipe).write+0x21f                                                                  /home/croll/local/go-1.3.3/src/pkg/io/pipe.go:94
#       0x4f2b8f        io.(*PipeWriter).Write+0x5f                                                             /home/croll/local/go-1.3.3/src/pkg/io/pipe.go:161
#       0x575c4c        github.com/ncw/swift.(*ObjectCreateFile).Write+0x1cc                                    /$GOPATH/github.com/ncw/swift/swift.go:1062
#       0x4f16eb        io.Copy+0x27b                                                                           /home/croll/local/go-1.3.3/src/pkg/io/io.go:355
#       0x4bb8e3        mycompany/myproject/resource.swiftResource.Upload+0x503        /mycompany/myproject/resource/swift.go:160
...

I think this deadlock is caused by this race:

If (*ObjectCreateFile).Write() fails its nonblocking read on file.done (and thus passes through), it sends the byte string over to the pipeWriter half and to the hasher.
If during the same time, the request finishes unexpectedly (for example, due to a timeout), there's a time period in between that and the helper goroutine of (*Connection).ObjectCreate()'s call to close(file.done). The Write() call has already made its check on file.done, and thus blocks forever waiting for a read on file.pipeReader.

This patch changes (*Connection).ObjectCreate()'s helper goroutine to close pipeReader when the request finishes in addition to closing file.done, and (*ObjectCreateFile).Write() is changed to use the io.Pipe's signaling instead of file.done's signaling. file.done is still neccessary to wait for completion of the request in (*ObjectCreateFile).Close().

Hi,

The documentation for ObjectDelete() states that it may return ObjectDoesNotExist.
However, looking at the errObjectMap, it seems that that error does not exist.
Maybe the comment should read:

// May return ObjectNotFound if the object isn't found

Inspecting the 'curl' results for deleting a non-existing object name seems to, in fact, return a 404. So a ObjectNotFound seems right.

Thanks!

Livio

Hello,

I'm new very to go and I just installed it on ubuntu 14.04 using apt-get.
Here is my go version go version xgcc (Ubuntu 4.9.3-0ubuntu4) 4.9.3 linux/amd64

my test script:

package main


import (
    "fmt"
    "bytes"
    "github.com/ncw/swift"
)

func main() {
    // Create a v1 auth connection
    c := &swift.Connection{
        // This should be your username
        UserName: "username",
        // This should be your api key
        ApiKey: "foobar",
        // This should be a v1 auth url, eg
        //  Rackspace US        https://auth.api.rackspacecloud.com/v1.0
        //  Rackspace UK        https://lon.auth.api.rackspacecloud.com/v1.0
        //  Memset Memstore UK  https://auth.storage.memset.com/v1.0
        AuthUrl: "https://sao01.objectstorage.softlayer.net/auth/v1.0/",
    }

    // Authenticate
    err := c.Authenticate()
    if err != nil {
        panic(err)
    }

    /*
    // List all the containers
    containers, err := c.ContainerNames(nil)
    fmt.Println(containers)
    // etc...
*/

    // c.ObjectPut("bnovo-201608", "testgox", "XABC", "")
    headers := swift.Headers{}
    // Now set content size correctly
    contents := bytes.NewBufferString("ABCDE")
    //headers["Content-Length"] = strconv.FormatInt(CONTENT_SIZE, 10)
    h, err := c.ObjectPut("saopaulo", "testgo", contents, false, "", "text/plain", headers)
    if err != nil {
        fmt.Println(err)
    }
    fmt.Println(h)

    // as above...
}

When I run this, sometimes it work, sometimes not:

renato@renato-Inspiron-5420:~/projetos/go/src/github.com/renatocron/hello-world-swift$ go run hello.go 
map[X-Trans-Id:tx4c759a66c5924032b88fb-0057c2a657 Date:Sun, 28 Aug 2016 08:52:39 GMT Last-Modified:Sun, 28 Aug 2016 08:52:40 GMT Content-Type:text/html; charset=UTF-8 Content-Length:0 Etag:2ecdde3959051d913f61b14579ea136d]
renato@renato-Inspiron-5420:~/projetos/go/src/github.com/renatocron/hello-world-swift$ go run hello.go 
signal: Segmentation fault (core dumped)
renato@renato-Inspiron-5420:~/projetos/go/src/github.com/renatocron/hello-world-swift$ go run hello.go 
signal: Segmentation fault (core dumped)
renato@renato-Inspiron-5420:~/projetos/go/src/github.com/renatocron/hello-world-swift$ go run hello.go 
signal: Segmentation fault (core dumped)
renato@renato-Inspiron-5420:~/projetos/go/src/github.com/renatocron/hello-world-swift$ go run hello.go 
signal: Segmentation fault (core dumped)
renato@renato-Inspiron-5420:~/projetos/go/src/github.com/renatocron/hello-world-swift$ go run hello.go 
^[[Amap[X-Trans-Id:tx6725b0e88b854f1488f73-0057c2a65b Date:Sun, 28 Aug 2016 08:52:43 GMT Last-Modified:Sun, 28 Aug 2016 08:52:44 GMT Content-Type:text/html; charset=UTF-8 Content-Length:0 Etag:2ecdde3959051d913f61b14579ea136d]
renato@renato-Inspiron-5420:~/projetos/go/src/github.com/renatocron/hello-world-swift$ go run hello.go 
signal: Segmentation fault (core dumped)
renato@renato-Inspiron-5420:~/projetos/go/src/github.com/renatocron/hello-world-swift$ go run hello.go 
signal: Segmentation fault (core dumped)
renato@renato-Inspiron-5420:~/projetos/go/src/github.com/renatocron/hello-world-swift$ go run hello.go 
signal: Segmentation fault (core dumped)

I'm using sao01.objectstorage.softlayer.net and I'm running mtr to see if I have any package loss, but that do not seem to happen too often (none in a few minutes)

The internal struct implementing the LargeObjectFile interface keeps track of which segments it has uploaded since {Dynamic,Static}LargeObjectCreate. When the upload fails (either during PUT of another segment or PUT of the manifest), it is useful to know which segments were already uploaded in order to perform cleanup.

Since LargeObjectFile does not have a method to expose its segments, one can only perform this cleanup by listing the uploaded segments, i.e.

names, err := conn.ObjectNameAll(segmentContainer,
  &swift.ObjectsOpts{Prefix: segmentPrefix},
)
... handle err ...
result, err := conn.BulkDelete(segmentContainer, names)

However, this is prone to eventual consistency issues: The segments might not yet be visible in the container listing. I see two possible options:

1. Add a method like SegmentNames() []string to the LargeObjectFile interface.
2. Add a method Cleanup() error or Reset() error to the LargeObjectFile interface that deletes all segments which were uploaded since {Dynamic,Static}LargeObjectCreate.

Openstack Swift API will return HTTP 404 not only when trying to access a nonexistent object but also when trying to put an object in a nonexistent container. Hence, the description for 404 "Object Not Found" will be bizarre when returned by an ObjectPut()

"Container or Object Not Found" may be a proper one.

Hello !

Per the swift docs, copying a Large Object requires the query string ?multipart-manifest=get to be appended to the COPY request URI. If not provided the LO copy will fail for files > 5GB or convert the source to a standard object otherwise.

Similarly, actually deleting a LO with its segments requires ?multipart-manifest=delete to be appended to the DELETE request URI. If not, only the manifest object will be deleted leaving orphan segments in place.

Moving a LO is a combination of the first operation with a standard delete operation (first copy the manifest file to a new destination then delete the old manifest file but not the segments).

These features does not seem supported for now, did I miss something in the lib ? If not, how do you think a good implementation of this could be added to your package ?

Hello,

Thank for your work. I've setup an openstack swift with keystone identity and kerberos. My usual workflow to access swift is an 3 step process.

1. call our identity service to get an access_token
2. call keystone with the access_token, to get a auth_token and storage_url
3. call swift api with the auth_token and storage_url

Using the lib, I just have to do step 1 and 2 and then provide AuthToken, StorageUrl and bypass authenticate call to get a working example.
It works fine

Is this intentional, or is it working by accident ?
I've done some small update to rclone to get it working too.

working example.

       package main

       import (
          "fmt"

          "github.com/ncw/swift"
       )

func main() {


        auth_token  := "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
        storage_url := "https://api.example.net/v1/AUTH_project1"


        c := &swift.Connection{
                AuthVersion: 0,
                AuthToken:   auth_token,
                StorageUrl:  storage_url,
        }

        // Authenticate  XXX
        //err := c.Authenticate()
        //if err != nil {
        //        panic(err)
        //}

        info, headers, err := c.Account()
        if err != nil {
              panic(err)
        }

        fmt.Println(info)
        fmt.Println(headers)
}
     

Keystone v3 has introduced trust. It is necessary to add support for it.

I have a very simple script where I try to BulkUpload a tar.gz unfortunately rackspace return a BadRequest. I am wondering if it is that rackspace do not support this feature or it is because I am not using the api as it should ?

package main                                                                                                                                                              [1/39]

import (
        "compress/gzip"
        "fmt"
        "github.com/ncw/swift"
        "os"
)

func main() {
        fmt.Println("Connecting to rackspace:", os.Getenv("UserName"))
        // Create a connection
        c := swift.Connection{
                UserName: os.Getenv("UserName"),
                ApiKey:   os.Getenv("ApiKey"),
                AuthUrl:  "https://identity.api.rackspacecloud.com/v1.0",
        }
        // Authenticate
        err := c.Authenticate()
        if err != nil {
                panic(err)
        }

        // List all the containers
        containers, err := c.ContainerNames(nil)
        fmt.Println(containers)

        file, err := os.Open("/home/yml/golang/cloud_importer/testNew.tar.gz")
        if err != nil {
                fmt.Println("An error occured while opening the tar.gz file")
        }
        defer file.Close()
        dataStream, err := gzip.NewReader(file)
        if err != nil {
                fmt.Println("An error occured while creating the data stream")
        }

        // Bulk upload to rackspace
        bulkUploadResult, err := c.BulkUpload("myContainer", dataStream, swift.UploadTarGzip, nil)
        if err != nil {
                fmt.Println("An error occured while bulk uploading the tar.gz\n", err)
        }
        fmt.Println("Uploaded: ", bulkUploadResult)
}

For some reason rackspace return a BadRequest

yml@smithsonian$  go run src/cloud_importer.go 
Connecting to rackspace: MyUsername
[myContainer, ... all my other containers]
An error occured while bulk uploading the tar.gz
 Bad Request
Uploaded:  {0 map[] map[Content-Type:application/json X-Trans-Id:txf46e68078c12414f8d2ea-005245590cord1 Date:Fri, 27 Sep 2013 10:08:12 GMT]} 

i have old version of swift and can't get ObjectGet because of error:

parsing time "" as "Mon, 02 Jan 2006 15:04:05 GMT": cannot parse "" as "Mon"

as i understand default TimeFormat does not equal of my output, how can i deal with this?

Checking MD5SUMs when downloading a static large object doesn't work.

The library should be checking for X-Static-Large-Object: True and disabling the check - swift doesn't seem to be putting the X-Object-Manifest on like it used to for dynamic large objects.

Object does not provide for a way to set the X-Newest: true header. Refer to the API reference for details.

I'm aware that I can give extra headers to ObjectOpen, ObjectGet etc., but I'm dealing with large objects and therefore cannot afford downloading the entire file.

For restic, we'll switch to dep shortly, and it would be very helpful if you could tag a release. Thanks!

https://dave.cheney.net/2016/06/24/gophers-please-tag-your-releases

Hi Nick,

Can you please provide any hints how to use this library with Rackspace ORD (Chicago) region? Official docs are not very helpful: http://docs.rackspace.com/files/api/v1/cf-devguide/content/Multi_region-dl2200.html

Do you have plans to "export " the Authenticator interface?
I'm working on Hubic(OVH) API but i have to do a little hack to get this "working"

here-> dadc254

Thanks