go-goose / goose Goto Github PK

The OS docs say that depending on the auth type, a different domain name might be needed, either OS_USER_DOMAIN_NAME, OS_PROJECT_DOMAIN_NAME, or OS_DOMAIN_NAME work as env variables.

http://docs.openstack.org/developer/python-openstackclient/authentication.html

My openstack install provides me with OS_USER_DOMAIN_NAME in its generated rc, so the credential finding here fails.

I think we should at a minimum add all options to CredEnvDomainName in identity.go:
OS_DOMAIN_NAME
OS_USER_DOMAIN_NAME
OS_PROJECT_DOMAIN_NAME
OS_DEFAULT_DOMAIN

Really, we should be picking the right one based on the credential type.

Currently, go-goose has no support for GBP, and as a result, Juju and other projects using this library cannot provision GBP primitives required to provision connectivity in Cisco ACI based OpenStack implementations.

At a minimum, creation of PTs (policy targets) for launched instances would be required to allow instances to be launched and connected to the network on an ACI-based OpenStack.

This is a wishlist item rather than a defect.

~work/src/gopkg.in/goose.v1/nova$ go test -live -check.v -image 7e23b72e-bb45-4e3a-94fa-aa0ace208de8 -vendor canonistack -flavor m1.smaller ./...
PASS: json_test.go:31: JsonSuite.TestMarshallEntityLargeIntId 0.000s
PASS: json_test.go:37: JsonSuite.TestMarshallFlavorDetailLargeIntId 0.000s
PASS: json_test.go:51: JsonSuite.TestMarshallFloatingIPLargeIntId 0.000s
PASS: json_test.go:43: JsonSuite.TestMarshallServerDetailLargeIntId 0.000s
PASS: json_test.go:58: JsonSuite.TestUnmarshallFloatingIPNilStrings 0.000s
PASS: json_test.go:73: JsonSuite.TestUnmarshallRuleInfoNilStrings 0.000s

FAIL: live_test.go:47: LiveTests.SetUpSuite

live_test.go:54:
c.Assert(err, gc.IsNil)
... value _errors.gooseError = &errors.gooseError{error:(_errors.errorString)(0xc8200fe3b0), errcode:"Unspecified", cause:(*http.HttpError)(0xc8200de690)} ("failed to run a server with nova.RunServerOpts{Name:"nova_test_server", FlavorId:"1", ImageId:"f2ca48ce-30d5-4f1f-9075-12e64510368d", UserData:[]uint8(nil), SecurityGroupNames:[]nova.SecurityGroupName(nil), Networks:[]nova.ServerNetworks(nil), AvailabilityZone:"", Metadata:map[string]string(nil), ConfigDrive:false}\ncaused by: request (http://10.172.45.130:8774/v2/5b6daa83031146c784971b5f01005d6b/servers) returned unexpected status: 400; error info: {"badRequest": {"message": "Invalid input for field/attribute networks. Value: None. None is not of type 'array'", "code": 400}}")

MISS: live_test.go:267: LiveTests.TestCreateAndDeleteSecurityGroup
MISS: live_test.go:326: LiveTests.TestCreateAndDeleteSecurityGroupRules
MISS: live_test.go:318: LiveTests.TestDuplicateSecurityGroupError
....

The novaservice might need related updates as well.

heather@ubuntu-maverick:~/work/src/gopkg.in/goose$ ./test.py --verbose
Could not find "src/gopkg.in/goose.v1" in cwd: /home/heather/work/src/gopkg.in/goose
Unable to automatically set GOPATH
Running: go fmt ./...
Running: go build ./...
testservices/cmd/main.go:9:2: cannot find package "launchpad.net/gnuflag" in any of:
/usr/lib/go-1.6/src/launchpad.net/gnuflag (from $GOROOT)
/home/heather/work/src/launchpad.net/gnuflag (from $GOPATH)
FAIL: failed running: go build ./...

Per https://launchpad.net/gnuflag: "NOTE: The official location of this package has now moved to https://github.com/juju/gnuflag ."

heather@ubuntu-maverick:~/work/src/github.com/juju/juju$ find $GOPATH -name gnuflag
/home/heather/work/src/github.com/lxc/lxd/shared/gnuflag
/home/heather/work/src/github.com/juju/gnuflag

I'm running Xenial and juju 2.0-beta18-xenial-amd64

It's possible to have a novarc file with a keystone v3 set of credential attributes like this:

export OS_USERNAME=fred
export OS_PASSWORD=secret
export OS_USER_DOMAIN_NAME=user
export OS_PROJECT_NAME=vinu
export OS_PROJECT_ID=cab03d5e990e438dae80aedb58ffde77
export OS_PROJECT_DOMAIN_NAME=user

These will be parsed into a credential where the "TenantName" attribute is set to "cab03d5e990e438dae80aedb58ffde77". But that value is the tenant id, not name. And then when we go to authenticate, we do this:

auth.Auth.Scope = &v3AuthScope{
Project: &v3AuthProject{
Domain: &v3AuthDomain{
Name: projectDomain,
},
Name: creds.TenantName,
},
}

The auth v3AuthProject struct does have an ID field and that's what should be set in this case.

The identity.Credentials struct could gain a TenantId field and we use that if set. But maybe we can simply not use the id at all because:

1. We only use Name in v3AuthDomain (we ignore ID)
2. I tested auth with v3AuthProject.Name = "" and it worked

So we could start by simply modifying the CredentialsFromEnv() function and associated validation to not set TenantName from OS_TENANT_ID etc.

When allocating a public IP using the AllocatePublicIP() API, we use the neutron network API to query networks, and subsequently go through the result to look for networks matching availability zones.

We use the API "GET /v2.0/networks" which should return all networks.

However, some Openstacks require a query param to explicitly ask for external networks:

GET /v2.0/networks?router:external=True

We need to tweak the neutron.GetNetworkV2() method to take query params and then modify the getExternalNeutronNetworksByAZ() method to pass in router:external=True

In goose/testservices/hook/service_gccgo.go there is some logic to extract the method name using the call stack from the runtime package. This works well for the default gc compiler, bug causes test failures with gccgo compiler, e.g. on ppc64el - http://paste.ubuntu.com/10356017/ This happens regularly in Juju CI tests (job: run-unit-tests-trusty-ppc64el-lxc). After looking through the code I found that I can reproduce the same error locally on a trusty amd64 machine, using go test -check.v -check.f TestBootstrapFailsWhenPublicIPError -compiler gccgo: http://paste.ubuntu.com/10356073/

I have a patch in mind which should resolve this, but I'm still testing it as I'm hitting random failures with gccgo as described in this bug https://bugs.launchpad.net/ubuntu/+source/gccgo-go/+bug/1393825

In authernticatingClient doAuthenticate(), we return a generic error string when there's an authentication error:

if authDetails, err = c.authMode.Auth(c.creds); err != nil {
return gooseerrors.Newf(err, "authentication failed")
}

And also further down in the method when there's other types of error, eg

if err := c.createServiceURLs(); err != nil {
return gooseerrors.Newf(err, "cannot create service URLs")
}

This makes it hard for callers (eg juju) to distinguish between auth errors and other errors. goose does have an UnauthorisedError code so that should be used where appropriate.

Running the swift live test produces from TestPublicList(). The contents can't be listed, the container has no ACL settings to allow this. Ran with both admin and user credentials for OpenStack (Mitaka).

I noticed this behavior on the command line for OpenStack previously also. If you set the ACLs for a container after creation, they are ignored. But if you sent the ACLs while creating the container it works.

heather@mitaka:/work/src/gopkg.in/goose.v1/swift$ env | grep OS_
OS_REGION_NAME=RegionOne
OS_PASSWORD=openstack
OS_AUTH_URL=http://10.172.45.216:5000/v2.0
OS_USERNAME=admin
OS_TENANT_NAME=admin
heather@mitaka:/work/src/gopkg.in/goose.v1/swift$ go test -check.v -live ./... > results-live.out 2>&1

FAIL: live_test.go:195: LiveTestsPublicContainer.TestPublicList

live_test.go:206:
c.Check(err, gc.IsNil)
... value _errors.gooseError = &errors.gooseError{error:(_errors.errorString)(0xc8201a9610), errcode:"Unspecified", cause:(*errors.gooseError)(0xc8201e46c0)} ("failed to list contents of container: test_containerd06997de54da7083\ncaused by: Unauthorised URL http://10.172.45.99:80/swift/v1/test_containerd06997de54da7083?delimiter=&format=json&marker=&prefix=\ncaused by: request (http://10.172.45.99:80/swift/v1/test_containerd06997de54da7083?delimiter=&format=json&marker=&prefix=) returned unexpected status: 401; error info: {"Code":"AccessDenied","BucketName":"test_containerd06997de54da7083","RequestId":"tx0000000000000000000c9-0057f3a77b-1081-default","HostId":"1081-default-default"}")

live_test.go:207:
c.Check(len(items), gc.Equals, len(files))
... obtained int = 0
... expected int = 2

heather@mitaka:$ source novarc
heather@mitaka:$ swift stat test_containerd06997de54da7083
Account: v1
Container: test_containerd06997de54da7083
Objects: 1
Bytes: 15
Read ACL:
Write ACL:
Sync To:
Sync Key:
Accept-Ranges: bytes
X-Storage-Policy: default-placement
X-Container-Bytes-Used-Actual: 4096
X-Timestamp: 0.00000
X-Trans-Id: tx0000000000000000000dd-0057f3b015-1081-default
Content-Type: text/plain; charset=utf-8
heather@mitaka:$ swift list test_containerd06997de54da7083
test_obj2
heather@mitaka:$ source heatherrc
heather@mitaka:~$ swift list test_containerd06997de54da7083
Container GET failed: http://10.172.45.99:80/swift/v1/test_containerd06997de54da7083?format=json 401 Unauthorized [first 60 chars of response] {"Code":"AccessDenied","RequestId":"tx0000000000000000000e1-

gopkg.in/goose.v2/client$ go test -check.v ./...
SKIP: :3: localLiveSuite.TestAuthDiscover (this test will not work with legacy auth)
PASS: :2: localLiveSuite.TestAuthenticate 0.000s
PASS: :1: localLiveSuite.TestAuthenticateFail 0.000s
SKIP: local_test.go:290: localLiveSuite.TestAuthenticationForbidsMultipleCallers (legacy authentication)
PASS: local_test.go:268: localLiveSuite.TestAuthenticationSuccess 0.001s
PASS: local_test.go:239: localLiveSuite.TestAuthenticationTimeout 0.001s
SKIP: local_test.go:153: localLiveSuite.TestInexactRegionMatch (legacy authentication doesn't use regions)
SKIP: local_test.go:137: localLiveSuite.TestInvalidRegion (legacy authentication doesn't use regions)

FAIL: apiversion_test.go:124: localLiveSuite.TestMakeServiceURL

#1 of 13. compute []
#2 of 13. compute v2.1 [foo bar/]
#3 of 13. compute v2.0 []
#4 of 13. compute v2.0 [foo bar/]
#5 of 13. compute v2 [foo bar/]
#6 of 13. object-store [foo bar]
#7 of 13. object-store q2.0 [foo bar/]
apiversion_test.go:140:
c.Assert(err, gc.ErrorMatches, t.err)
... error string = "strconv.Atoi: parsing "q2": invalid syntax"
... regex string = "strconv.ParseInt: parsing "q2": invalid syntax"

PASS: apiversion_test.go:145: localLiveSuite.TestMakeServiceURLAPIVersionDiscoveryDisabled0.000s
2017-05-09 18:53:26 WARNING goose.client :38 API version discovery failed: Resource at http://localhost:3005/ not found
caused by: request (http://localhost:3005/) returned unexpected status: 404; error info: {"message":"Api does not exist","request_id":"83A781AE-9A0C-43C7-B405-310A5A94566E"}
2017-05-09 18:53:26 WARNING goose.client :38 falling back to catalogue service URL
PASS: apiversion_test.go:156: localLiveSuite.TestMakeServiceURLNoAPIVersionEndpoint 0.000s

FAIL: apiversion_test.go:168: localLiveSuite.TestMakeServiceURLValues

#1 of 13. compute []
#2 of 13. compute v2.1 [foo bar/]
#3 of 13. compute v2.0 []
#4 of 13. compute v2.0 [foo bar/]
#5 of 13. compute v2 [foo bar/]
#6 of 13. object-store [foo bar]
#7 of 13. object-store q2.0 [foo bar/]
apiversion_test.go:184:
c.Assert(err, gc.ErrorMatches, t.err)
... error string = "strconv.Atoi: parsing "q2": invalid syntax"
... regex string = "strconv.ParseInt: parsing "q2": invalid syntax"

SKIP: local_test.go:373: localLiveSuite.TestNonAccessibleServiceType (legacy authentication)
PASS: :3: localLiveSuite.TestAuthDiscover 0.000s
PASS: :2: localLiveSuite.TestAuthenticate 0.001s
PASS: :1: localLiveSuite.TestAuthenticateFail 0.001s
PASS: local_test.go:290: localLiveSuite.TestAuthenticationForbidsMultipleCallers 0.001s
PASS: local_test.go:268: localLiveSuite.TestAuthenticationSuccess 0.000s
PASS: local_test.go:239: localLiveSuite.TestAuthenticationTimeout 0.001s
PASS: local_test.go:153: localLiveSuite.TestInexactRegionMatch 0.001s
PASS: local_test.go:137: localLiveSuite.TestInvalidRegion 0.001s

FAIL: apiversion_test.go:124: localLiveSuite.TestMakeServiceURL

#1 of 13. compute []
#2 of 13. compute v2.1 [foo bar/]
#3 of 13. compute v2.0 []
#4 of 13. compute v2.0 [foo bar/]
#5 of 13. compute v2 [foo bar/]
#6 of 13. object-store [foo bar]
#7 of 13. object-store q2.0 [foo bar/]
apiversion_test.go:140:
c.Assert(err, gc.ErrorMatches, t.err)
... error string = "strconv.Atoi: parsing "q2": invalid syntax"
... regex string = "strconv.ParseInt: parsing "q2": invalid syntax"

PASS: apiversion_test.go:145: localLiveSuite.TestMakeServiceURLAPIVersionDiscoveryDisabled0.000s
2017-05-09 18:53:26 WARNING goose.client :38 API version discovery failed: Resource at http://localhost:3005/ not found
caused by: request (http://localhost:3005/) returned unexpected status: 404; error info: {"message":"Api does not exist","request_id":"83A781AE-9A0C-43C7-B405-310A5A94566E"}
2017-05-09 18:53:26 WARNING goose.client :38 falling back to catalogue service URL
PASS: apiversion_test.go:156: localLiveSuite.TestMakeServiceURLNoAPIVersionEndpoint 0.000s

FAIL: apiversion_test.go:168: localLiveSuite.TestMakeServiceURLValues

#1 of 13. compute []
#2 of 13. compute v2.1 [foo bar/]
#3 of 13. compute v2.0 []
#4 of 13. compute v2.0 [foo bar/]
#5 of 13. compute v2 [foo bar/]
#6 of 13. object-store [foo bar]
#7 of 13. object-store q2.0 [foo bar/]
apiversion_test.go:184:
c.Assert(err, gc.ErrorMatches, t.err)
... error string = "strconv.Atoi: parsing "q2": invalid syntax"
... regex string = "strconv.ParseInt: parsing "q2": invalid syntax"

PASS: local_test.go:373: localLiveSuite.TestNonAccessibleServiceType 0.001s
PASS: :3: localLiveSuite.TestAuthDiscover 0.000s
PASS: :2: localLiveSuite.TestAuthenticate 0.001s
PASS: :1: localLiveSuite.TestAuthenticateFail 0.000s
PASS: local_test.go:290: localLiveSuite.TestAuthenticationForbidsMultipleCallers 0.001s
PASS: local_test.go:268: localLiveSuite.TestAuthenticationSuccess 0.001s
PASS: local_test.go:239: localLiveSuite.TestAuthenticationTimeout 0.001s
PASS: local_test.go:153: localLiveSuite.TestInexactRegionMatch 0.001s
PASS: local_test.go:137: localLiveSuite.TestInvalidRegion 0.000s

FAIL: apiversion_test.go:124: localLiveSuite.TestMakeServiceURL

#1 of 13. compute []
#2 of 13. compute v2.1 [foo bar/]
#3 of 13. compute v2.0 []
#4 of 13. compute v2.0 [foo bar/]
#5 of 13. compute v2 [foo bar/]
#6 of 13. object-store [foo bar]
#7 of 13. object-store q2.0 [foo bar/]
apiversion_test.go:140:
c.Assert(err, gc.ErrorMatches, t.err)
... error string = "strconv.Atoi: parsing "q2": invalid syntax"
... regex string = "strconv.ParseInt: parsing "q2": invalid syntax"

PASS: apiversion_test.go:145: localLiveSuite.TestMakeServiceURLAPIVersionDiscoveryDisabled0.000s
2017-05-09 18:53:26 WARNING goose.client :38 API version discovery failed: Resource at http://localhost:3005/ not found
caused by: request (http://localhost:3005/) returned unexpected status: 404; error info: {"message":"Api does not exist","request_id":"83A781AE-9A0C-43C7-B405-310A5A94566E"}
2017-05-09 18:53:26 WARNING goose.client :38 falling back to catalogue service URL
PASS: apiversion_test.go:156: localLiveSuite.TestMakeServiceURLNoAPIVersionEndpoint 0.000s

FAIL: apiversion_test.go:168: localLiveSuite.TestMakeServiceURLValues

#1 of 13. compute []
#2 of 13. compute v2.1 [foo bar/]
#3 of 13. compute v2.0 []
#4 of 13. compute v2.0 [foo bar/]
#5 of 13. compute v2 [foo bar/]
#6 of 13. object-store [foo bar]
#7 of 13. object-store q2.0 [foo bar/]
apiversion_test.go:184:
c.Assert(err, gc.ErrorMatches, t.err)
... error string = "strconv.Atoi: parsing "q2": invalid syntax"
... regex string = "strconv.ParseInt: parsing "q2": invalid syntax"

PASS: local_test.go:373: localLiveSuite.TestNonAccessibleServiceType 0.000s
PASS: local_test.go:490: localHTTPSSuite.TestAuthDiscover 0.002s
2017/05/09 14:53:26 http: TLS handshake error from 127.0.0.1:60036: remote error: tls: bad certificate
PASS: local_test.go:436: localHTTPSSuite.TestDefaultClientRefusesSelfSigned 0.010s
2017/05/09 14:53:26 http: TLS handshake error from 127.0.0.1:35364: remote error: tls: bad certificate
PASS: local_test.go:474: localHTTPSSuite.TestDefaultPublicClientRefusesSelfSigned 0.005s
PASS: local_test.go:442: localHTTPSSuite.TestNonValidatingClientAcceptsSelfSigned 0.004s
PASS: local_test.go:482: localHTTPSSuite.TestNonValidatingPublicClientAcceptsSelfSigned 0.003s
OOPS: 33 passed, 5 skipped, 6 FAILED
--- FAIL: Test (0.06s)
FAIL
exit status 1
FAIL gopkg.in/goose.v2/client 0.060s

The symptom is that pushes of resources (or even charms) to the staging charmstore fails.

The problem is though to be here https://github.com/go-goose/goose/blob/v2/client/client.go#L219.
In the case where it has an expired authentication token then performs the first PUT get's an authorization failure re-authorizes with keystone and just tries again, without ever trying to reset the reader containing the request body, as it's already read it once it's at EOF and thus fails on the second attempt.

The openrc file for a non-admin user downloaded from OpenStack Horizon does not include project domain name but includes project domain id [1].
The request is to add Project domain id to credentials structure.
Juju uses goose v2 identity and requires project domain id to autload juju credentials, see [2]

[1] https://opendev.org/openstack/horizon/src/branch/master/openstack_dashboard/dashboards/project/api_access/templates/api_access/openrc.sh.template#L22
[2] https://bugs.launchpad.net/juju/+bug/1772649

Currently, the cinder/autogenerated_client.go file is auto-generated, but in a disconnected way from the build process.

We are currently in the process of landing Cinder support. This patch will land without a test-double.