Comments (12)
When you say mantash isn't working inside the container are you getting the error described here for python-manta? https://github.com/joyent/python-manta#x509-certificate-routinesx509_load_cert_crl_file-error.
I'll be honest and say I've never tested this with a self-signed cert so this might be an intentional limitation of the python-manta
library. But I don't think so -- I dug into the source a bit and it looks like we're passing the disable_ssl_certificate_validation
parameter to httplib2 from the MANTA_TLS_INSECURE
variable correctly so I'm not sure where it's getting dropped. Can you provide the exact command line, environment vars, and messages you're getting back from mantash? That way we can see if it's something we need to fix here or whether it's something we can help @trentm (the main python-manta
maintainer) with.
Somewhat unrelated:
2016/09/21 18:24:20 httplib2.SSLHandshakeError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:581)
Is this the only error? I feel like we should be louder about this.
from mysql.
Nope. Its not the "x509 certificate routines:X509_load_cert_crl_file error" error. It is:
root@86c2ccad4c59:/# env |grep -i manta
MANTA_USER=myuser
MANTA_KEY_ID=c7:6d:b4:e9:f9:33:44:7d:cb:6e:58:41:b6:b6:7f:c7
MANTA_ROLE=
MANTA_BUCKET=/myuser/stor/triton-mysql
MANTA_URL=https://10.64.243.97
MANTA_SUBUSER=
MANTA_PRIVATE_KEY=-----BEGIN RSA PRIVATE KEY-----#<keydata>#-----END RSA PRIVATE KEY-----
MANTA_TLS_INSECURE=1
root@86c2ccad4c59:/# mantash ls
mantash: ERROR: could not find key info for signing: no ssh-agent key with fingerprint "c7:6d:b4:e9:f9:33:44:7d:cb:6e:58:41:b6:b6:7f:c7"; no '~/.ssh/*.pub' key found with fingerprint 'c7:6d:b4:e9:f9:33:44:7d:cb:6e:58:41:b6:b6:7f:c7'
As soon as I paste $MANTA_PRIVATE_KEY into .ssh/key and paste my ssh-pub key into .ssh/key.pub "mantash ls" works without a problem (MANTA_TLS_INSECURE=1 is set). As soon as I unset MANTA_TLS_INSECURE, mantash stops working.
Obviously mantash is reading my MANTA_KEY_ID from the environment and it's looking for key and key.pub in the .ssh-directory (which are obviously not there).
Here's the complete error message from "docker logs 86c2ccad4c59":
2016/09/21 18:34:30 INFO manage snapshot completed, uploading to object store
2016/09/21 18:34:31 Traceback (most recent call last):
2016/09/21 18:34:31 File "/usr/local/bin/manage.py", line 486, in <module>
2016/09/21 18:34:31 main()
2016/09/21 18:34:31 File "/usr/local/bin/manage.py", line 483, in main
2016/09/21 18:34:31 cmd(node)
2016/09/21 18:34:31 File "/usr/local/bin/manager/utils.py", line 64, in wrapper
2016/09/21 18:34:31 out = apply(fn, args, kwargs)
2016/09/21 18:34:31 File "/usr/local/bin/manage.py", line 131, in health
2016/09/21 18:34:31 assert_initialized_for_state(node)
2016/09/21 18:34:31 File "/usr/local/bin/manager/utils.py", line 64, in wrapper
2016/09/21 18:34:31 out = apply(fn, args, kwargs)
2016/09/21 18:34:31 File "/usr/local/bin/manage.py", line 381, in assert_initialized_for_state
2016/09/21 18:34:31 if not run_as_primary(node):
2016/09/21 18:34:31 File "/usr/local/bin/manager/utils.py", line 64, in wrapper
2016/09/21 18:34:31 out = apply(fn, args, kwargs)
2016/09/21 18:34:31 File "/usr/local/bin/manage.py", line 442, in run_as_primary
2016/09/21 18:34:31 write_snapshot(node)
2016/09/21 18:34:31 File "/usr/local/bin/manager/utils.py", line 64, in wrapper
2016/09/21 18:34:31 out = apply(fn, args, kwargs)
2016/09/21 18:34:31 File "/usr/local/bin/manage.py", line 289, in write_snapshot
2016/09/21 18:34:31 create_snapshot(node)
2016/09/21 18:34:31 File "/usr/local/bin/manager/utils.py", line 64, in wrapper
2016/09/21 18:34:31 out = apply(fn, args, kwargs)
2016/09/21 18:34:31 File "/usr/local/bin/manage.py", line 320, in create_snapshot
2016/09/21 18:34:31 node.manta.put_backup(backup_id, '/tmp/backup.tar')
2016/09/21 18:34:31 File "/usr/local/bin/manager/libmanta.py", line 54, in put_backup
2016/09/21 18:34:31 self.client.put_object(mpath, file=f)
2016/09/21 18:34:31 File "/usr/local/lib/python2.7/dist-packages/manta/client.py", line 351, in put_object
2016/09/21 18:34:31 headers=headers)
2016/09/21 18:34:31 File "/usr/local/lib/python2.7/dist-packages/manta/client.py", line 210, in _request
2016/09/21 18:34:31 return http.request(url, method, ubody, headers)
2016/09/21 18:34:31 File "/usr/local/lib/python2.7/dist-packages/httplib2/__init__.py", line 1570, in request
2016/09/21 18:34:31 (response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey)
2016/09/21 18:34:31 File "/usr/local/lib/python2.7/dist-packages/manta/client.py", line 100, in _request
2016/09/21 18:34:31 res, content = httplib2.Http._request(self, conn, host, absolute_uri, request_uri, method, body, headers, redirections, cachekey)
2016/09/21 18:34:31 File "/usr/local/lib/python2.7/dist-packages/httplib2/__init__.py", line 1317, in _request
2016/09/21 18:34:31 (response, content) = self._conn_request(conn, request_uri, method, body, headers)
2016/09/21 18:34:31 File "/usr/local/lib/python2.7/dist-packages/httplib2/__init__.py", line 1252, in _conn_request
2016/09/21 18:34:31 conn.connect()
2016/09/21 18:34:31 File "/usr/local/lib/python2.7/dist-packages/httplib2/__init__.py", line 1044, in connect
2016/09/21 18:34:31 raise SSLHandshakeError(e)
2016/09/21 18:34:31 httplib2.SSLHandshakeError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:581)
2016/09/21 18:34:31 2016/09/21 18:34:31 [ERR] http: Request PUT /v1/agent/check/pass/mysql-86c2ccad4c59?note=ok, error: CheckID does not have associated TTL from=127.0.0.1:35245
2016/09/21 18:34:31 Unexpected response code: 500 (CheckID does not have associated TTL)
Service not registered, registering...
from mysql.
Ah, I think I misunderstood the original problem. The Manta API has mutual authentication. There are two components here:
- Transport over the wire: this is always secured by TLS but the certificate for that transport that identifies the server can be either verified or not (as you're doing with
MANTA_TLS_INSECURE=1
). The TLS ensures that you're writing data to Manta and not somewhere else, and that the communication is untampered-with and private. - Authentication to Manta: whether or not the connection is over verified TLS, the Manta API still requires an SSH key to verify the identity of the client writing the data.
So you're receiving the SSL handshake error when MANTA_TLS_INSECURE
is unset because you don't have a CA-signed certificate, as you'd expect. But when you try to write to Manta without the SSH key you can't, because Manta doesn't know who you are.
from mysql.
But what do we learn from that? When manage.py tries to upload the mysql-backup to Manta it throws "SSL: CERTIFICATE_VERIFY_FAILED" (although MANTA_TLS_INSECURE=1 is set).In the environment we have MANTA_KEY_ID, MANTA_PRIVATE_KEY and MANTA_USER. When using Manta with the commands from the node.js world (like "mls", etc.) I don't even have to provide MANTA_PRIVATE_KEY. What am I missing? How am I supposed to transfer the required ssh-keys into the container (or am I)?
from mysql.
The keys are transferred from the environment directly into the python-manta
client library by the manage.py
application (when we instantiate the client here). Mantash doesn't come into this at all unless you're using it for debugging, in which case yes you need to provide it with the key directly.
But now that I've pointed that little tidbit out, I realize we're not passing thru the command line interface that might be parsing the MANTA_TLS_INSECURE
, and sure enough that's exactly the problem. The constructor for our Manta client wrapper should look like:
class Manta(object):
"""
The Manta class wraps access to the Manta object store, where we'll put
our MySQL backups.
"""
def __init__(self, envs=os.environ):
self.account = env('MANTA_USER', None, envs)
self.user = env('MANTA_SUBUSER', None, envs)
self.role = env('MANTA_ROLE', None, envs)
self.key_id = env('MANTA_KEY_ID', None, envs)
self.url = env('MANTA_URL', 'https://us-east.manta.joyent.com', envs)
self.bucket = env('MANTA_BUCKET', '/{}/stor'.format(self.account), envs)
is_tls = env('MANTA_TLS_INSECURE', False)
# we don't want to use `env` here because we have a different
# de-munging to do
self.private_key = envs.get('MANTA_PRIVATE_KEY', '').replace('#', '\n')
self.signer = pymanta.PrivateKeySigner(self.key_id, self.private_key)
self.client = pymanta.MantaClient(self.url,
self.account,
subuser=self.user,
role=self.role,
disable_ssl_certificate_validation=is_tls,
signer=self.signer)
This is a pretty quick change. If you want to put up a PR for it I'd be happy to merge it, or I can try and hit it in the next day or so.
from mysql.
I'm not sure, whether my github and/or python skills are ready for a PR. I'll have a look over the weekend.
from mysql.
I took a crack at that in #57
from mysql.
I've checked out the patch and I was able to build the container but I was not able to start it (and I have no logs to see, what happened). Now I see, you merged the patch already (libmanta.py has the changes), but the container still has the old version, right?
from mysql.
Right, that hasn't been released yet. I'm trying to figure out why our 3rd-party test runner isn't working quite right #64 and then I'll cut a new release.
from mysql.
Released in https://github.com/autopilotpattern/mysql/releases/tag/5.6r3.1.0
from mysql.
@neuroserve autopilotpattern/mysql:latest
or autopilotpattern/mysql:5.6r3.1.0
images will have this fix.
from mysql.
And works. Backups are now uploaded to Manta.
from mysql.
Related Issues (20)
- Document SNAPSHOT_BACKEND
- Health check should verify replication
- Simplify CNS search domain for consumers HOT 4
- MySQL not starting HOT 1
- Not starting cluster, can't find mysql-primary HOT 5
- Re-configure after a consul system failure. HOT 1
- This container has its own installation of consul? HOT 2
- Replication failure HOT 41
- Create snapshot when mysql-primary is restarted/recreated HOT 2
- consul rpc error: invalid session HOT 2
- Consul 2 private interfaces HOT 2
- Failover not reporting to consul HOT 4
- Switch to Percona 5.7 HOT 1
- Support for semisynchronous replication HOT 3
- Consider adding Percona Monitoring and Management agent
- Failover lost master HOT 1
- Snapshots not backing up HOT 2
- Minio restarting causes unhealthy mysql-primary
- InnoDB restore in the future
- Not able to run cluster. HOT 6
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from mysql.