Code Monkey home page Code Monkey logo

Comments (12)

tgross avatar tgross commented on August 20, 2024

When you say mantash isn't working inside the container are you getting the error described here for python-manta? https://github.com/joyent/python-manta#x509-certificate-routinesx509_load_cert_crl_file-error.

I'll be honest and say I've never tested this with a self-signed cert so this might be an intentional limitation of the python-manta library. But I don't think so -- I dug into the source a bit and it looks like we're passing the disable_ssl_certificate_validation parameter to httplib2 from the MANTA_TLS_INSECURE variable correctly so I'm not sure where it's getting dropped. Can you provide the exact command line, environment vars, and messages you're getting back from mantash? That way we can see if it's something we need to fix here or whether it's something we can help @trentm (the main python-manta maintainer) with.

Somewhat unrelated:

2016/09/21 18:24:20 httplib2.SSLHandshakeError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:581)

Is this the only error? I feel like we should be louder about this.

from mysql.

neuroserve avatar neuroserve commented on August 20, 2024

Nope. Its not the "x509 certificate routines:X509_load_cert_crl_file error" error. It is:

root@86c2ccad4c59:/# env |grep -i manta
MANTA_USER=myuser
MANTA_KEY_ID=c7:6d:b4:e9:f9:33:44:7d:cb:6e:58:41:b6:b6:7f:c7
MANTA_ROLE=
MANTA_BUCKET=/myuser/stor/triton-mysql
MANTA_URL=https://10.64.243.97
MANTA_SUBUSER=
MANTA_PRIVATE_KEY=-----BEGIN RSA PRIVATE KEY-----#<keydata>#-----END RSA PRIVATE KEY-----
MANTA_TLS_INSECURE=1
root@86c2ccad4c59:/# mantash ls
mantash: ERROR: could not find key info for signing: no ssh-agent key with fingerprint "c7:6d:b4:e9:f9:33:44:7d:cb:6e:58:41:b6:b6:7f:c7"; no '~/.ssh/*.pub' key found with fingerprint 'c7:6d:b4:e9:f9:33:44:7d:cb:6e:58:41:b6:b6:7f:c7'

As soon as I paste $MANTA_PRIVATE_KEY into .ssh/key and paste my ssh-pub key into .ssh/key.pub "mantash ls" works without a problem (MANTA_TLS_INSECURE=1 is set). As soon as I unset MANTA_TLS_INSECURE, mantash stops working.

Obviously mantash is reading my MANTA_KEY_ID from the environment and it's looking for key and key.pub in the .ssh-directory (which are obviously not there).

Here's the complete error message from "docker logs 86c2ccad4c59":

2016/09/21 18:34:30 INFO manage snapshot completed, uploading to object store
2016/09/21 18:34:31 Traceback (most recent call last):
2016/09/21 18:34:31   File "/usr/local/bin/manage.py", line 486, in <module>
2016/09/21 18:34:31     main()
2016/09/21 18:34:31   File "/usr/local/bin/manage.py", line 483, in main
2016/09/21 18:34:31     cmd(node)
2016/09/21 18:34:31   File "/usr/local/bin/manager/utils.py", line 64, in wrapper
2016/09/21 18:34:31     out = apply(fn, args, kwargs)
2016/09/21 18:34:31   File "/usr/local/bin/manage.py", line 131, in health
2016/09/21 18:34:31     assert_initialized_for_state(node)
2016/09/21 18:34:31   File "/usr/local/bin/manager/utils.py", line 64, in wrapper
2016/09/21 18:34:31     out = apply(fn, args, kwargs)
2016/09/21 18:34:31   File "/usr/local/bin/manage.py", line 381, in assert_initialized_for_state
2016/09/21 18:34:31     if not run_as_primary(node):
2016/09/21 18:34:31   File "/usr/local/bin/manager/utils.py", line 64, in wrapper
2016/09/21 18:34:31     out = apply(fn, args, kwargs)
2016/09/21 18:34:31   File "/usr/local/bin/manage.py", line 442, in run_as_primary
2016/09/21 18:34:31     write_snapshot(node)
2016/09/21 18:34:31   File "/usr/local/bin/manager/utils.py", line 64, in wrapper
2016/09/21 18:34:31     out = apply(fn, args, kwargs)
2016/09/21 18:34:31   File "/usr/local/bin/manage.py", line 289, in write_snapshot
2016/09/21 18:34:31     create_snapshot(node)
2016/09/21 18:34:31   File "/usr/local/bin/manager/utils.py", line 64, in wrapper
2016/09/21 18:34:31     out = apply(fn, args, kwargs)
2016/09/21 18:34:31   File "/usr/local/bin/manage.py", line 320, in create_snapshot
2016/09/21 18:34:31     node.manta.put_backup(backup_id, '/tmp/backup.tar')
2016/09/21 18:34:31   File "/usr/local/bin/manager/libmanta.py", line 54, in put_backup
2016/09/21 18:34:31     self.client.put_object(mpath, file=f)
2016/09/21 18:34:31   File "/usr/local/lib/python2.7/dist-packages/manta/client.py", line 351, in put_object
2016/09/21 18:34:31     headers=headers)
2016/09/21 18:34:31   File "/usr/local/lib/python2.7/dist-packages/manta/client.py", line 210, in _request
2016/09/21 18:34:31     return http.request(url, method, ubody, headers)
2016/09/21 18:34:31   File "/usr/local/lib/python2.7/dist-packages/httplib2/__init__.py", line 1570, in request
2016/09/21 18:34:31     (response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey)
2016/09/21 18:34:31   File "/usr/local/lib/python2.7/dist-packages/manta/client.py", line 100, in _request
2016/09/21 18:34:31     res, content = httplib2.Http._request(self, conn, host, absolute_uri, request_uri, method, body, headers, redirections, cachekey)
2016/09/21 18:34:31   File "/usr/local/lib/python2.7/dist-packages/httplib2/__init__.py", line 1317, in _request
2016/09/21 18:34:31     (response, content) = self._conn_request(conn, request_uri, method, body, headers)
2016/09/21 18:34:31   File "/usr/local/lib/python2.7/dist-packages/httplib2/__init__.py", line 1252, in _conn_request
2016/09/21 18:34:31     conn.connect()
2016/09/21 18:34:31   File "/usr/local/lib/python2.7/dist-packages/httplib2/__init__.py", line 1044, in connect
2016/09/21 18:34:31     raise SSLHandshakeError(e)
2016/09/21 18:34:31 httplib2.SSLHandshakeError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:581)
2016/09/21 18:34:31     2016/09/21 18:34:31 [ERR] http: Request PUT /v1/agent/check/pass/mysql-86c2ccad4c59?note=ok, error: CheckID does not have associated TTL from=127.0.0.1:35245
2016/09/21 18:34:31 Unexpected response code: 500 (CheckID does not have associated TTL)
Service not registered, registering...

from mysql.

tgross avatar tgross commented on August 20, 2024

Ah, I think I misunderstood the original problem. The Manta API has mutual authentication. There are two components here:

  • Transport over the wire: this is always secured by TLS but the certificate for that transport that identifies the server can be either verified or not (as you're doing with MANTA_TLS_INSECURE=1). The TLS ensures that you're writing data to Manta and not somewhere else, and that the communication is untampered-with and private.
  • Authentication to Manta: whether or not the connection is over verified TLS, the Manta API still requires an SSH key to verify the identity of the client writing the data.

So you're receiving the SSL handshake error when MANTA_TLS_INSECURE is unset because you don't have a CA-signed certificate, as you'd expect. But when you try to write to Manta without the SSH key you can't, because Manta doesn't know who you are.

from mysql.

neuroserve avatar neuroserve commented on August 20, 2024

But what do we learn from that? When manage.py tries to upload the mysql-backup to Manta it throws "SSL: CERTIFICATE_VERIFY_FAILED" (although MANTA_TLS_INSECURE=1 is set).In the environment we have MANTA_KEY_ID, MANTA_PRIVATE_KEY and MANTA_USER. When using Manta with the commands from the node.js world (like "mls", etc.) I don't even have to provide MANTA_PRIVATE_KEY. What am I missing? How am I supposed to transfer the required ssh-keys into the container (or am I)?

from mysql.

tgross avatar tgross commented on August 20, 2024

The keys are transferred from the environment directly into the python-manta client library by the manage.py application (when we instantiate the client here). Mantash doesn't come into this at all unless you're using it for debugging, in which case yes you need to provide it with the key directly.

But now that I've pointed that little tidbit out, I realize we're not passing thru the command line interface that might be parsing the MANTA_TLS_INSECURE, and sure enough that's exactly the problem. The constructor for our Manta client wrapper should look like:

class Manta(object):
    """
    The Manta class wraps access to the Manta object store, where we'll put
    our MySQL backups.
    """
    def __init__(self, envs=os.environ):
        self.account = env('MANTA_USER', None, envs)
        self.user = env('MANTA_SUBUSER', None, envs)
        self.role = env('MANTA_ROLE', None, envs)
        self.key_id = env('MANTA_KEY_ID', None, envs)
        self.url = env('MANTA_URL', 'https://us-east.manta.joyent.com', envs)
        self.bucket = env('MANTA_BUCKET', '/{}/stor'.format(self.account), envs)
        is_tls = env('MANTA_TLS_INSECURE', False)

        # we don't want to use `env` here because we have a different
        # de-munging to do
        self.private_key = envs.get('MANTA_PRIVATE_KEY', '').replace('#', '\n')
        self.signer = pymanta.PrivateKeySigner(self.key_id, self.private_key)
        self.client = pymanta.MantaClient(self.url,
                                          self.account,
                                          subuser=self.user,
                                          role=self.role,
                                          disable_ssl_certificate_validation=is_tls,
                                          signer=self.signer)

This is a pretty quick change. If you want to put up a PR for it I'd be happy to merge it, or I can try and hit it in the next day or so.

from mysql.

neuroserve avatar neuroserve commented on August 20, 2024

I'm not sure, whether my github and/or python skills are ready for a PR. I'll have a look over the weekend.

from mysql.

tgross avatar tgross commented on August 20, 2024

I took a crack at that in #57

from mysql.

neuroserve avatar neuroserve commented on August 20, 2024

I've checked out the patch and I was able to build the container but I was not able to start it (and I have no logs to see, what happened). Now I see, you merged the patch already (libmanta.py has the changes), but the container still has the old version, right?

from mysql.

tgross avatar tgross commented on August 20, 2024

Right, that hasn't been released yet. I'm trying to figure out why our 3rd-party test runner isn't working quite right #64 and then I'll cut a new release.

from mysql.

tgross avatar tgross commented on August 20, 2024

Released in https://github.com/autopilotpattern/mysql/releases/tag/5.6r3.1.0

from mysql.

tgross avatar tgross commented on August 20, 2024

@neuroserve autopilotpattern/mysql:latest or autopilotpattern/mysql:5.6r3.1.0 images will have this fix.

from mysql.

neuroserve avatar neuroserve commented on August 20, 2024

And works. Backups are now uploaded to Manta.

from mysql.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.