Unable to post to ssl endpoint with custom CA about scrapy-elasticsearch HOT 7 CLOSED

@blee1170 so you have custom cert any key files? I can expose a few more ssl related settings like what's provided in example (https://elasticsearch-py.readthedocs.io/en/master/#ssl-and-authentication), will these be sufficient?

from scrapy-elasticsearch.

Something like this:

es = Elasticsearch(
    ['localhost', 'otherhost'],
    http_auth=('user', 'secret'),
    port=443,
    use_ssl=True,
    ca_certs='/path/to/cacert.pem',
    client_cert='/path/to/client_cert.pem',
    client_key='/path/to/client_key.pem',
)

Would help a ton.

from scrapy-elasticsearch.

ok, I will add these over the weekend.

from scrapy-elasticsearch.

That would be fantastic.

from scrapy-elasticsearch.

@blee1170 I checked in an open PR (#68), since I don't have a testing environment to test custom CA files, do you mind pulling this pr to your local environment to give it a test before I merge?

You will need to clone this repo, check out the custom_ca branch and issue a python setup.py install. It accepts a new setting ELASTICSEARCH_CA (dictionary, see https://github.com/knockrentals/scrapy-elasticsearch/pull/68/files#diff-88b99bb28683bd5b7e3a204826ead112R49) for you to pass in ca files.

Let me know if it works and your thoughts.

from scrapy-elasticsearch.

@jayzeng We did some testing and it seems to work for us!

This is the config we used:

ELASTICSEARCH_CA = {
'CA_CERT': os.environ['CA_BUNDLE_PATH'],
'CLIENT_KEY': None,
'CLIENT_CERT': None
}

We are using a bundle, so we need not need the key and cert. Still had to set those values to 'None'. But everything is working well. Let me know when its released.

from scrapy-elasticsearch.

@blee1170 thanks for testing and glad it works, I just cut the latest release (https://pypi.python.org/pypi/ScrapyElasticSearch/0.9.0), cheers!

from scrapy-elasticsearch.