gosquared / aws-elasticsearch-js Goto Github PK

View Code? Open in Web Editor NEW

14.0 3.0 6.0 87 KB

Use elasticsearch-js with Amazon ES

License: MIT License

JavaScript 100.00%

aws-elasticsearch-js's Introduction

AWS Elasticsearch

Use Elastic's official elasticsearch-js with Amazon Elasticsearch.

Install

npm i aws-elasticsearch-js

Usage

const elasticsearch = require('@elastic/elasticsearch');
const { createConnector } = require('aws-elasticsearch-js');

const region = process.env.AWS_REGION || 'us-east-1';
const domain = 'https://domain-1-abc123.region.es.amazonaws.com/';

const client = new elasticsearch.Client({
  nodes: [ domain ],
  Connection: createConnector({ region })
});

// use in the normal way
async function example() {
  await client.index({ index: 'test', body: { test: 'hello' } });
  await client.indices.refresh();
  let result = await client.search({ index: 'test' });
}
example();

Credentials

The connector will use AWS credentials from the environment.

If you have another method of loading / refreshing credentials, you can pass them using getCreds:

// heads up: this is called on every request
const getCreds = cb => {
  // load creds from somewhere...
  const credentials = { accessKeyId, secretAccessKey };
  // or credentials = { sessionToken }
  const err = null; // if you give an error, the request will abort
  cb(err, credentials);
};
createConnector({ region, getCreds });

Options

// options and example values
const options = {
  region: 'us-east-1', // AWS region (defaults to process.env.AWS_REGION)
  getCreds: cb => cb(err, credentials) // see above
};
createConnector(options);

Test

# make sure aws creds are defined in the environment
AWS_REGION=us-east-1 \
AWS_ES_DOMAIN=https://domain-1-abc123.region.es.amazonaws.com/ \
npm test

Troubleshooting

If you get status code 403:

check your code is running with the right role / aws access credentials.
make sure your role / user is authorised in the domain's access policy, or choose "allow open access to the domain" if your domain is in VPC.
make sure you've allowed your user/role if you're using aws elasticsearch's security and access management.
- One quick way to do this is go to Kibana > Security > roles > all_access > mapped users > manage mapping > users/backend roles and add the ARNs there.

aws-elasticsearch-js's People

Contributors

Stargazers

Watchers

Forkers

antonshevel markusz hiredscore lmborba eladass samhstn

aws-elasticsearch-js's Issues

Upgrade to use `@aws-sdk/*` packages

AWS has published the AWS SDK v3 as modular packages: https://aws.amazon.com/blogs/developer/modular-packages-in-aws-sdk-for-javascript/

This package should upgrade to using those packages in order to reduce disk space used after install.

how to set http protocol?

this are my options:

const options = {
  Connection: createConnector({ region, getCreds }),
  index: 'users',
  type: 'users',
  log: 'error',
  protocol: 'http',
  bulk: {
    size: 10, 
    delay: 100,
  },
  saveOnSynchronize: false,
  keepAlive: true,
};

I'm keep getting this error:

But debugging this error:
TypeError [ERR_INVALID_PROTOCOL]: Protocol "https:" not supported. Expected "http:"

It is being generated here: node_modules/aws-elasticsearch-js/index.js line 52
const endpoint = new AWS.Endpoint(domain);

that endpoint is this object:
{ protocol: 'https:', host: 'elasticsearch-dev.website.co', port: 443, hostname: 'elasticsearch-dev.website.co', pathname: '/', path: '/', href: 'https://elasticsearch-dev.website.co/' }

How can I set that protocol to http?

pump is no longer a dependency

As of 2f4bd69, pump is no longer used in the library. It should be removed from package.json.

ES 6.x: Content-Type header is missing

Hi, I can't seem to use the module (v6.8.8), steps to reproduce below:

// env on Elastic Beanstalk EC2 machine:
npm i @elastic/elasticsearch@6
export AWS_REGION='xx-xxxx-1'
export ES_ENDPOINT='https://.../'

// code:
const ES = require('@elastic/elasticsearch');
const { createConnector } = require('aws-elasticsearch-js');
_es = new ES.Client({
    node: process.env.ES_ENDPOINT,
    Connection: createConnector()
});
const q = {
    "index": "lyrics_catalog",
    "type": "_doc",
    "size":1,
    "body": {
        "query": {
            "match_all":{}
       }
    }
}
_es.search(q).then(console.log, console.error)

// output:
Promise {
  <pending>,
  [Symbol(async_id_symbol)]: 3468,
  [Symbol(trigger_async_id_symbol)]: 3465,
  [Symbol(destroyed)]: { destroyed: false }
}
> ResponseError: Response Error
    at IncomingMessage.<anonymous> (/xxx/node_modules/@elastic/elasticsearch/lib/Transport.js:310:25)
    at IncomingMessage.emit (node:events:532:35)
    at IncomingMessage.emit (node:domain:537:15)
    at endReadableNT (node:internal/streams/readable:1346:12)
    at processTicksAndRejections (node:internal/process/task_queues:83:21) {
  meta: {
    body: { error: 'Content-Type header is missing', status: 406 },
    statusCode: 406,
    headers: {
      date: 'Sun, 27 Feb 2022 16:06:53 GMT',
      'content-type': 'application/json; charset=UTF-8',
      'content-length': '55',
      connection: 'keep-alive',
      'access-control-allow-origin': '*'
    },
    warnings: null,
    meta: {
      context: null,
      request: [Object],
      name: 'elasticsearch-js',
      connection: [Object],
      attempts: 0,
      aborted: false
    }
  }
}

Note that adding headers option to the ES client (as described here) breaks the initialization with another error:

Uncaught TypeError: Cannot read properties of undefined (reading 'username')
    at getAuth (/xxx/node_modules/@elastic/elasticsearch/index.js:231:16)
    at new Client (/xxx/node_modules/@elastic/elasticsearch/index.js:70:23)

External Environment Variables overwrite credentials in code

While this might be intentional I have the problem that AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY from my environment are overwriting the credentials stated in the sourcecode. Is there an option to "force" credentials from the code?

Here is my code:

const { Client } = require('@elastic/elasticsearch')
// const elasticsearch = require('@elastic/elasticsearch')
const { createConnector } = require('aws-elasticsearch-js')

const domain = process.env.ES_HOST
const region = process.env.AWS_DEFAULT_REGION
const accessKeyId = process.env.AWS_ACCESS_KEY_ID
const secretAccessKey = process.env.AWS_SECRET_ACCESS_KEY

// heads up: this is called on every request
const getCreds = (cb) => {
	// load creds from somewhere...
	const credentials = { accessKeyId, secretAccessKey }
	// or credentials = { sessionToken }
	const err = null // if you give an error, the request will abort
	cb(err, credentials)
}

export const es = new Client({
	node: domain,
	Connection: createConnector({ region, getCreds }),
})