cloudflare / cloudflare-elastic Goto Github PK

License: Apache License 2.0

Shell 7.83% Java 92.17%

cloudflare-elastic's Introduction

cloudflare-elastic

Use Elasticsearch and Kibana to visualize Cloudflare logs

AWS Lambda Function for Forwarding Cloudflare Logs to Elasticsearch

This lambda function will collect Cloudflare logs from an S3 bucket and forward them to an Elasticsearch cluster running on Elastic Cloud.

The official documentation can be found on Cloudflare's site

Build

./gradlew clean build

Install

Upload build/distributions/cloudflare-elastic-aws.zip to AWS. Due to the size of the distribution, you may need to upload the file to an S3 bucket before configuring.

Configure

Configure handler method

com.cloudflare.elastic.ElasticLambdaForwarder::handleRequest

Configure environment variables

Certain environment variables must be configured so that the function can connect to the Elasticsearch cluster.

Environment Variable	Description
elastic_hostname	Fully qualified domain name of the Elasticsearch endpoint
elastic_username	The username of the Elasticsearch user, e.g. elastic
elastic_password	The password of the Elasticsearch user

Additionally, the following environment variables can optionally be configured.

Environment Variable	Description	Default value
elastic_port	Endpoint port number	9243
elastic_index	The index pattern to use	cloudflare-*
elastic_pipeline	The ingest pipeline to use for pre-processing (cloudflare-pipeline-(weekly, daily))	cloudflare-pipeline-weekly
elastic_use_https	Whether to use SSL/TLS to connect	true
elastic_bulk_actions	Number of log messages to send to Elasticsearch per batch; can be tuned for scale and speed	100
elastic_bulk_concurrency	Number of concurrent requests to Elasticsearch; can be tuned for scale and speed	2
elastic_debug	Enable verbose logging	false
aws_access_key	Can be used to override permissions from execution role; typically not needed
aws_secret_key	Can be used to override permissions from execution role; typically not needed

Install Additional Artifacts

Change to the conf directory and issue the following command.

./install-artifacts.sh -u elastic -p <RANDOM PASSWORD> -e https://_<YOUR DEPLOYMENT ENDPOINT>_:9243

This will install the ingest node processor and index templates.

cloudflare-elastic's People

Contributors

Stargazers

Watchers

cloudflare-elastic's Issues

ClassNotFoundException when forwarding events from S3 to Elastic Cloud

Lambda: Failure opening selector

Dear, there is an error in the execution of the lambda, when you see it, it may be an overload of the function for reading a large number of registers.

Error:

org.apache.http.nio.reactor.IOReactorException: Failure opening selector
java.lang.IllegalStateException: org.apache.http.nio.reactor.IOReactorException: Failure opening selector

Metric lambda:
https://ibb.co/tZpZmYt

Thank you for your reply.

Regards

Support for AWS Elasticsearch Service

Issue: Cannot connect to AWS Elastic Service due to requiring Signed Requests, and additionally, whitelisting the IP (and thus bypassing authentication and the AWS Signature requirement) is unable to be established due to the Username and Password field being mandatory.

Suggestion: It seems there a number of "Cloud" offerings for Elasticsearch available, specifically for me AWS but looks like there is a GCP requirement too, it would be good if there could be additional variables that could be set e.g:

Username and Password as optional for requests originating from IP access lists
Port number optional e.g. just HTTPS (Yes, :443 works but it's not really required)
Ability to enable AWS Signature v4 on the request, and specify the required Access_Key and Secret_Key to sign the request with

That's all the features I can think of at the moment - I appreciate there hasn't been a release to this in some time but If you don't ask, you don't get!

Thank You!

Oliver

install-artifacts.sh - Root mapping definition has unsupported parameters

When following the https://developers.cloudflare.com/logs/analytics-integrations/elastic#task-3---configure-the-deployment this error occurs while running the below command:

./install-artifacts.sh -u elastic -p < redacted > -e https://cloudflare-elastic-logs.es.eastus2.azure.elastic-cloud.com:9243

Deployment version
v7.12.1

Installing ingest pipeline to https://cloudflare-elastic-logs.es.eastus2.azure.elastic-cloud.com:9243 for daily indices
{"acknowledged":true}


Installing ingest pipeline to https://cloudflare-elastic-logs.es.eastus2.azure.elastic-cloud.com:9243 for weekly indices
{"acknowledged":true}

Installing index template to https://cloudflare-elastic-logs.es.eastus2.azure.elastic-cloud.com:9243
{"error":{"root_cause":[{"type":"mapper_parsing_exception","reason":"Root mapping definition has unsupported parameters:  [doc : {properties={WAFRuleID={type=text, fields={keyword={ignore_above=256, type=keyword}}}, ZoneID={type=long}, OriginResponseTime={type=long}, OriginResponseHTTPExpires={fields={keyword={ignore_above=256, type=keyword}}, type=text}, RayID={type=keyword}, SecurityLevel={type=keyword}, EdgeResponseCompressionRatio={type=float}, EdgeColoID={type=long}, ClientRequestProtocol={type=text, fields={keyword={ignore_above=256, type=keyword}}}, ClientSrcPort={type=long}, EdgeResponseStatus={type=long}, OriginResponseBytes={type=long}, OriginResponseHTTPLastModified={fields={keyword={ignore_above=256, type=keyword}}, type=text}, ClientSSLProtocol={fields={keyword={ignore_above=256, type=keyword}}, type=text}, EdgePathingOp={type=keyword}, ClientIP={properties={timezone={type=keyword}, ip={type=keyword}, latitude={type=float}, continent_code={type=keyword}, city_name={type=keyword}, dma_code={type=long}, country_code2={type=keyword}, country_name={type=keyword}, country_code3={type=keyword}, location={type=geo_point}, region_name={fields={keyword={ignore_above=256, type=keyword}}, type=text}, postal_code={type=keyword}, longitude={type=float}, region_code={type=keyword}}}, CacheTieredFill={type=boolean}, ClientSSLCipher={fields={keyword={ignore_above=256, type=keyword}}, type=text}, ParentRayID={type=keyword}, EdgeRequestHost={type=keyword}, ClientRequestUserAgent={fields={keyword={ignore_above=512, type=keyword}}, type=text}, EdgePathingSrc={type=keyword}, EdgeRateLimitID={type=long}, ClientDeviceType={type=keyword}, ClientIPClass={fields={keyword={ignore_above=256, type=keyword}}, type=text}, WorkerSubrequestCount={type=long}, OriginSSLProtocol={type=text, fields={keyword={ignore_above=256, type=keyword}}}, WAFMatchedVar={type=text, fields={keyword={ignore_above=256, type=keyword}}}, EdgeRateLimitAction={fields={keyword={ignore_above=256, type=keyword}}, type=text}, EdgeResponseBytes={type=long}, ClientRequestURI={type=text, fields={keyword={ignore_above=512, type=keyword}}}, WorkerSubrequest={type=boolean}, EdgeStartTimestamp={type=date}, WAFFlags={fields={keyword={ignore_above=256, type=keyword}}, type=text}, ClientRequestHost={type=keyword}, WorkerStatus={type=text, fields={keyword={ignore_above=256, type=keyword}}}, ClientRequestPath={type=text, fields={keyword={ignore_above=256, type=keyword}}}, UserAgent={properties={patch={type=keyword}, minor={type=keyword}, major={type=keyword}, os={type=text, fields={keyword={ignore_above=256, type=keyword}}}, build={type=keyword}, os_minor={type=keyword}, os_major={type=keyword}, name={type=keyword}, os_name={type=text, fields={keyword={ignore_above=256, type=keyword}}}, device={type=keyword}}}, @version={type=keyword}, OriginResponseStatus={type=long}, CacheCacheStatus={type=keyword}, OriginIP={type=keyword}, ClientASN={type=long}, WAFProfile={type=text, fields={keyword={ignore_above=256, type=keyword}}}, WAFAction={type=text, fields={keyword={ignore_above=256, type=keyword}}}, ClientCountry={type=keyword}, ClientRequestReferer={fields={keyword={ignore_above=512, type=keyword}}, type=text}, message={fields={keyword={ignore_above=256, type=keyword}}, type=text}, ClientRequestBytes={type=long}, WAFRuleMessage={fields={keyword={ignore_above=256, type=keyword}}, type=text}, WorkerCPUTime={type=long}, EdgeResponseContentType={fields={keyword={ignore_above=256, type=keyword}}, type=text}, @timestamp={type=date}, EdgeServerIP={type=keyword}, EdgePathingStatus={type=keyword}, CacheResponseBytes={type=long}, ClientRequestMethod={type=keyword}, EdgeEndTimestamp={type=date}, CacheResponseStatus={type=long}}}]"}],"type":"mapper_parsing_exception","reason":"Failed to parse mapping [_doc]: Root mapping definition has unsupported parameters:  [doc : {properties={WAFRuleID={type=text, fields={keyword={ignore_above=256, type=keyword}}}, ZoneID={type=long}, OriginResponseTime={type=long}, OriginResponseHTTPExpires={fields={keyword={ignore_above=256, type=keyword}}, type=text}, RayID={type=keyword}, SecurityLevel={type=keyword}, EdgeResponseCompressionRatio={type=float}, EdgeColoID={type=long}, ClientRequestProtocol={type=text, fields={keyword={ignore_above=256, type=keyword}}}, ClientSrcPort={type=long}, EdgeResponseStatus={type=long}, OriginResponseBytes={type=long}, OriginResponseHTTPLastModified={fields={keyword={ignore_above=256, type=keyword}}, type=text}, ClientSSLProtocol={fields={keyword={ignore_above=256, type=keyword}}, type=text}, EdgePathingOp={type=keyword}, ClientIP={properties={timezone={type=keyword}, ip={type=keyword}, latitude={type=float}, continent_code={type=keyword}, city_name={type=keyword}, dma_code={type=long}, country_code2={type=keyword}, country_name={type=keyword}, country_code3={type=keyword}, location={type=geo_point}, region_name={fields={keyword={ignore_above=256, type=keyword}}, type=text}, postal_code={type=keyword}, longitude={type=float}, region_code={type=keyword}}}, CacheTieredFill={type=boolean}, ClientSSLCipher={fields={keyword={ignore_above=256, type=keyword}}, type=text}, ParentRayID={type=keyword}, EdgeRequestHost={type=keyword}, ClientRequestUserAgent={fields={keyword={ignore_above=512, type=keyword}}, type=text}, EdgePathingSrc={type=keyword}, EdgeRateLimitID={type=long}, ClientDeviceType={type=keyword}, ClientIPClass={fields={keyword={ignore_above=256, type=keyword}}, type=text}, WorkerSubrequestCount={type=long}, OriginSSLProtocol={type=text, fields={keyword={ignore_above=256, type=keyword}}}, WAFMatchedVar={type=text, fields={keyword={ignore_above=256, type=keyword}}}, EdgeRateLimitAction={fields={keyword={ignore_above=256, type=keyword}}, type=text}, EdgeResponseBytes={type=long}, ClientRequestURI={type=text, fields={keyword={ignore_above=512, type=keyword}}}, WorkerSubrequest={type=boolean}, EdgeStartTimestamp={type=date}, WAFFlags={fields={keyword={ignore_above=256, type=keyword}}, type=text}, ClientRequestHost={type=keyword}, WorkerStatus={type=text, fields={keyword={ignore_above=256, type=keyword}}}, ClientRequestPath={type=text, fields={keyword={ignore_above=256, type=keyword}}}, UserAgent={properties={patch={type=keyword}, minor={type=keyword}, major={type=keyword}, os={type=text, fields={keyword={ignore_above=256, type=keyword}}}, build={type=keyword}, os_minor={type=keyword}, os_major={type=keyword}, name={type=keyword}, os_name={type=text, fields={keyword={ignore_above=256, type=keyword}}}, device={type=keyword}}}, @version={type=keyword}, OriginResponseStatus={type=long}, CacheCacheStatus={type=keyword}, OriginIP={type=keyword}, ClientASN={type=long}, WAFProfile={type=text, fields={keyword={ignore_above=256, type=keyword}}}, WAFAction={type=text, fields={keyword={ignore_above=256, type=keyword}}}, ClientCountry={type=keyword}, ClientRequestReferer={fields={keyword={ignore_above=512, type=keyword}}, type=text}, message={fields={keyword={ignore_above=256, type=keyword}}, type=text}, ClientRequestBytes={type=long}, WAFRuleMessage={fields={keyword={ignore_above=256, type=keyword}}, type=text}, WorkerCPUTime={type=long}, EdgeResponseContentType={fields={keyword={ignore_above=256, type=keyword}}, type=text}, @timestamp={type=date}, EdgeServerIP={type=keyword}, EdgePathingStatus={type=keyword}, CacheResponseBytes={type=long}, ClientRequestMethod={type=keyword}, EdgeEndTimestamp={type=date}, CacheResponseStatus={type=long}}}]","caused_by":{"type":"mapper_parsing_exception","reason":"Root mapping definition has unsupported parameters:  [doc : {properties={WAFRuleID={type=text, fields={keyword={ignore_above=256, type=keyword}}}, ZoneID={type=long}, OriginResponseTime={type=long}, OriginResponseHTTPExpires={fields={keyword={ignore_above=256, type=keyword}}, type=text}, RayID={type=keyword}, SecurityLevel={type=keyword}, EdgeResponseCompressionRatio={type=float}, EdgeColoID={type=long}, ClientRequestProtocol={type=text, fields={keyword={ignore_above=256, type=keyword}}}, ClientSrcPort={type=long}, EdgeResponseStatus={type=long}, OriginResponseBytes={type=long}, OriginResponseHTTPLastModified={fields={keyword={ignore_above=256, type=keyword}}, type=text}, ClientSSLProtocol={fields={keyword={ignore_above=256, type=keyword}}, type=text}, EdgePathingOp={type=keyword}, ClientIP={properties={timezone={type=keyword}, ip={type=keyword}, latitude={type=float}, continent_code={type=keyword}, city_name={type=keyword}, dma_code={type=long}, country_code2={type=keyword}, country_name={type=keyword}, country_code3={type=keyword}, location={type=geo_point}, region_name={fields={keyword={ignore_above=256, type=keyword}}, type=text}, postal_code={type=keyword}, longitude={type=float}, region_code={type=keyword}}}, CacheTieredFill={type=boolean}, ClientSSLCipher={fields={keyword={ignore_above=256, type=keyword}}, type=text}, ParentRayID={type=keyword}, EdgeRequestHost={type=keyword}, ClientRequestUserAgent={fields={keyword={ignore_above=512, type=keyword}}, type=text}, EdgePathingSrc={type=keyword}, EdgeRateLimitID={type=long}, ClientDeviceType={type=keyword}, ClientIPClass={fields={keyword={ignore_above=256, type=keyword}}, type=text}, WorkerSubrequestCount={type=long}, OriginSSLProtocol={type=text, fields={keyword={ignore_above=256, type=keyword}}}, WAFMatchedVar={type=text, fields={keyword={ignore_above=256, type=keyword}}}, EdgeRateLimitAction={fields={keyword={ignore_above=256, type=keyword}}, type=text}, EdgeResponseBytes={type=long}, ClientRequestURI={type=text, fields={keyword={ignore_above=512, type=keyword}}}, WorkerSubrequest={type=boolean}, EdgeStartTimestamp={type=date}, WAFFlags={fields={keyword={ignore_above=256, type=keyword}}, type=text}, ClientRequestHost={type=keyword}, WorkerStatus={type=text, fields={keyword={ignore_above=256, type=keyword}}}, ClientRequestPath={type=text, fields={keyword={ignore_above=256, type=keyword}}}, UserAgent={properties={patch={type=keyword}, minor={type=keyword}, major={type=keyword}, os={type=text, fields={keyword={ignore_above=256, type=keyword}}}, build={type=keyword}, os_minor={type=keyword}, os_major={type=keyword}, name={type=keyword}, os_name={type=text, fields={keyword={ignore_above=256, type=keyword}}}, device={type=keyword}}}, @version={type=keyword}, OriginResponseStatus={type=long}, CacheCacheStatus={type=keyword}, OriginIP={type=keyword}, ClientASN={type=long}, WAFProfile={type=text, fields={keyword={ignore_above=256, type=keyword}}}, WAFAction={type=text, fields={keyword={ignore_above=256, type=keyword}}}, ClientCountry={type=keyword}, ClientRequestReferer={fields={keyword={ignore_above=512, type=keyword}}, type=text}, message={fields={keyword={ignore_above=256, type=keyword}}, type=text}, ClientRequestBytes={type=long}, WAFRuleMessage={fields={keyword={ignore_above=256, type=keyword}}, type=text}, WorkerCPUTime={type=long}, EdgeResponseContentType={fields={keyword={ignore_above=256, type=keyword}}, type=text}, @timestamp={type=date}, EdgeServerIP={type=keyword}, EdgePathingStatus={type=keyword}, CacheResponseBytes={type=long}, ClientRequestMethod={type=keyword}, EdgeEndTimestamp={type=date}, CacheResponseStatus={type=long}}}]"}},"status":400}

Object mapping issue and failed to execute bulk item

I am getting following issue shown in elasticsearch.log
[2019-08-19T16:03:30,213][DEBUG][o.e.a.b.TransportShardBulkAction] [9pMb_Kb] [cloudflare-2019-08-19][0] failed to execute bulk item (index) index {[<cloudflare-{2019-08-19||/w{yyyy-MM-dd|UTC}}>][doc][ezrCqWwBC8sWyWQexwgZ], source[n/a, actual length: [2kb], max length: 2kb]} org.elasticsearch.index.mapper.MapperParsingException: object mapping for [ClientIP] tried to parse field [ClientIP] as object, but found a concrete value at org.elasticsearch.index.mapper.DocumentParser.parseObjectOrNested(DocumentParser.java:360) ~[elasticsearch-6.7.2.jar:6.7.2]

Moreover I have opened the thread in elastic forum but didn't get any solution.
https://discuss.elastic.co/t/object-mapping-issue/195739/15

I would appreciate your consideration of this matter

Regards,
Usama

Migrate field names to Elastic Common Schema (ECS)

Migrate all Cloudflare field names to be consistent with Elastic Common Schema. This will enable Cloudflare users to take advantage of new Kibana features such as SIEM.

Ingest Pipeline issue for upgrading to Elastic/Kibana v8

Before upgrading, users run the "Upgrade Assistant" to determine if there are any issues that need to be resolved before upgrading to the next major version. For the upgrade from 7.x to 8.x the assistant points out an issue with the cloudflare ingest pipelines:

**The User-Agent ingest processor's ecs parameter is deprecated**
Remove the ecs parameter from your ingest pipelines. The User-Agent ingest processor always returns Elastic Common Schema (ECS) fields in 8.0.

5 minute blocks

Hi - in Kibana, i am getting 5 minute chunks of data coming in from the Lambda function, or the way it is being indexed. This is not a true representation of the web requests.

I am using the default ingest pipelines which i believe is the weekly index.

ps. I am using this is data more for security that anything, can i remove the weekly/daily rounding and just have a "normal" index? If so how?

Change type to _doc

Change dashboards and index templates to use _doc for the document type in preparation for supporting Elasticsearch 7.x.

errors ingesting logs

index [cloudflare-2020-07-27], type [doc], id [YrQipHMBdg6HnJMNnOLH], message [ElasticsearchException[Elasticsearch exception [type=mapper_parsing_exception, reason=object mapping for [ClientIP] tried to parse field [ClientIP] as object, but found a concrete value]]]

any idea?

Scripted fields incompatible with Elasticsearch/Kibana v8

We're on version 7.x and the upgrade assistant gives the following output:

You have 1 index patterns (cf-http-*...) that use scripted fields. Scripted fields are deprecated and will be removed in future. Use runtime fields instead.

Issues reported by the upgrade assistant must fixed before upgrading to v8. Seems like multiple fields are scripted. That logic will likely need to be moved from scripted fields to either runtime fields or the ingest pipeline.

Reference:
https://www.elastic.co/de/blog/introducing-elasticsearch-runtime-fields
https://www.elastic.co/guide/en/elasticsearch/reference/7.17/runtime.html

Adaptation for GCP.

Hello!
Now also popular GCP cloud. Are you planning adapt this tool to GCP?
Thanks!

Issue with geo_point mapping upon log import into ElasticSearch

Hello. I'm attempting to leverage these Cloudflare Elastic integration config files and I'm running into an issue that is preventing the logs from being imported. Here is a detailed account of my issue:

I’m attempting to follow the instructions here:
https://developers.cloudflare.com/logs/analytics-integrations/elastic/

I’m not using the Elastic Cloud, but my own installation. I’ve got both ElasticSearch and Kibana set up and talking to each other, your ingest pipelines imported, the index template imported, the AWS lambda function deployed, and logs flowing into S3. When the Lamdba function is trying to send the logs into Elastic is where I’m encountering the issue.

I do see a ton of these errors in my “cluster.log” file on my Elastic EC2 server every time it seems to be trying to index logs:

[2020-05-20T12:06:31,803][INFO ][o.e.a.b.TransportShardBulkAction] [logs-node-1] [cloudflare-2020-05-18][0] mapping update rejected by primary
java.lang.IllegalArgumentException: mapper [source.geo.location] of different type, current_type [geo_point], merged_type [ObjectMapper]

Here is what I’m seeing in the AWS Cloudwatch logs from the Lamdba function:

So, this seems to be having an issue with the “geo-point” data type.

Looking in your cloudflare-index-template.json file, I do see this:

And I can see this mapping in the your “weekly” ingest pipeline:

Doing a quick bit of research, “geoip” seems to be available in Logstash:

https://www.elastic.co/guide/en/logstash/current/plugins-filters-geoip.html

I did not install Logstash since I didn’t think it was needed for this implementation. Looks like “geoip” might be used to derive all of the other properties (timezone, city, etc) all from the “ClientIP” field in the logs. However, I do see that this is also available as a default Elastic ingest processor:

https://www.elastic.co/guide/en/elasticsearch/reference/7.7/geoip-processor.html

When I run a quick API call against Elastic to look for available plugins, I do see geoip referenced:

GET "###.##.###.##:9243/_nodes/plugins" | python -m json.tool | grep geo

{
                    "classname": "org.elasticsearch.ingest.geoip.IngestGeoIpPlugin",
                    "description": "Ingest processor that uses looksup geo data based on ip adresses using the Maxmind geo database",
                    "elasticsearch_version": "7.7.0",
                    "extended_plugins": [],
                    "has_native_controller": false,
                    "java_version": "1.8",
                    "name": "ingest-geoip",
                    "version": "7.7.0"
                }

So, it does seem that I have this installed, as far as I can tell. I did a real quick test to make sure geoip is working properly. I ran a couple of API commands to create a small pipeline with just the “geoip”:

curl --user <user>:<password> -X PUT "###.##.###.##:9243/_ingest/pipeline/testgeoip" -H "Content-Type: application/json" -d '{"description" : "Add geoip info","processors" : [{"geoip" : {"field" : "ip"}}]}'

I then created a small index using that pipeline with just a random IP:

curl --user <user>:<password> -X PUT "###.##.###.##:9243/my_index/_doc/my_id?pipeline=testgeoip" -H "Content-Type: application/json" -d '{"ip":"8.8.8.8"}'

I then fetched the contents of the index:

curl --user <user>:<password> -X GET "###.##.###.##:9243/my_index/_doc/my_id" | python -m json.tool

{
"_id": "my_id",
"_index": "my_index",
"_primary_term": 1,
"_seq_no": 0,
"_source": {
      "geoip": {
      "continent_name": "North America",
      "country_iso_code": "US",
      "location": {
      "lat": 37.751,
      "lon": -97.822
      }
},
"ip": "8.8.8.8"
},
"_type": "_doc",
"_version": 1,
"found": true
}

So, it seems that geoip is working. However, the error message is targeting the “location” field specifically. Here, it looks to be an object (lat and lon values). I have also performed a more accurate test:

Create pipeline (pulled from the Cloudflare file):

PUT /_ingest/pipeline/jmggeoip
{
  "description": "Jason Log Pipeline",
  "processors": [
    {
      "geoip": {
        "field": "ClientIP",
        "target_field": "source.geo",
        "properties": [
          "ip",
          "country_name",
          "continent_name",
          "region_iso_code",
          "region_name",
          "city_name",
          "timezone",
          "location"
        ]
      }
    }
  ]
}

Create index template mapping (pulled from the Cloudflare file):

PUT /_template/jmgtemplate
{
   "index_patterns": [
     "jmgindex-*"
   ],
   "mappings": {
      "properties": {
         "source.geo": {
            "properties": {
               "ip": {
                  "type": "ip"
               },
               "postal_code": {
                  "type": "keyword"
               },
               "location": {
                  "type": "geo_point"
               },
               "dma_code": {
                  "type": "long"
               },
               "country_code3": {
                  "type": "keyword"
               },
               "latitude": {
                  "type": "float"
               },
               "longitude": {
                  "type": "float"
               },
               "region_name": {
                  "type": "keyword"
               },
               "city_name": {
                  "type": "keyword"
               },
               "timezone": {
                  "type": "keyword"
               },
               "country_code2": {
                  "type": "keyword"
               },
               "continent_code": {
                  "type": "keyword"
               },
               "country_name": {
                  "type": "keyword"
               },
               "region_code": {
                  "type": "keyword"
               },
               "continent_name": {
                  "type": "keyword"
               },
               "region_iso_code": {
                  "type": "keyword"
              }
            }
         }
      }
   },
   "settings": {
      "index": {
         "number_of_shards": "1",
         "number_of_replicas": "1",
         "mapping.ignore_malformed": true
      }
   }
}

Create index (index pattern matching above and pipline created above):

PUT /jmgindex-test/_doc/my_id?pipeline=jmggeoip
{"ClientIP":"8.8.8.8"}

Fetch the index:
GET /jmgindex-test/_doc/my_id

This call returns the following information:

{
  "_index" : "jmgindex-test",
  "_type" : "_doc",
  "_id" : "my_id",
  "_version" : 1,
  "_seq_no" : 0,
  "_primary_term" : 1,
  "found" : true,
  "_source" : {
    "source" : {
      "geo" : {
        "continent_name" : "North America",
        "timezone" : "America/Chicago",
        "ip" : "8.8.8.8",
        "country_name" : "United States",
        "location" : {
          "lon" : -97.822,
          "lat" : 37.751
        }
      }
    },
    "ClientIP" : "8.8.8.8"
  }
}

So, as you can see, we are still getting latitude and longitude back. Now, let’s look at the field mapping:

Now, we are properly mapping to “geo_point”. However, while this example seems to be working, the ingest process I set up for Cloudflare is not working. So, there must be something about the setup process that is missing. Just to reiterate, here is the error I’m getting when your Lamdba function tries to insert a log into ElasticSearch:

This is why I’m hitting a wall. Everything “seems” to be setup properly from the Elastic side and I think the above proves the geo_point mapping and geoip functionality is working fine.

If this is a bug, please let me know. Otherwise, I would appreciate some assistance in narrowing down this issue. Thanks.

failure in bulk execution

Hello, i am using Elastic Cloud, 7.8.0, and getting this failure in Cloudwatch logs.

failure in bulk execution:
[0]: index [cloudflare], type [doc], id [null], message [ElasticsearchException[Elasticsearch exception [type=illegal_argument_exception, reason=field [ClientRequestProtocol] not present as part of path [ClientRequestProtocol]]]]

I can see this in the ingest pipelines

{
      "dissect": {
        "field": "ClientRequestProtocol",
        "pattern": "HTTP/%{http.version}"
      }
    },
    {
      "rename": {
        "field": "ClientRequestProtocol",
        "target_field": "cloudflare.client.request.protocol",
        "ignore_missing": true
      }
    }

and this in the index template as a field

"cloudflare.client.request.protocol" : {
            "type": "keyword"
         }

Everything here looks "fine" i think..

Please help.

connection to an ES using a self-signed certificate?

Hi,
any idea on how I could do this?

NDJSON format for dashboard.json

Hello in the tutorial (https://developers.cloudflare.com/logs/analytics-integrations/elastic) please take note of the newer NDJSON format coming soon.

Writing to a fixed index

Elastic.co ILM currently requires the writer to write to a fixed alias index, and let Elastic Index Lifecycle Management handle which actual index is written to. Does this repository support that?

Blog on ILM: https://www.elastic.co/blog/optimizing-costs-elastic-cloud-hot-warm-index-lifecycle-management