Comments (10)
jeremy-rutman
commented on August 15, 2024
1
For a plain-vanilla install of portland-metro exactly according to install instructions (no proxies or other changes) , using docker 18.09.4, and docker-compose version 1.23.2 on a fresh ubuntu 18.04.2 LTS on an AWS m5a.4xlarge instance (16 cpu 64 GiB ram, 700Gssd) I get the following:
Aggregate test results
Pass: 196
Improvements: 0
Fail: 41
Placeholders: 0
Regressions: 239
Took 7205ms
Test success rate 49.79%
FATAL ERROR: 239 regression(s) detected.
There were a bunch like this ( I guess these are the 'fails') :
✘ [65] "{"text":"14587 SW DAUER CT, Portland, OR"}": score 3 out of 4
diff:
name
expected: 14587 SW DAUER CT
actual: PORTLAND
and regressions like this :
** ✘ regression [40] "{"text":"17175 SW TVH aloha Oregon"}": score 3 out of 4
diff:
name
expected: 17175 SW TUALATIN VALLEY HIGHWAY
actual: ALOHA
and like this
** ✘ regression [466] "{"text":"309 SE 129TH AVE, Portland, Oregon"}": Unexpected status code 500
As an aside it looks like the 16 cpus are way overkill at least for import , unless I have missed some config file. The load average looks south of 4%.
from docker.
orangejulius
commented on August 15, 2024
Hey @chriszrc,
Sorry, we missed this one somehow.
The warnings in your second code snippet are normal, but the Elasticsearch rejected execution errors are the standard error when Elasticsearch is a bit overloaded.
It does mean that some records weren't inserted into Elasticsearch, so you don't quite have a complete build, although as you can see with your 83% test success rate, you're mostly there (it should be close to but not quite 100% most of the time).
How much CPU power and RAM does the machine you were running this on have? 2-4 CPUs and 8GB RAM is a good minimum. You can also tweak some settings to make things work on an even smaller machine, but the defaults should work well given that level of hardware.
from docker.
chriszrc
commented on August 15, 2024
Hi,
Thanks for getting back to me, that's great to hear that some of these are expected, and for the other errors, it might be a machine issue. This was running in docker, which had access to the resources specified in the readme (4 processors, 8gb ram) as specified in the docker preferences. I'll try increasing it some more and see if that fixes it, thanks!
from docker.
missinglink
commented on August 15, 2024
The es_rejected_execution_exception is caused when the data importer is sending data to elasticsearch faster than it can accept it.
This is most often caused when the elasticsearch process is starved of CPU.
I don't suppose you're running docker on a mac? if so, then docker might not have access to all the resources of your system.
There is some info in the Prerequisites section of the readme which explains how to configure that.
from docker.
chriszrc
commented on August 15, 2024
You are indeed correct, docker on a macbook pro. I did see this section:
https://github.com/pelias/docker#prerequisites
And I did install coreutils, but I didn't completely max out the docker resources, I'll give that a try-
from docker.
missinglink
commented on August 15, 2024
Cool, let us know how you get on, this is a screenshot from the Docker settings on my laptop:
from docker.
jeremy-rutman
commented on August 15, 2024
I'm also seeing a bunch of es_rejected_execution_exeption errors during pelias prepare all :
error: [dbclient-openaddresses] [429] type=es_rejected_execution_exception, reason=rejected execution of org.elasticsearch.transport.TransportService$4@780a9e2c on EsThreadPoolExecutor[bulk, queue capacity = 50, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@5556905f[Running, pool size = 16, active threads = 16, queued tasks = 50, completed tasks = 2456798]]
error: [dbclient-openaddresses] [429] type=es_rejected_execution_exception, reason=rejected execution of org.elasticsearch.transport.TransportService$4@737e3062 on EsThreadPoolExecutor[bulk, queue capacity = 50, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@5556905f[Running, pool size = 16, active threads = 16, queued tasks = 50, completed tasks = 2456798]]
error: [dbclient-openaddresses] [429] type=es_rejected_execution_exception, reason=rejected execution of org.elasticsearch.transport.TransportService$4@780a9e2c on EsThreadPoolExecutor[bulk, queue capacity = 50, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@5556905f[Running, pool size = 16, active threads = 16, queued tasks = 50, completed tasks = 2456798]]
info: [dbclient-openaddresses] retrying batch [429]
info: [dbclient-openaddresses] retrying batch [429]
error: [dbclient-openaddresses] transaction error invalid bulk payload length. Payload received: []
info: [dbclient-openaddresses] retrying batch [429]
info: [dbclient-openaddresses] retrying batch [429]
info: [dbclient-openaddresses] retrying batch [429]
info: [dbclient-openaddresses] retrying batch [429]
error: [dbclient-openaddresses] transaction error reached max retries
info: [dbclient-openaddresses] paused=false, transient=1, current_length=1, indexed=97841000, batch_ok=195682, batch_retries=0, failed_records=0, address=97841000, persec=5350, batch_error=35
and an exotic 'invalid bulk payload length':
error: [dbclient-openaddresses] [429] type=es_rejected_execution_exception, reason=rejected execution of org.elasticsearch.transport.TransportService$4@19226afa on EsThreadPoolExecutor[bulk, queue capacity = 50, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@5556905f[Running, pool size = 16, active threads = 16, queued tasks = 50, completed tasks = 7489140]]
info: [dbclient-openaddresses] retrying batch [429]
info: [dbclient-openaddresses] retrying batch [429]
error: [dbclient-openaddresses] transaction error invalid bulk payload length. Payload received: []
info: [dbclient-openaddresses] retrying batch [429]
info: [dbclient-openaddresses] retrying batch [429]
info: [dbclient-openaddresses] retrying batch [429]
info: [dbclient-openaddresses] retrying batch [429]
error: [dbclient-openaddresses] transaction error reached max retries
info: [dbclient-openaddresses] paused=false, transient=1, current_length=48, indexed=307163000, batch_ok=614326, batch_retries=0, failed_records=0, address=307163000, persec=6600, batch_error=430
from docker.
jeremy-rutman
commented on August 15, 2024
curl -XPUT localhost:9200/_cluster/settings -d '{ "transient" : { "threadpool.bulk.queue_size" : 500 } } at the cli will let you change the queue size, this helped my installs when using a single node , avoiding the rejected_execution_exception
from docker.
chriszrc
commented on August 15, 2024
@orangejulius Ok, I maxed out all the resouces on my macbook, and just as you predicted, the build now completes without any of the elasticsearch exceptions. This is now what I'm seeing for the test run:
Aggregate test results
Pass: 390
Improvements: 2
Fail: 39
Placeholders: 0
Regressions: 45
Took 18248ms
Test success rate 90.51%
Is that what's expected?
from docker.
orangejulius
commented on August 15, 2024
90% success rate looks about right. I also just opened pelias/dbclient#108 which might help setups with less CPU available to have a higher chance of success.
Since I think we've probably figured out what the problem was, I'm going to close this.
from docker.
Related Issues (20)
- Venue data importers
- How to init pelias with the data for multiple countries HOT 2
- pelias elastic create returns invalid host header HOT 3
- PeliasModelError: invalid regex test, https://yavuzgallery.com should not match /https?:\/\// HOT 2
- Self hosted planet build result deviation HOT 5
- Elasticsearch Container keeps restarting HOT 7
- Can't import pelias docker for Quebec HOT 7
- Planet pelias json config openaddresses HOT 9
- Crash / Abort in Node create_thread on Ubuntu 20.04 HOT 1
- Pelias openaddress import seem not working
- North America openaddresses missing data and incomplete
- Missing Address Layer HOT 3
- network pelias_default was found but has incorrect label com.docker.compose.network set to "" HOT 2
- Duplicate key `csv-importer`
- A network with name pelias_default exists but was not created by compose HOT 4
- Cant start docker folowing documentation commands HOT 6
- update TIGER download to use the 2023 data HOT 2
- Interpolation error HOT 3
- Full Planet Import only takes 300GB in Elastic Search
- How to run Pelias geocoder in docker with my own .pbf file data without external data HOT 5
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from docker.