Comments (2)
Recently while working a Support Ticket, I discovered to my very great surprise that it is possible for an Elasticsearch cluster to grow too big. When a cluster grows too big it effectively stops functioning because every single operation returns a circuit breaker failure. Customers whose production clusters have been functioning perfectly well can suddenly find themselves dead in the water with little or no warning.
Not only did this come to a surprise to me, but it came to a surprise to the customer as well. I think both of us thought in RDBMS terms, whereby a data-heavy cluster might have so much data on disk that every operation would cause hundreds of calls to the disk system and would be very slow ... but would still function. In fact this is not true in the case of an elastic node or cluster.
It turns out that the fact that a cluster can grow to be too big is not documented very clearly anywhere that I can find, and I certainly don't recall any discussion of this in any of the Training classes which I have taken or taught.
In my opinion, this is a serious omission. This is an especially serious omission as customers increasingly operate large clusters which are potentially susceptible to operating limit failures.
I'm attaching the 1st draft of a Support Note I've written on the problem. I suggest the documentation should include the following points:
-
that every cluster has a maximum operating limit in terms of the amount of indexed data it can support. That when this limit is reached all operations will fail with circuit breaker exceptions and that the cluster will effectively be down.
-
that the exact limit is difficult to determine and depends on the size of node JVMs, the amount of data being indexed, the nature of the index keys, the amount of field data associated with each index and other factors. The documentation should emphasize that the maximum operating limit is in terabytes of indexed data per node.
-
what can be done proactively to anticipate reaching a cluster's maximum operating limit, specifically by checking various entries in node statistics
from elasticsearch-definitive-guide.
Closing: this enhancement has been moved to elastic/elasticsearch#37664
from elasticsearch-definitive-guide.
Related Issues (20)
- Cluster Health - Drilling Deeper: Finding Problematic Indices - clarifications
- I can't visit the test data page HOT 1
- Mention a link to `FlakeID scheme` explanation.
- tribe
- Possible grammar mistake
- Date Math Typo
- prefix not result
- GET cannot use body HOT 1
- String is no longer a type
- Aggregations do not work because of disabled fielddata
- Sense/Kibana installation and links
- missing value in example query
- text exact match example outdated
- Outdated doc about the parent/child relationship (mapping, indexing, query syntax)
- A mapping example from the doc, gives `mapper_parsing_exception` error HOT 5
- Mismatch between query and explanation
- [DOCS] Replace `/master/` and `/current/` references to Elasticsearch Reference Guide
- [Question] What version of Elasticsearch is the guide currently targeting? HOT 1
- can't start the doc
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from elasticsearch-definitive-guide.