Elasticsearch medium
Why does Elasticsearch strictly limit the default `index.max_result_window` to 10,000 documents?
- A To prevent unauthorized users from scraping the entire dataset.
- B Memory Protection: The coordinating node must fetch and sort all results from all shards before selecting the requested page. Querying page 5,000 with a size of 10 would require sorting 50,010 documents across all shards, causing massive heap pressure.
- C The inverted index logic fails if more than 10,000 pointers are active.
- D It is an artificial limit to encourage the use of the paid 'Elastic Search' plugin.
Answer: B. Memory Protection: The coordinating node must fetch and sort all results from all shards before selecting the requested page. Querying page 5,000 with a size of 10 would require sorting 50,010 documents across all shards, causing massive heap pressure.
Deep pagination on a distributed system is expensive. For large dataset exports, developers should use the `search_after` parameter with a unique Sort Key (like `_id`), which allows Elasticsearch to efficiently 'skip' to the next set of results without re-sorting everything from Page 1.
One system design question a day, with spaced repetition so the ones you miss come back.
Get the appRelated questions
- What are the two primary reasons for configuring one or more 'Replica Shards' for an index?
- When a String field is mapped as `text`, what process is performed to enable full-text search capability?
- What is the primary benefit of splitting an Elasticsearch index into multiple 'Primary Shards'?
- Every document stored in an Elasticsearch index is uniquely tracked by which mandatory metadata field?
- Elasticsearch search results include a metadata field called `_score`. What does this value primarily indicate?
- What is the primary architectural purpose of Elasticsearch in a modern software stack?