Handling Large Blobs hard
In distributed file systems like HDFS, why is storing billions of small files (KB size) problematic?
- A It wastes disk space.
- B It is slow to read.
- C NameNode Memory Pressure. The metadata for a file (permissions, location) takes ram (~150 bytes). Billions of files consume hundreds of GB of RAM on the NameNode, crashing the cluster.
- D It is impossible.
Answer: C. NameNode Memory Pressure. The metadata for a file (permissions, location) takes ram (~150 bytes). Billions of files consume hundreds of GB of RAM on the NameNode, crashing the cluster.
Solution: HAR (Hadoop Archive) or aggregating small files into SequenceFiles.
One system design question a day, with spaced repetition so the ones you miss come back.
Get the appRelated questions
- Why do modern sites serve images in WebP or AVIF format?
- What does the acronym BLOB stand for?
- Why do we sometimes convert binary images to Base64 strings?
- What is an 'Origin Shield' or 'Mid-Tier Cache' in a CDN architecture?
- For a feed with 100 images, why use 'Lazy Loading'?
- How does HLS/DASH handle a user moving from WiFi to 4G (Variable Bandwidth)?