Hadoop Ecosystem¶
Hadoop is not a single product, but rather a software family. Modern data ecosystems have evolved to prioritize the separation of compute and storage (e.g., cloud-native object storage like S3 replacing HDFS in many cloud deployments).
Its core and common components consist of the following:
- HDFS: The distributed file system for Hadoop, now often supplemented or replaced by cloud object stores (e.g., AWS S3, Azure Data Lake).
- YARN: The resource management and job scheduling technology.
- Hive: Used to facilitate structure for the data, running SQL-like queries.
- HBase: A scalable, distributed, NoSQL database built on top of Hadoop.
- Spark: A unified analytics engine for large-scale data processing, largely replacing MapReduce.
- Kafka / Flink: Often integrated for real-time stream processing.
- Packaging for Hadoop: BigTop
Modern Big Data Architecture¶
+-----------------------+ +-------------------------+ +-------------------+
| Data Sources | | Compute / Processing | | Storage |
| (DBs, Streams, Logs) +------>+ (Spark, Flink, Hive) +<----->+ (S3, ADLS, HDFS) |
+-----------------------+ +-------------------------+ +-------------------+
^ |
| v
+-------------------------+ +-------------------+
| Orchestration | | Serving |
| (Airflow, Dagster) | | (HBase, Trino) |
+-------------------------+ +-------------------+
Hadoop and Mongo¶
AWS EMR¶
Amazon EMR is a cloud-native big data platform that simplifies running big data frameworks. Modern EMR includes:
- Hadoop (HDFS, YARN)
- Spark
- Hive
- HBase
- Presto / Trino
- Flink
- Zeppelin
- ZooKeeper