Batch Processing
Spark: the shuffle, the DAG and the cost of a wide transformation.
Spark: the shuffle, the DAG and the cost of a wide transformation.
dbt and Airflow: transforming data in SQL and scheduling the DAG around it.
pandas and Polars: columnar memory, vectorized expressions, lazy plans and streaming, join and null semantics, and the point where a single machine stops being enough.
Turning a notebook into a parameterized and testable script: arguments and config, logging and exit codes, atomic writes, idempotent runs, validation and scheduling.
Iceberg and Delta Lake: table formats that bring ACID and time travel to object storage.
Flink: event time, watermarks and state in an unbounded stream.