跳到主要内容

6 篇文档带有标签「data」

查看所有标签

Batch Processing

Spark: the shuffle, the DAG and the cost of a wide transformation.

Data Pipelines

dbt and Airflow: transforming data in SQL and scheduling the DAG around it.

Ecosystem: Dataframes

pandas and Polars: columnar memory, vectorized expressions, lazy plans and streaming, join and null semantics, and the point where a single machine stops being enough.

Ecosystem: Scripting Workflow

Turning a notebook into a parameterized and testable script: arguments and config, logging and exit codes, atomic writes, idempotent runs, validation and scheduling.

Lakehouse

Iceberg and Delta Lake: table formats that bring ACID and time travel to object storage.