Idempotency, checkpointing and self-healing recovery aren't luxuries — they're the price of sleeping through the night.
Strip away the tools and every batch pipeline is the same shape. The order of thinking — design backward, build forward — that outlasts whatever stack you use this year.
Data pipelines fail silently — the job stays green while the numbers go wrong. Mental models for batch pipelines that fail loudly, and stay correct when they don't.
End-to-end exactly-once is really at-least-once plus idempotency. Getting that right changes how you design every sink.
Late data is normal, not an edge case. Watermarks and windowing handle stragglers without reprocessing the world.
The dual-write, reconcile, and cutover strategy that moved a legacy warehouse to Delta without anyone noticing.
not_null and unique are table stakes. These are the semantic tests that have saved me from silent data incidents.
Safe backfills need idempotency, partitioning, and a throttle. The pattern that lets me reprocess a year of data with confidence.
Monitoring business metrics is too late. Catch breakage at the schema boundary with contracts — before it flows downstream.