27 February 2025 / Data systems / 8 chapters

Designing resumable large-data pipelines

Gold contact pads and fine traces on a dark printed circuit board.
Photo: Vishnu Mohanan (opens in a new tab)
Read the Introduction

Say an import stops halfway through a big file. Here's how I'd set up the state and restart rules so you can tell what's already done, set aside the records that failed and carry on without running the whole thing again.

Chapter 1: What you need to know when a run stops halfway

Got a take on this?

Share it on X. I read the replies.

All articles