playbooks

Data-Oriented Workflow

A strategy for building and iterating on data workflows while minimizing compute time, software + data understanding, and patience.

1. The core idea

When a dataset is large, running an entire workflow end-to-end on the full data is slow and expensive. Every iteration (a bug fix, a new parameter, a renamed column) costs you the full runtime again, and the feedback loop drags out until it is easy to lose the thread entirely. On a batch-scheduled HPC cluster like Alpine, the crunch is sharper still: each iteration costs queue wait time on top of runtime, and the compute is a metered allocation you spend down. A job that runs in minutes can still cost wall-clock hours before it even starts, so the feedback loop stretches by the queue, not just by the code.

The fix is simple: get the workflow working at all on the smallest possible subset of the data, iterate on building your workflow, then scale up. Treat the full dataset as where the workflow becomes real, and run against it only once it is correct and stable. This allows one to make it work, make it right, and make it fast in succession.

1.1 Why the order matters: the three phases mapped onto the pilot loop

That line is the well-known engineering maxim make it work, make it right, make it fast, and the pilot workflow loop maps onto it almost exactly. First pick the smallest meaningful subset (step 1); the three phases then run on that subset:

The phases have to happen in this order, and the pilot enforces that ordering by making each phase cheap before you commit to the next:

Concretely: a workflow that takes 6 hours to run end-to-end on the full data but 20 seconds on a one-plate subset lets you do ~1,000 pilot iterations in a single full-run’s worth of wall-clock time. That ratio is what turns “make it right” from an all-day ordeal into a few minutes of typing, and it is also why the subset must stay small enough to keep in memory (step 1), because once an iteration leaves the notebook session, the feedback loop snaps back open.

So the claim is not just aspirational. The pilot loop is the mechanism by which work → right → fast becomes a practical sequence rather than three things you hope to do eventually: the small subset is what makes “work” cheap, the cheap iterations are what make “right” tractable, and a stable “right” is what earns you the right to spend a full run on “fast.”

1.2 A worked example from real-world Way Lab projects

The SK-N-AS CytoTable pilot is a direct instance of this loop. The source dataset is a single-plate CellProfiler SQLite export (BR00148945.sqlite) of roughly 62 GB, far too large to iterate quickly on. Instead of converting it outright, the pilot shrinks it (make shrink) to a one-image subset, BR00148945_image1.sqlite (~17 MB, down from 62 GB): well A01, site 1, 632 joined cell/nuclei/cytoplasm objects and their six matching source-channel TIFFs. That subset is the smallest thing that still exercises the full conversion path, exactly as step 1 of the pilot workflow loop asks for.

The three phases then play out as predicted:

Note what the lab did not do: try to optimize the 55-minute serial run first, or chase correctness against the 62 GB file. Both would have paid full-run prices (potentially multiple times) for work that belonged in the pilot. The subset is what made “fast” worth tuning, because “right” was already settled.

The phrasing is most commonly attributed to Stephen C. Johnson and Brian W. Kernighan who wrote “first make it work, then make it right, and, finally, make it fast” in a 1983 Byte article. See the entries in References and further reading.

2. When to apply this

Apply this approach any time a single full run would take longer than you are willing to wait for an iteration. A rough proxy is when the data no longer fit comfortably in memory, so each iteration leaves your interactive session and becomes a job you wait on; below that, the feedback loop is fast enough that a pilot adds overhead without much benefit.

3. The pilot workflow loop

  1. Take the smallest meaningful subset. Use a sample small enough to work with comfortably, ideally small enough to load into memory in a notebook or R/Python session, not a distributed framework. A handful of files, a single plate, one time point, or a few thousand rows is often enough. The goal is the least amount of data that still exercises every step of your workflow. It is the stone in Stone Soup: it contributes almost nothing on its own, yet it makes the whole pot boil and draws the real ingredients in around it.

  2. Run the full workflow end-to-end on the subset. Do not optimize individual steps in isolation before the whole pipeline runs. The point of the pilot is to prove the pipeline connects, not to perfect any one stage.

  3. Iterate until it works. Fix bugs, adjust parameters, and refine outputs against the subset. Each iteration is cheap because the data is small. This is where you do the bulk of your development.

  4. Validate the subset against reality. Before scaling up, confirm the subset is representative: compare summary statistics (mean, spread, correlation) between the sample and the full dataset where you can. A pilot built on an unrepresentative slice will mislead you at full scale.

  5. Scale up to the production data. Once the workflow is correct and stable on the subset, run it against the full dataset. Expect this to surface scale-specific issues (memory, I/O, runtime), but logic bugs should already be behind you.

  6. Keep the pilot as an end-to-end test. A working pilot on a tiny subset doubles as a fast regression test for future changes. Re-run it before every full-scale run to catch breakage cheaply.

4. Why this works

This approach is an instance of data-oriented programming (DOP): treat data as a first-class citizen, keep it separate from the code that operates on it, and reason about the computation through the data rather than through the code (Sharvit, 2022). Code is easy to fixate on because it is what we write and read, but code is only a theory about what may happen; only the data that actually flow through it prove what it does. A pipeline that compiles, type-checks, and passes unit tests on synthetic input can still produce garbage on real data, so the work is not complete until the data are present.

The pilot loop subscribes to this by making data present early and cheaply, so it can testify before you commit to an expensive run. A small, in-memory subset is a plain table you can eyeball, slice, summarize, and diff across runs, and step 4’s check that the subset’s statistics match the full dataset’s is a schema check in disguise, the yardstick that lets you trust the sample as a proxy.

The upshot connects back to why the order matters: all three phases are about the code and its relationship to compute, but none of them is complete until data have run through and testified. What differs across phases is which property of the data you are judging: that it appeared at all (“work”), that it is correct (“right”), and that it arrived within acceptable runtime, memory, and I/O at scale (“fast”). “Make it fast” comes last because its verdict is the code’s cost in compute, which is only meaningful measured against the full dataset, so you earn that run only once the data have already proven the logic.

5. References and further reading

This site is open source. Improve this page.