Project documentation#

Supported feature table#

Generated by scripts/docs/render_feature_table.py by joining spec/features.yaml (the frozen empirical CellProfiler 4.2.8 oracle inventory, M1 – never hand-edited or mutated by this script) with the real parity reports (benchmarks/reports/native-parity.json, benchmarks/reports/wasm-parity.json, M3-M5) and the cp_measure mapping (spec/cp_measure-map.yaml, M2). Regenerate after any parity re-run; do not hand-edit.

Total declared columns: 119 (see spec/compatibility.md for the full scope contract).

Module

Feature

Object scope

Dims

Rust

WASM

cp_measure

MeasureImageIntensity

Intensity_LowerQuartileIntensity_Actin

Image

2/3

pass_exact

pass_exact

unsupported

MeasureImageIntensity

Intensity_LowerQuartileIntensity_DNA

Image

2/3

pass_exact

pass_exact

unsupported

MeasureImageIntensity

Intensity_MADIntensity_Actin

Image

2/3

pass_exact

pass_exact

unsupported

MeasureImageIntensity

Intensity_MADIntensity_DNA

Image

2/3

pass_exact

pass_exact

unsupported

MeasureImageIntensity

Intensity_MaxIntensity_Actin

Image

2/3

pass_exact

pass_exact

unsupported

MeasureImageIntensity

Intensity_MaxIntensity_DNA

Image

2/3

pass_exact

pass_exact

unsupported

MeasureImageIntensity

Intensity_MeanIntensity_Actin

Image

2/3

pass_exact

pass_exact

unsupported

MeasureImageIntensity

Intensity_MeanIntensity_DNA

Image

2/3

pass_tolerance

pass_tolerance

unsupported

MeasureImageIntensity

Intensity_MedianIntensity_Actin

Image

2/3

pass_exact

pass_exact

unsupported

MeasureImageIntensity

Intensity_MedianIntensity_DNA

Image

2/3

pass_exact

pass_exact

unsupported

MeasureImageIntensity

Intensity_MinIntensity_Actin

Image

2/3

pass_exact

pass_exact

unsupported

MeasureImageIntensity

Intensity_MinIntensity_DNA

Image

2/3

pass_exact

pass_exact

unsupported

MeasureImageIntensity

Intensity_PercentMaximal_Actin

Image

2/3

pass_exact

pass_exact

unsupported

MeasureImageIntensity

Intensity_PercentMaximal_DNA

Image

2/3

pass_exact

pass_exact

unsupported

MeasureImageIntensity

Intensity_Percentile_10_Actin

Image

2/3

pass_exact

pass_exact

unsupported

MeasureImageIntensity

Intensity_Percentile_10_DNA

Image

2/3

pass_tolerance

pass_tolerance

unsupported

MeasureImageIntensity

Intensity_Percentile_50_Actin

Image

2/3

pass_exact

pass_exact

unsupported

MeasureImageIntensity

Intensity_Percentile_50_DNA

Image

2/3

pass_exact

pass_exact

unsupported

MeasureImageIntensity

Intensity_Percentile_90_Actin

Image

2/3

pass_exact

pass_exact

unsupported

MeasureImageIntensity

Intensity_Percentile_90_DNA

Image

2/3

pass_tolerance

pass_tolerance

unsupported

MeasureImageIntensity

Intensity_StdIntensity_Actin

Image

2/3

pass_tolerance

pass_tolerance

unsupported

MeasureImageIntensity

Intensity_StdIntensity_DNA

Image

2/3

pass_tolerance

pass_tolerance

unsupported

MeasureImageIntensity

Intensity_TotalArea_Actin

Image

2/3

pass_exact

pass_exact

unsupported

MeasureImageIntensity

Intensity_TotalArea_DNA

Image

2/3

pass_exact

pass_exact

unsupported

MeasureImageIntensity

Intensity_TotalIntensity_Actin

Image

2/3

pass_exact

pass_exact

unsupported

MeasureImageIntensity

Intensity_TotalIntensity_DNA

Image

2/3

pass_tolerance

pass_tolerance

unsupported

MeasureImageIntensity

Intensity_UpperQuartileIntensity_Actin

Image

2/3

pass_exact

pass_exact

unsupported

MeasureImageIntensity

Intensity_UpperQuartileIntensity_DNA

Image

2/3

pass_tolerance

pass_tolerance

unsupported

MeasureObjectIntensity

Intensity_IntegratedIntensityEdge_Actin

Cells

2/3

pass_exact

pass_exact

mapped

MeasureObjectIntensity

Intensity_IntegratedIntensityEdge_Actin

Nuclei

2/3

pass_exact

pass_exact

mapped

MeasureObjectIntensity

Intensity_IntegratedIntensityEdge_DNA

Cells

2/3

pass_exact

pass_exact

mapped

MeasureObjectIntensity

Intensity_IntegratedIntensityEdge_DNA

Nuclei

2/3

pass_exact

pass_exact

mapped

MeasureObjectIntensity

Intensity_IntegratedIntensity_Actin

Cells

2/3

pass_exact

pass_exact

mapped

MeasureObjectIntensity

Intensity_IntegratedIntensity_Actin

Nuclei

2/3

pass_exact

pass_exact

mapped

MeasureObjectIntensity

Intensity_IntegratedIntensity_DNA

Cells

2/3

pass_exact

pass_exact

mapped

MeasureObjectIntensity

Intensity_IntegratedIntensity_DNA

Nuclei

2/3

pass_exact

pass_exact

mapped

MeasureObjectIntensity

Intensity_LowerQuartileIntensity_Actin

Cells

2/3

pass_exact

pass_exact

mapped

MeasureObjectIntensity

Intensity_LowerQuartileIntensity_Actin

Nuclei

2/3

pass_exact

pass_exact

mapped

MeasureObjectIntensity

Intensity_LowerQuartileIntensity_DNA

Cells

2/3

pass_exact

pass_exact

mapped

MeasureObjectIntensity

Intensity_LowerQuartileIntensity_DNA

Nuclei

2/3

pass_exact

pass_exact

mapped

MeasureObjectIntensity

Intensity_MADIntensity_Actin

Cells

2/3

pass_exact

pass_exact

mapped

MeasureObjectIntensity

Intensity_MADIntensity_Actin

Nuclei

2/3

pass_exact

pass_exact

mapped

MeasureObjectIntensity

Intensity_MADIntensity_DNA

Cells

2/3

pass_exact

pass_exact

mapped

MeasureObjectIntensity

Intensity_MADIntensity_DNA

Nuclei

2/3

pass_exact

pass_exact

mapped

MeasureObjectIntensity

Intensity_MassDisplacement_Actin

Cells

2/3

pass_exact

pass_exact

mapped

MeasureObjectIntensity

Intensity_MassDisplacement_Actin

Nuclei

2/3

pass_exact

pass_exact

mapped

MeasureObjectIntensity

Intensity_MassDisplacement_DNA

Cells

2/3

pass_exact

pass_exact

mapped

MeasureObjectIntensity

Intensity_MassDisplacement_DNA

Nuclei

2/3

pass_exact

pass_exact

mapped

MeasureObjectIntensity

Intensity_MaxIntensityEdge_Actin

Cells

2/3

pass_exact

pass_exact

mapped

MeasureObjectIntensity

Intensity_MaxIntensityEdge_Actin

Nuclei

2/3

pass_exact

pass_exact

mapped

MeasureObjectIntensity

Intensity_MaxIntensityEdge_DNA

Cells

2/3

pass_exact

pass_exact

mapped

MeasureObjectIntensity

Intensity_MaxIntensityEdge_DNA

Nuclei

2/3

pass_exact

pass_exact

mapped

MeasureObjectIntensity

Intensity_MaxIntensity_Actin

Cells

2/3

pass_exact

pass_exact

mapped

MeasureObjectIntensity

Intensity_MaxIntensity_Actin

Nuclei

2/3

pass_exact

pass_exact

mapped

MeasureObjectIntensity

Intensity_MaxIntensity_DNA

Cells

2/3

pass_exact

pass_exact

mapped

MeasureObjectIntensity

Intensity_MaxIntensity_DNA

Nuclei

2/3

pass_exact

pass_exact

mapped

MeasureObjectIntensity

Intensity_MeanIntensityEdge_Actin

Cells

2/3

pass_exact

pass_exact

mapped

MeasureObjectIntensity

Intensity_MeanIntensityEdge_Actin

Nuclei

2/3

pass_exact

pass_exact

mapped

MeasureObjectIntensity

Intensity_MeanIntensityEdge_DNA

Cells

2/3

pass_exact

pass_exact

mapped

MeasureObjectIntensity

Intensity_MeanIntensityEdge_DNA

Nuclei

2/3

pass_exact

pass_exact

mapped

MeasureObjectIntensity

Intensity_MeanIntensity_Actin

Cells

2/3

pass_exact

pass_exact

mapped

MeasureObjectIntensity

Intensity_MeanIntensity_Actin

Nuclei

2/3

pass_exact

pass_exact

mapped

MeasureObjectIntensity

Intensity_MeanIntensity_DNA

Cells

2/3

pass_exact

pass_exact

mapped

MeasureObjectIntensity

Intensity_MeanIntensity_DNA

Nuclei

2/3

pass_exact

pass_exact

mapped

MeasureObjectIntensity

Intensity_MedianIntensity_Actin

Cells

2/3

pass_exact

pass_exact

mapped

MeasureObjectIntensity

Intensity_MedianIntensity_Actin

Nuclei

2/3

pass_exact

pass_exact

mapped

MeasureObjectIntensity

Intensity_MedianIntensity_DNA

Cells

2/3

pass_exact

pass_exact

mapped

MeasureObjectIntensity

Intensity_MedianIntensity_DNA

Nuclei

2/3

pass_exact

pass_exact

mapped

MeasureObjectIntensity

Intensity_MinIntensityEdge_Actin

Cells

2/3

pass_exact

pass_exact

mapped

MeasureObjectIntensity

Intensity_MinIntensityEdge_Actin

Nuclei

2/3

pass_exact

pass_exact

mapped

MeasureObjectIntensity

Intensity_MinIntensityEdge_DNA

Cells

2/3

pass_exact

pass_exact

mapped

MeasureObjectIntensity

Intensity_MinIntensityEdge_DNA

Nuclei

2/3

pass_exact

pass_exact

mapped

MeasureObjectIntensity

Intensity_MinIntensity_Actin

Cells

2/3

pass_exact

pass_exact

mapped

MeasureObjectIntensity

Intensity_MinIntensity_Actin

Nuclei

2/3

pass_exact

pass_exact

mapped

MeasureObjectIntensity

Intensity_MinIntensity_DNA

Cells

2/3

pass_exact

pass_exact

mapped

MeasureObjectIntensity

Intensity_MinIntensity_DNA

Nuclei

2/3

pass_exact

pass_exact

mapped

MeasureObjectIntensity

Intensity_StdIntensityEdge_Actin

Cells

2/3

pass_exact

pass_exact

mapped

MeasureObjectIntensity

Intensity_StdIntensityEdge_Actin

Nuclei

2/3

pass_exact

pass_exact

mapped

MeasureObjectIntensity

Intensity_StdIntensityEdge_DNA

Cells

2/3

pass_exact

pass_exact

mapped

MeasureObjectIntensity

Intensity_StdIntensityEdge_DNA

Nuclei

2/3

pass_exact

pass_exact

mapped

MeasureObjectIntensity

Intensity_StdIntensity_Actin

Cells

2/3

pass_exact

pass_exact

mapped

MeasureObjectIntensity

Intensity_StdIntensity_Actin

Nuclei

2/3

pass_exact

pass_exact

mapped

MeasureObjectIntensity

Intensity_StdIntensity_DNA

Cells

2/3

pass_exact

pass_exact

mapped

MeasureObjectIntensity

Intensity_StdIntensity_DNA

Nuclei

2/3

pass_exact

pass_exact

mapped

MeasureObjectIntensity

Intensity_UpperQuartileIntensity_Actin

Cells

2/3

pass_exact

pass_exact

mapped

MeasureObjectIntensity

Intensity_UpperQuartileIntensity_Actin

Nuclei

2/3

pass_exact

pass_exact

mapped

MeasureObjectIntensity

Intensity_UpperQuartileIntensity_DNA

Cells

2/3

pass_exact

pass_exact

mapped

MeasureObjectIntensity

Intensity_UpperQuartileIntensity_DNA

Nuclei

2/3

pass_exact

pass_exact

mapped

MeasureObjectIntensity

Location_CenterMassIntensity_X_Actin

Cells

2/3

pass_exact

pass_exact

mapped

MeasureObjectIntensity

Location_CenterMassIntensity_X_Actin

Nuclei

2/3

pass_exact

pass_exact

mapped

MeasureObjectIntensity

Location_CenterMassIntensity_X_DNA

Cells

2/3

pass_exact

pass_exact

mapped

MeasureObjectIntensity

Location_CenterMassIntensity_X_DNA

Nuclei

2/3

pass_exact

pass_exact

mapped

MeasureObjectIntensity

Location_CenterMassIntensity_Y_Actin

Cells

2/3

pass_exact

pass_exact

mapped

MeasureObjectIntensity

Location_CenterMassIntensity_Y_Actin

Nuclei

2/3

pass_exact

pass_exact

mapped

MeasureObjectIntensity

Location_CenterMassIntensity_Y_DNA

Cells

2/3

pass_exact

pass_exact

mapped

MeasureObjectIntensity

Location_CenterMassIntensity_Y_DNA

Nuclei

2/3

pass_exact

pass_exact

mapped

MeasureObjectIntensity

Location_CenterMassIntensity_Z_Actin

Cells

2/3

pass_exact

pass_exact

mapped

MeasureObjectIntensity

Location_CenterMassIntensity_Z_Actin

Nuclei

2/3

pass_exact

pass_exact

mapped

MeasureObjectIntensity

Location_CenterMassIntensity_Z_DNA

Cells

2/3

pass_exact

pass_exact

mapped

MeasureObjectIntensity

Location_CenterMassIntensity_Z_DNA

Nuclei

2/3

pass_exact

pass_exact

mapped

MeasureObjectIntensity

Location_MaxIntensity_X_Actin

Cells

2/3

pass_exact

pass_exact

mapped

MeasureObjectIntensity

Location_MaxIntensity_X_Actin

Nuclei

2/3

pass_exact

pass_exact

mapped

MeasureObjectIntensity

Location_MaxIntensity_X_DNA

Cells

2/3

pass_exact

pass_exact

mapped

MeasureObjectIntensity

Location_MaxIntensity_X_DNA

Nuclei

2/3

pass_exact

pass_exact

mapped

MeasureObjectIntensity

Location_MaxIntensity_Y_Actin

Cells

2/3

pass_exact

pass_exact

mapped

MeasureObjectIntensity

Location_MaxIntensity_Y_Actin

Nuclei

2/3

pass_exact

pass_exact

mapped

MeasureObjectIntensity

Location_MaxIntensity_Y_DNA

Cells

2/3

pass_exact

pass_exact

mapped

MeasureObjectIntensity

Location_MaxIntensity_Y_DNA

Nuclei

2/3

pass_exact

pass_exact

mapped

MeasureObjectIntensity

Location_MaxIntensity_Z_Actin

Cells

2/3

pass_exact

pass_exact

mapped

MeasureObjectIntensity

Location_MaxIntensity_Z_Actin

Nuclei

2/3

pass_exact

pass_exact

mapped

MeasureObjectIntensity

Location_MaxIntensity_Z_DNA

Cells

2/3

pass_exact

pass_exact

mapped

MeasureObjectIntensity

Location_MaxIntensity_Z_DNA

Nuclei

2/3

pass_exact

pass_exact

mapped

MeasureObjectNeighbors

Neighbors_AngleBetweenNeighbors_Adjacent

Cells

2/3

pass_exact

pass_tolerance

mapped

MeasureObjectNeighbors

Neighbors_FirstClosestDistance_Adjacent

Cells

2/3

pass_exact

pass_exact

mapped

MeasureObjectNeighbors

Neighbors_FirstClosestObjectNumber_Adjacent

Cells

2/3

pass_exact

pass_exact

mapped

MeasureObjectNeighbors

Neighbors_NumberOfNeighbors_Adjacent

Cells

2/3

pass_exact

pass_exact

mapped

MeasureObjectNeighbors

Neighbors_PercentTouching_Adjacent

Cells

2/3

pass_exact

pass_exact

mapped

MeasureObjectNeighbors

Neighbors_SecondClosestDistance_Adjacent

Cells

2/3

pass_exact

pass_exact

mapped

MeasureObjectNeighbors

Neighbors_SecondClosestObjectNumber_Adjacent

Cells

2/3

pass_exact

pass_exact

mapped

M3 native Rust core#

Result#

The Rust library (crates/cp_anywhere) implements the complete declared scope from spec/compatibility.md: MeasureImageIntensity, MeasureObjectIntensity over Cells/Nuclei × DNA/Actin, and MeasureObjectNeighbors same-set Adjacent with directed relationship rows. crates/cp_anywhere_python is a thin PyO3/NumPy binding with no scientific logic of its own; src/cp_anywhere/api.py validates inputs and exposes cp_anywhere.measure(...).

scripts/parity/compare_native.py compares the Rust result against all 13 saved golden fixtures and all 119 declared columns plus the directed relationship table: 1,560 of 1,560 rows pass (1,535 exact, 25 within the spec/features.yaml float64 tolerance). The 25 tolerance rows are all MeasureImageIntensity scalar reductions with errors between 1e-17 and 1e-14 — float64 summation-order variance, not a scope or semantics bug. Full counts and reasons are recorded in spec/native-coverage.yaml and benchmarks/reports/native-parity.json.

cargo test --manifest-path crates/cp_anywhere/Cargo.toml
env -u CONDA_PREFIX uv run maturin develop --release --uv
uv run --frozen python scripts/parity/compare_native.py
CP_ANYWHERE_REFERENCE_PYTHON=/path/to/reference/venv/bin/python \
  PYTHONPATH=/path/to/pinned/CellProfiler \
  uv run --frozen pytest

env -u CONDA_PREFIX is required on hosts where both CONDA_PREFIX and VIRTUAL_ENV are set; maturin refuses to build with both present.

Hard-case source decisions#

These follow spec/reference-semantics.md, which records the exact original behavior read from the pinned CellProfiler 4.2.8 and NumPy 1.23.5 sources, not a cleaned-up reimplementation:

  • Integer images are promoted to f64 without dtype-maximum rescaling, matching the reference runner’s Image(..., convert=False).

  • Object-quantile and MAD formulas use n * fraction (not (n-1) * fraction) and n / dimensions, matching the original module exactly.

  • Maximum-intensity coordinate ties are resolved by a literal Rust port of NumPy 1.23.5’s introsort (numpy_argsort/argsort_range/ numpy_heapsort in lib.rs), derived from numpy/core/src/npysort/quicksort.cpp and npysort_heapsort.h at de82cd9468704a033702974010ee7e7efc85b393 (BSD-3-Clause; see licenses/numpy.txt and THIRD_PARTY_NOTICES.md). A stable per-object sort does not reproduce the golden tie-break positions; this port does.

  • Same-set adjacent neighbors use only the first two center axes even in 3D (z/y, omitting x) for nearest-distance and angle, a separate ordering for distance/angle selection versus the ID ordering, and the original centrosome.outline.outline-equivalent first-two-axes diagonal outline (also applied unchanged in 3D). These are preserved reference quirks, not defects introduced by this port.

  • Sparse object IDs keep their array position through the maximum label; missing slots are not compacted.

Resource and memory contract#

  • Inputs: 2D or 3D only, nonzero shape, checked shape-product multiplication (checked_mul), matching channel/object/mask lengths, labels must be unsigned and <= MAX_LABEL (1,000,000; a documented limit, not an attempt to allocate u32::MAX rows).

  • crates/cp_anywhere_python/src/lib.rs borrows C-contiguous f64 image arrays and u32 label arrays directly as slices — no copy for that common case. src/cp_anywhere/api.py performs exactly one conversion via np.ascontiguousarray for other numeric dtypes or non-contiguous input; it does not claim zero-copy for those cases.

  • The kernel runs with py.detach(...) (PyO3’s GIL release), so Rust work does not hold the GIL. Callers must not mutate input arrays while measure() is running, since the Rust side holds borrowed slices for the call’s duration.

  • Rejected inputs: 1D or higher-than-3D shapes, boolean/float label arrays, non-real-numeric image dtypes, mismatched shapes, labels above MAX_LABEL, and (when neighbors are selected) no Cells object set.

  • The library (cp_anywhere) has no Python, filesystem, or platform dependency: Input/ResultTable are plain Rust types over borrowed slices, which keeps a future WASM target possible without a rewrite.

API#

cp_anywhere.measure(images, objects, *, mask=None, image_id="Image", families=None) returns a frozen MeasurementResult with .features (mirroring the golden JSON structure: Image scope holds scalars, object scopes hold NumPy arrays aligned to .object_ids), .object_ids, and .relationships (directed edges with both image/object IDs). .rows() gives a long table; .to_csv()/.to_json() write it and the reference-shaped structure respectively, with nonfinite floats encoded explicitly ({"nonfinite": "NaN"}, "+Infinity", "-Infinity") rather than silently dropped or turned into null. families restricts computation to a subset of ("image_intensity", "object_intensity", "neighbors") so later benchmarking can time an equivalent subset without computing unselected families.

The CLI (cp-anywhere --image NAME=PATH --objects NAME=PATH [--mask PATH] --output PATH) loads saved NPY arrays and writes CSV or JSON depending on the output suffix.

Wheel build and installed-wheel smoke test#

maturin build --release produced cp_anywhere-0.1.0-cp311-abi3-macosx_11_0_arm64.whl. It was installed into a fresh, isolated uv venv with no project checkout on sys.path. The imported module resolved to that venv’s site-packages (not the source tree), and cp_anywhere.measure(...) on the touching_gradient fixture returned all 119 declared columns. This proves the installed wheel, not just the editable development install, produces the full declared scope.

cargo fmt --check and cargo clippy --all-targets are clean for both crates. cargo test passes the one Rust unit test (invalid shape and over-limit label rejection). The full project pytest suite (22 tests, including the real CellProfiler 4.2.8 replay) passes.

Known limitations#

  • Only the one rejects_bad_shape_and_large_id Rust unit test exists; most correctness assurance currently comes from the Python-side golden parity script, not from Rust-level property tests (e.g. proptest) on the sorting or neighbor kernels.

  • Timing against cp_measure and against a WASM build is explicitly out of scope for M3; see docs/decisions/0002-rust-core.md and M4/M6.

M4 WASM feasibility and browser integration spike#

Result#

The Rust core compiles to wasm32-unknown-unknown with its default features, including the .cppipe executor. The core does not depend on Python or browser bindings. crates/cp_anywhere_wasm is a thin wasm-bindgen binding (JSON in/out for small metadata, typed arrays for the numeric payload). The build script scripts/wasm/build_package.py uses wasm-pack and names the npm package cp-anywhere. The Vite example (examples/browser-vite) that runs the module inside a dedicated web worker.

Both measurement and segmentation run entirely client-side with verified parity against the real CellProfiler 4.2.8 oracle. A real, reproducible headless-Chromium run (Playwright, not a mock DOM; examples/browser-vite/run-browser-check.mjs) executes the worker against:

  • All 13 declared measurement golden fixtures (2D and 3D) via measure(), compared by scripts/parity/compare_wasm.py – 1,560 of 1,560 rows pass (1,533 exact, 27 within the declared float64 tolerance; zero failures). 13 fixtures × 120 rows (119 features + 1 relationship table) – the identical row count the native build passes (docs/native.md), closing M5’s “100% … in native Rust, Python, and WASM” gate for the current declared manifest.

  • All 100 segmentation golden fixtures (fixtures/segmentation/) via identifyPrimaryObjects(), compared by scripts/parity/compare_wasm_segmentation.py – 100 of 100 pass exactly on labels and object count (one fixture’s scalar threshold uses the same documented, narrow, label-safe float32/float64 tolerance exception as the native/Python build; see tests/segmentation/test_reference_alignment.py’s _KNOWN_THRESHOLD_MAP_TOLERANCE_GAP) – see docs/segmentation-coverage.md.

The 27 measurement tolerance rows are the same MeasureImageIntensity reduction-order families already seen in the native build (docs/native.md), plus two Neighbors_AngleBetweenNeighbors_Adjacent rows at 1e-14-scale absolute / 1e-16-scale relative error – consistent with a transcendental-function (acos/atan2) last-bit difference between wasm32’s and native arm64’s libm, not a computation bug.

cargo fmt --manifest-path crates/cp_anywhere_wasm/Cargo.toml --check
cargo clippy --manifest-path crates/cp_anywhere_wasm/Cargo.toml --target wasm32-unknown-unknown --all-targets
python scripts/wasm/build_package.py

cd examples/browser-vite
npm install
uv run --frozen python ../../scripts/wasm/export_fixtures.py               # measurement fixtures (gitignored)
uv run --frozen python ../../scripts/wasm/export_segmentation_fixtures.py  # segmentation fixtures (gitignored)
npm run build
npx playwright install chromium   # once
npm run browser-check             # writes browser-results.json
cd ../..
uv run --frozen python scripts/parity/compare_wasm.py
uv run --frozen python scripts/parity/compare_wasm_segmentation.py

Browser demo and npm package#

The documentation also includes Cell profiling in your browser. It runs illumination correction, segmentation, and measurements on the same crop as the Python notebook. To build and serve that page locally from the repository root:

python scripts/wasm/build_package.py
uv run --frozen --group docs sphinx-build -W docs/src docs/_build/html
uv run --frozen python scripts/wasm/stage_docs_demo.py docs/_build/html
python -m http.server 8000 --bind 127.0.0.1 --directory docs/_build/html

Open http://127.0.0.1:8000/browser-demo.html. The published docs build includes these assets. It does not fetch the runtime from npm or a CDN.

examples/browser-vite is a real Vite project (not a static mockup): its src/worker.ts imports the generated cp-anywhere package, fetches each fixture’s raw binary arrays (see below), calls measure() and identifyPrimaryObjects(), and posts the results back; src/main.ts drives the worker from the page and renders status in #status. npm run build produces a genuine production bundle:

dist/index.html                             0.46 kB
dist/assets/worker-*.js                     ~9.0 kB
dist/assets/cp_anywhere_wasm_bg-*.wasm    377.62 kB  (146.80 kB gzip-equivalent)
dist/assets/index-*.js                      ~2.5 kB

npm run preview (or run-browser-check.mjs’s own tiny static server) serves this real build; nothing is dev-server-only or dynamically recompiled per request.

Segmentation: identifyPrimaryObjects#

identifyPrimaryObjects(pixels, rows, cols, mask, config_json) shares its exact settings surface with the Python binding (cp_anywhere::segmentation::build_config/ConfigParams – see spec/segmentation-modules.md), so the same config_json field names (threshold_method, unclump_method, watershed_method, …) work identically in both bindings, eliminating a second, driftable mapping. Its result (SegmentationResult) is exposed as typed-array/scalar getters (labels, smallRemovedLabels, objectCount, threshold, thresholdMap, guideThreshold), not a JSON string: the label arrays are the same size as the input image, unlike measure()’s much smaller per-object feature output, so JSON-encoding them would be wasteful. identifyPrimaryObjectsJson(input) is a JSON-string overload (pixels as a plain array) kept for parity with measureJson, for debugging/small inputs without constructing a typed array by hand.

examples/zarr-onnx-duckdb demonstrates this chained into a full Zarr -> segmentation -> measurement -> Parquet pipeline in a real browser (see docs/browser-integration.md) – this is the shape a browser-based consumer needing both segmentation and feature extraction from cp-anywhere would use, not a theoretical API surface.

Typed-array inputs, output conversion, memory#

  • Input: measure(shape, image_names, images, object_names, objects, mask, image_id, families) takes real typed arrays (Uint32Array/Float64Array/Uint8Array) across the wasm-bindgen boundary — the plan’s “typed-array inputs” requirement. wasm-bindgen copies &[T] parameters into wasm linear memory once per call; this is the standard, documented cost of that FFI pattern (see crates/cp_anywhere_wasm/src/lib.rs), not something hidden.

  • Numeric fixture data is fetched as raw binary, not JSON. The first export attempt used plain JSON and failed immediately: nonfinite’s pixel array contains literal NaN/Infinity, which are not valid JSON numbers (JSON.parse correctly rejects them). scripts/wasm/export_fixtures.py now writes small JSON metadata (shape, names, image_id) plus .images.f64 / .objects.u32 / .mask.u8 raw binary blobs; the worker fetches each as an ArrayBuffer and constructs a typed-array view directly over the bytes, which preserves the exact IEEE754 bit pattern and is also the more realistic typed-array transfer path for a browser demo.

  • Output conversion: measure() returns one JSON string matching MeasurementResult.to_json()’s shape exactly (same {"nonfinite": "NaN"}-style markers), so the same Python comparator (scripts/parity/compare_cp_measure.py’s compare_feature/ compare_relationships) verifies both the native and the WASM result without a second, independently-trusted comparison implementation.

  • Memory: WebAssembly.Memory.buffer.byteLength was 3,538,944 bytes (~3.46 MiB) after running all 13 fixtures sequentially in one worker instance — a post-hoc read, not a tracked peak (see Known limitations).

Worker execution and browser compatibility#

The module runs inside a dedicated Worker({ type: "module" }), loaded via Vite’s new URL(...) + new Worker(...) pattern, which Vite bundles correctly (confirmed by the production build above). Testing used real headless Chromium (Playwright, Chrome for Testing 159.0.8154.6); Firefox and Safari were not tested in this spike (see Known limitations). crossOriginIsolated / COOP-COEP were not required for this single-threaded wasm-bindgen build.

Native vs WASM comparison#

Both the native build (docs/native.md) and this WASM build pass the same golden fixtures exactly (modulo the same class of float64 reduction/transcendental-order tolerance). Measured per-call timing (benchmarks/reports/wasm-baseline.json, 3 warmups / 10 runs, same four representative workloads from spec/benchmark-contract.yaml):

Fixture

WASM median ms (worker round trip)

touching_gradient

0.15

real_cell_cc0

2.85

grid_medium

11.80

grid_large

174.00

Each sample times one measure() call, which includes the in-WASM serde_json::to_string() encoding of the full 119-column result — a worker-round-trip cost, not an isolated kernel time. It is not yet comparable to the native/cp_measure Python baseline (benchmarks/baseline/initial-cp-measure-timing.json) on an apples-to-apples kernel-only basis; that isolation, plus a real cross-target regression gate, is M6’s job. spec/benchmark-contract.yaml’s browser_worker_budget, browser_memory_budget, and browser_package_size_budget are updated from “provisional until M4” to real first-measured numbers with a documented margin.

Known limitations#

  • Only Chromium was tested (via Playwright). Firefox and Safari are untested; no WASM feature used here (no SIMD, no threads, no SharedArrayBuffer) is Chromium-specific, so cross-browser support is expected but not verified in this spike.

  • Per-call timing includes JSON-encoding every result; kernel-only timing (e.g., via a numeric-only output mode) is left to M6.

  • wasm_linear_memory_bytes_after_run is a single post-run read, not a tracked peak across the run.

  • This spike did not test a crossOriginIsolated/threaded build; the current build is intentionally single-threaded, matching the plan’s “test… browser compatibility” bar without requiring COOP/COEP deployment headers.

Browser integration: Zarr, ONNX Runtime Web, DuckDB-WASM#

What this is#

examples/zarr-onnx-duckdb/ is a real, end-to-end browser integration proving the pieces named in the plan work together with cp-anywhere’s own WASM build, with no mocked stage:

  1. zarrita.js reads a real OME-Zarr v2 array (public/dna.zarr, written by scripts/zarr/export_demo_fixture.py) over HTTP.

  2. onnxruntime-web runs a real, checked ONNX graph (public/threshold.onnx) against that array, producing a reference threshold mask (logged for comparison; not used downstream – see “Why the mask is only a reference” below).

  3. cp-anywhere’s own identifyPrimaryObjects (the same CellProfiler-compatible, oracle-verified segmentation the Python binding exposes – Otsu threshold + Intensity declumping/watershed in this demo) segments the same Zarr pixels into real object labels.

  4. cp-anywhere’s measure() (this project’s own Rust/WASM kernel, the same build verified in docs/wasm.md) measures the Zarr image using cp-anywhere’s own segmentation output as the Cells object labels – both segmentation and feature extraction now come from cp-anywhere, not just the latter.

  5. DuckDB-WASM loads cp-anywhere’s own measurement output into a SQL table and exports it as a real Parquet file (COPY ... FORMAT PARQUET).

scripts/zarr/validate_export.py then validates that Parquet file with native DuckDB (not the WASM build under test) and cross-checks every value – and the segmentation’s own object_count/threshold – against a fresh, from-source native Python cp_anywhere.identify_primary_objects()

  • cp_anywhere.measure() call on the identical Zarr bytes – proving the browser pipeline (both its segmentation and its measurement) produced numerically correct output, not just that it ran without an error.

Why the mask is only a reference#

The ONNX threshold model (public/threshold.onnx, a checked Greater(image, 0.5) -> Cast(..., INT32) graph) still runs in this demo to exercise the real Zarr -> ONNX -> cp-anywhere -> DuckDB pipeline named in the plan, and its foreground pixel count is logged for comparison. Its output is not fed into measurement: main.ts segments with cp-anywhere’s own identifyPrimaryObjects instead (step 3 above), so the object labels measurement actually uses come from real CellProfiler- compatible segmentation (with declumping), not a disconnected-foreground threshold mask that would merge every touching object into a single label. A future revision could use the ONNX path for a learned model (e.g. StarDist) instead of the fixed-threshold demo graph; that remains independent of cp-anywhere’s own segmentation path either way.

Run it#

uv sync --group demo-assets
uv run --frozen python scripts/zarr/export_demo_fixture.py

cd examples/zarr-onnx-duckdb
npm install
npm run build
npm run browser-check   # real Playwright/Chromium run; writes
                         # cp_anywhere_features.parquet and browser-results.json
cd ../..

uv run --group demo-assets python scripts/zarr/validate_export.py

Verified results#

A real run (Playwright 1.65.1, Chrome for Testing, npm run browser-check) on the touching_gradient fixture’s DNA channel (12x12, normalized to [0, 1]):

  • Zarr read: shape [12, 12], 144 pixels.

  • ONNX inference (reference only): 72 foreground pixels (mask > 0.5).

  • cp-anywhere identifyPrimaryObjects() (Otsu + Intensity declump/watershed, min_diameter=4.0): 1 object, threshold 0.486328125.

  • cp-anywhere measure(): 21 object_intensity columns for that object.

  • DuckDB-WASM Parquet export: real PAR1 magic bytes at both ends of the file, independently readable by native DuckDB.

  • scripts/zarr/validate_export.py: all 21 feature values, plus the segmentation’s own threshold, match a from-source native Python cp_anywhere.identify_primary_objects() + cp_anywhere.measure() run exactly (rtol=1e-6, atol=1e-9) – zero mismatches.

Known limitations#

  • Network required at run time. DuckDB-WASM fetches its parquet extension from extensions.duckdb.org on the first LOAD parquet; this demo does not vendor that extension. For a fully offline/CSP-restricted deployment, fetch and serve https://extensions.duckdb.org/v1.5.4/wasm_mvp/parquet.duckdb_extension.wasm locally and set custom_extension_repository before LOAD parquet (verify the embedded engine version first – it does not always match the npm package version; see the exact version-pinning caveats this was derived from in ~/.hermes/cache/scratch/cp-anywhere-browser-integration-research.md if still present, or re-derive from DuckDB-WASM’s own docs).

  • Single browser tested. Only Chromium (via Playwright) was exercised. No SharedArrayBuffer/COOP/COEP cross-origin isolation was used or is required for this single-threaded WASM MVP path.

  • 2D only. The Zarr stage of this demo uses one 2D channel; this does not affect cp-anywhere’s own 2D/3D coverage (docs/native.md, docs/wasm.md), which is unrelated and already complete.

  • Not an OME-Zarr multiscale/metadata demo. The written Zarr array has an empty .zattrs (no OME-NGFF multiscale metadata, no multichannel axes); it is a plain single-chunk array read for its pixel data only.

  • Segmentation scope is this demo’s own fixed choice, not every setting. main.ts hard-codes threshold_method="otsu", unclump_method="intensity", watershed_method="intensity", min_diameter=4.0; identifyPrimaryObjects’s full settings surface (every threshold method, adaptive scope, Shape/Propagate declump, hole filling, border/size filtering) is documented and independently oracle-verified in docs/segmentation-coverage.md and spec/segmentation-modules.md, not re-demonstrated here.

M6 performance report#

Headline result#

Fixed a real O(pixels^2) bug in the neighbors kernel, found by profiling a per-workload throughput regression the plan explicitly warns against hiding behind an aggregate: before the fix, cp-anywhere was 3.6x slower than the pinned cp_measure baseline on the grid_large workload (256 objects, 192x192 px), even though every smaller workload was already much faster. After the fix, all four representative workloads beat the baseline individually, and WASM round-trip times fell by 11x-33x at the same time.

How the bug was found#

crates/cp_anywhere/src/lib.rs’s “touching” check (used for PercentTouching in MeasureObjectNeighbors) needs to know, for every outline pixel of object k, whether any pixel within a small fixed radius (1.5 px) belongs to a different object. The original implementation tested that by scanning every pixel in the whole image for each outline pixel – an accidental O(perimeter_pixels * total_pixels) algorithm when O(perimeter_pixels) suffices, because the actual predicate only ever depends on a handful of neighboring offsets.

This was invisible in the declared fixture set’s smallest three workloads (it is a constant-factor cost there) but dominates at grid_large scale (36,864 px, 256 objects): a cargo bench run that split the combined object_intensity_and_neighbors timing by family (crates/cp_anywhere/benches/throughput.rs’s by_family group) showed neighbors alone was 151.45 ms on grid_large vs 9.68 ms on grid_medium – roughly a 15.6x time increase for a 4x object-count / 4x pixel-count increase, consistent with an O(pixels * objects) or O(pixels^2) pattern, not the near-linear scaling object_intensity showed over the same step (0.44 ms -> 2.18 ms, ~5x).

The fix#

Replaced the whole-image scan with a bounded local-neighborhood check using the same shifted(index, shape, delta) helper the adjacent-relationship loop just below it already used correctly: a small, fixed Cartesian product of candidate offsets (9 for 2D, 8 for 3D) covering exactly the same set of relative positions the original predicate tested, reusing the project’s existing bounds-checked index arithmetic instead of adding a new primitive. The predicate itself was not touched – including its asymmetric, reference-preserving 3D offset range (d == 0 || d == -1, not a symmetric radius) – only how it is evaluated changed, from O(total pixels) to O(1) per outline pixel.

Correctness was re-verified immediately after the change, not assumed:

cargo test --manifest-path crates/cp_anywhere/Cargo.toml   # pass
uv run --frozen python scripts/parity/compare_native.py          # 1440/1440 pass (unchanged)
# rebuild WASM, re-export fixtures, rebuild demo, re-run in Chromium:
uv run --frozen python scripts/parity/compare_wasm.py            # 1440/1440 pass (unchanged)

Both parity counts are bit-for-bit identical to before the fix – the change is a pure performance fix, not a semantics change.

Before / after timing#

Native Rust kernel only (cp_anywhere::measure, no Python/FFI, object_intensity + neighbors families, Criterion, median of 100 samples):

Workload

Before

After

Speedup

touching_gradient

38.05 us

35.48 us

1.07x

real_cell_cc0

2.53 ms

0.195 ms

13.0x

grid_medium

10.16 ms

0.695 ms

14.6x

grid_large

154.85 ms

7.354 ms

21.1x

Browser (WASM, Playwright/Chromium, median per measure() call including in-WASM JSON encoding, 10 timed runs):

Workload

Before

After

Speedup

touching_gradient

0.15 ms

0.20 ms

~unchanged (near timer resolution)

real_cell_cc0

2.85 ms

0.70 ms

4.1x

grid_medium

11.80 ms

2.00 ms

5.9x

grid_large

174.00 ms

11.85 ms

14.7x

Throughput vs the pinned cp_measure baseline#

Per spec/benchmark-contract.yaml’s native_acceptance gate: equal-weight arithmetic mean of per-workload cp_measure_time / cp-anywhere_time ratios on the object_intensity + neighbors scope (the only scope cp_measure implements), minimum 1.0. Computed by scripts/benchmarks/check_throughput_gate.py, which prints every per-workload ratio and fails if any workload is below the minimum, so an aggregate pass cannot hide an individual regression:

Workload

cp_measure (ms)

cp-anywhere Rust (ms)

Ratio

touching_gradient

1.034

0.0355

29.1x

real_cell_cc0

3.380

0.1952

17.3x

grid_medium

10.916

0.6952

15.7x

grid_large

42.933

7.3535

5.8x

Equal-weight mean: 17.0x. Every workload individually clears the 1.0x minimum (all_workloads_pass_individually: true in benchmarks/reports/native-throughput-gate.json) – before the fix, grid_large’s ratio was 0.28x (a genuine regression the aggregate of ~7.5x would have silently hidden had it not been checked per-workload).

Python binding overhead#

Per plan 5.2 (“Python binding overhead: quantified and bounded”), comparing the installed release wheel’s cp_anywhere.measure() against the bare Rust kernel on the same inputs (benchmarks/benchmark_native.py vs scripts/benchmarks/report_native_rust.py):

Workload

Rust (ms)

Python (ms)

Overhead

touching_gradient

0.0355

0.0744

2.09x

real_cell_cc0

0.1952

0.2614

1.34x

grid_medium

0.6952

0.7626

1.10x

grid_large

7.3535

7.7758

1.06x

The fixed per-call cost (roughly 0.04-0.07 ms: Python call dispatch plus NumPy array validation) shrinks in relative terms as the workload grows, and stays within the declared <=1.5x budget (spec/benchmark-contract.yaml’s python_overhead_budget) for every workload at or above real_cell_cc0 scale. No unnecessary full-array copy was found in the binding for the already-C-contiguous fixture inputs.

CI regression gate#

scripts/benchmarks/check_regression.py compares a fresh benchmarks/reports/native-rust-baseline.json against the committed benchmarks/baseline/native-rust-committed.json, failing only when a workload is more than 20% slower than committed – never on an improvement, and never on noise within that margin, per the plan’s explicit warning against hard-failing on noisy single-run CI timing. .github/workflows/benchmark.yml runs the full benchmark -> report -> throughput-gate -> regression-check chain on a weekly schedule and on manual dispatch (not on every PR, for the same noise reason), uploading the raw report as a 90-day build artifact for trend-watching.

To intentionally accept a new baseline after a verified, deliberate change, regenerate benchmarks/reports/native-rust-baseline.json and copy it over benchmarks/baseline/native-rust-committed.json.

Reproduce#

cargo bench --manifest-path crates/cp_anywhere/Cargo.toml
uv run --frozen python scripts/benchmarks/report_native_rust.py
env -u CONDA_PREFIX uv run maturin develop --release --uv
uv run --frozen python benchmarks/benchmark_native.py
uv run --frozen python scripts/benchmarks/check_throughput_gate.py
uv run --frozen python scripts/benchmarks/check_regression.py

Known limitations#

  • Single machine, single run per report regeneration (Criterion’s own statistical sampling, 100 samples per workload, is the only within-run repetition; see benchmark-analysis’s guidance on single-rep measurements – the gaps here are large (5.8x-29x, 21x on the previously-regressed workload), well above the “order of magnitude” bar for a conclusion to survive run-to-run noise).

  • cp_measure’s own timing snapshot (benchmarks/baseline/initial-cp-measure-timing.json) was captured once, in M2, on this same machine; it was not re-measured in M6. Re-running it would only change the comparison if this machine’s absolute performance characteristics changed between M2 and M6, which they did not (same hardware, same session).

  • The regression-gate tolerance (20%) and the schedule (weekly, not per-PR) are both judgment calls aimed at the plan’s explicit warning against noisy single-run CI failures; they have not been tuned against real CI runner variance data, because this session cannot execute a real scheduled GitHub Actions run.

Release process#

What publishes where, and how#

Package

Registry

Workflow

Trigger

Auth

cp-anywhere (sdist + abi3 wheels, 5 platforms)

PyPI

.github/workflows/publish-pypi.yml

release: published

OIDC trusted publishing (pypa/gh-action-pypi-publish); works from the first release via a PyPI “pending publisher”

cp-anywhere (Rust crate)

crates.io

.github/workflows/publish-crate.yml

release: published

CARGO_REGISTRY_TOKEN secret for the first release (crates.io OIDC trusted publishing can only be configured for a crate that already exists); can migrate to OIDC afterward

cp-anywhere (npm package)

npm

.github/workflows/publish-npm.yml

release: published

OIDC trusted publishing (id-token: write, no stored token) – but only once the package already exists on npm; see the bootstrap step below

Standalone .wasm + checksum

GitHub Release assets

.github/workflows/publish-npm.yml (build_and_publish_npm job)

release: published

GITHUB_TOKEN

The GitHub release tag is the only version you set. All three workflows build the commit attached to that tag. tools/sync_release_version.py updates the Cargo workspace and lockfile in each build checkout and checks all three workspace packages before building. Do not bump versions in source files or commit the generated changes.

For v1.2.3, all three registries get 1.2.3. For prerelease tags such as v1.2.3-rc.1, crates.io and npm get 1.2.3-rc.1 and PyPI gets 1.2.3rc1. Use -alpha.N, -beta.N, or -rc.N for prereleases. npm publishes prereleases under next and stable releases under latest.

cp-anywhere is the only crate configured for crates.io publishing. It includes segmentation, illumination, measurements, and .cppipe execution. To use the individual operations without pipeline parsing or its serialization dependencies, set default-features = false in your Cargo dependency. cp_anywhere_python and cp_anywhere_wasm remain internal binding crates. The Python and browser packages ship through PyPI and npm.

One-time setup (must be done once, outside this repo, before the first release)#

Each registry has different setup steps for a first release of its package:

  1. GitHub “release” environment. Already created (repos/wayscience/cp-anywhere/environments/release), referenced by all three publish workflows’ environment: release. Optional hardening: add required reviewers under Settings -> Environments -> release.

  2. PyPI – zero-touch, even for the first release. PyPI supports a “pending publisher” that creates the project on first use:

    1. Sign in at https://pypi.org/manage/account/publishing/.

    2. Add a pending GitHub publisher: PyPI project name cp-anywhere, owner wayscience, repository cp-anywhere, workflow filename publish-pypi.yml, environment name release.

    3. Nothing else to do – the first tagged release creates the project and publishes to it automatically.

  3. npm – needs one manual publish first. npm’s trusted publishing has no “pending publisher” equivalent: a trusted publisher can only be attached to a package that already exists, so the very first publish cannot go through OIDC.

    1. Use a disposable checkout. Publish a bootstrap prerelease so the GitHub workflow can publish the first stable version. For a first stable release of v0.1.0, use:

      python tools/sync_release_version.py v0.1.0-rc.1
      python scripts/wasm/build_package.py
      npm publish ./crates/cp_anywhere_wasm/pkg --access public --tag next
      
    2. Once cp-anywhere exists on npm, go to its package settings -> Publishing access -> add a trusted publisher: GitHub Actions, owner wayscience, repository cp-anywhere, workflow filename publish-npm.yml, environment release.

    3. Enable direct npm publish for the trusted publisher. Publish the first stable version through GitHub Releases, not by hand.

  4. crates.io – needs a token for the first release. Trusted publishing on crates.io reached general availability in 2025, but like npm, the trusted publisher can only be configured for a crate that already exists – so the first publish needs a classic API token.

    1. Sign in at https://crates.io/settings/tokens, create a token scoped to publish-new (and publish-update for later releases).

    2. Add it as a CARGO_REGISTRY_TOKEN secret in the GitHub release environment (Settings -> Environments -> release -> Environment secrets). publish-crate.yml already reads this secret, so no workflow change is needed.

    3. Optional, after the crate exists: migrate publish-crate.yml from the stored token to OIDC (rust-lang/crates-io-auth-action@v1, the modern equivalent of this project’s PyPI/npm setup) and delete the CARGO_REGISTRY_TOKEN secret. Not required – the token-based workflow keeps working for every future release too.

Cutting a release#

  1. Open a PR that updates CHANGELOG-equivalent notes if desired (this project uses release-drafter – .github/workflows/draft-release.yml drafts release notes automatically from merged PR labels on every push to main; review and edit the draft before publishing).

  2. Publish the drafted GitHub Release with a vX.Y.Z tag. This single release: published event fans out to all three publish workflows above, plus the .wasm/checksum release-asset upload. The tag is the only place a version is chosen; nothing else needs editing first.

  3. Check the three publishing workflows in the Actions tab. After registry setup, no per-release version edits or manual registry uploads are needed.

Local packaging checks#

Build and check the artifacts locally before cutting a release. These commands do not upload anything:

# crates.io: build and verify the archive
cargo package --locked -p cp-anywhere

# PyPI: sdist + wheel build, then install from the wheel into a throwaway
# venv with no project source on sys.path
uv build
python3 -m venv /tmp/smoke && /tmp/smoke/bin/pip install dist/*.whl numpy
/tmp/smoke/bin/python -c "import cp_anywhere; cp_anywhere.measure(...)"

# npm: wasm-pack build, then a full publish dry run
python scripts/wasm/build_package.py
cd crates/cp_anywhere_wasm/pkg && npm publish --dry-run --access public

Use --allow-dirty with cargo package to check uncommitted changes locally. The release workflow uses it because version sync modifies the checkout before building.

Known gaps#

  • The publish workflows themselves have not been exercised by a real GitHub Actions run (that would require actually cutting a release, which this session does not do). Local packaging checks do not verify registry authentication or publishing on GitHub’s runners.

  • The one-time setup steps above (PyPI pending publisher, npm bootstrap publish + trusted publisher, crates.io token) require the registry account owner’s own login and have not been done yet – they are not something a CI workflow or an automated tool can perform.

M5 implementation coverage dashboard#

Declared feature inventory#

The declared manifest is spec/features.yaml: 119 columns across MeasureImageIntensity, MeasureObjectIntensity, and same-set adjacent MeasureObjectNeighbors (full settings and exclusions in spec/compatibility.md). This is the project’s first release scope, not every CellProfiler module; expanding it requires its own fixtures and scope decision, per spec/compatibility.md’s scope decision gate.

Cross-target parity status#

Target

Fixtures

Rows checked

Pass (exact + tolerance)

Fail

Report

Native Rust

13

1,560 (119 + 1 rel.)

1,535 + 25 = 1,560

0

benchmarks/reports/native-parity.json, docs/native.md

Python (PyO3)

13

1,560

1,535 + 25 = 1,560

0

Same binding as native Rust (src/cp_anywhere/api.py calls the identical _native.measure); verified by the installed-wheel smoke test in docs/native.md

WASM (browser)

13

1,560

1,533 + 27 = 1,560

0

benchmarks/reports/wasm-parity.json, docs/wasm.md

100% of the 119 declared features pass CellProfiler 4.2.8 parity in native Rust, Python, and WASM, across all 13 golden fixtures (2D and 3D, including two independent real-microscopy sources). No fixture, feature, or target has a silent fallback or omission; any feature that could not be verified would show as fail_*, not pass_*, in the reports above. The “tolerance” pass count differs slightly by target (25 native, 27 WASM) because the WASM run includes two Neighbors_AngleBetweenNeighbors_Adjacent rows at a transcendental-function libm last-bit difference (documented in docs/wasm.md); every such row is still within spec/features.yaml’s declared atol/rtol, not a relaxed or hidden threshold.

M5 task checklist against the plan#

  • “Implement remaining features in the declared manifest.” None remained: M3 implemented and passed all 119 declared columns before M4 started (docs/native.md).

  • “Support documented multichannel, 2D/3D, calibration, and object relationship semantics.” Multichannel: DNA/Actin two-channel intensity is in the declared scope and passes on all targets. 2D/3D: volume_3d passes on native and (as of this update) WASM. Calibration: spec/compatibility.md explicitly declares “calibration-driven value conversion” out of scope for this release — there is no undocumented gap here, it is a declared exclusion. Object relationships: the directed Neighbors relationship table is implemented and checked on every fixture and every target.

  • “Add regression fixtures for each difference found in CellProfiler/cp_measure comparisons.” M2’s comparison (benchmarks/baseline/cp_measure-parity.json) found that cp_measure (not CellProfiler) mismatches CellProfiler on sparse-ID positions, the NumberOfNeighbors dtype, and nonfinite propagation. The golden fixtures that exposed those differences — sparse_ids, nonfinite — are exactly the fixtures this project’s own Rust/WASM implementation is checked against, and it passes all of them. No additional fixture was needed because the existing M1 fixture set already covered these cases; this was verified, not assumed.

  • “Update per-feature support and numerical parity reports.” Done: spec/features.yaml (oracle inventory), spec/cp_measure-map.yaml (cp_measure per-feature status), spec/native-coverage.yaml (native build status), and this dashboard (cross-target status) are all current.

Reproduce#

cargo test --manifest-path crates/cp_anywhere/Cargo.toml
env -u CONDA_PREFIX uv run maturin develop --release --uv
uv run --frozen python scripts/parity/compare_native.py

cd examples/browser-vite && npm install && npm run build
uv run --frozen python ../../scripts/wasm/export_fixtures.py
npm run browser-check
cd ../..
uv run --frozen python scripts/parity/compare_wasm.py

CP_ANYWHERE_REFERENCE_PYTHON=/path/to/reference/venv/bin/python \
  PYTHONPATH=/path/to/pinned/CellProfiler \
  uv run --frozen pytest

Segmentation coverage dashboard#

Declared scope#

cp_anywhere.identify_primary_objects targets CellProfiler 4.2.8’s IdentifyPrimaryObjects module (and the Threshold module it shares with every other segmentation module) as its compatibility oracle. Full declared scope, including every explicit gap, is in spec/segmentation-modules.md; this page is the distilled status dashboard.

Cross-target verification status#

Both native Rust/Python and the WASM (browser) build pass all 100 fixtures exactly (see docs/wasm.md for the WASM-specific run and its one documented tolerance exception, identical to the one below).

Tier

Status

Global threshold: Manual, Otsu (two- and three-class), Li, Robust Background

Covered

Adaptive (local) threshold: block-grid + B-spline interpolation, Sauvola

Covered

threshold_smoothing_scale, threshold_correction_factor, threshold_range

Covered

log_transform

Covered

Border exclusion, size/diameter filtering, object-count limiting

Covered

Hole filling (Never/After declumping only/After both), including nested-object absorption

Covered

Declumping (unclump_method: None/Intensity/Shape)

Covered

Watershed separation (watershed_method: None/Intensity/Shape/Propagate)

Covered

Manual smoothing-filter-size / maxima-suppression-size overrides

Covered

unclump_method = "Laplacian of Gaussian"

Not implemented (explicit gap)

low_res_maxima downsampled maxima detection

Not implemented (explicit gap)

threshold.global_operation = "Measurement"

Not implemented (explicit gap)

3D/volumetric segmentation, IdentifySecondaryObjects/IdentifyTertiaryObjects

Out of scope for this release

Every “Covered” row above is independently verified against the real, pinned CellProfiler 4.2.8 oracle — not assumed correct by code inspection. tests/segmentation/test_reference_alignment.py replays scripts/segmentation/generate_reference.py’s real-CellProfiler output against cp_anywhere.identify_primary_objects on 100 fixture cases (fixtures/segmentation/), spanning synthetic edge cases and two independent real microscopy datasets (BBBC007, real_cell_cc0). No fixture has a silent fallback or relaxed tolerance beyond the one documented, narrow, label-safe exception below.

One known tolerance exception: a single adversarial fixture (adaptive scope, three-class Otsu, log-transformed, a deliberately tiny 4-pixel window) can land threshold_map values up to ~0.012 off the oracle, from a float32-vs-float64 pixel-precision difference that a full pipeline rewrite was tried and reverted for (see spec/segmentation-modules.md’s “Adaptive (local) thresholding” section). Object labels are unaffected in every case.

Chaining into measurement#

seg = cp_anywhere.identify_primary_objects(dna_image, threshold="otsu")
result = cp_anywhere.measure({"DNA": dna_image}, {"Cells": seg.labels})

Reproduce#

env -u CONDA_PREFIX uv run maturin develop --release --uv
CP_ANYWHERE_REFERENCE_PYTHON=/path/to/reference/venv/bin/python \
  PYTHONPATH=/path/to/pinned/CellProfiler \
  uv run --frozen pytest tests/segmentation

# regenerate fixtures from the real oracle (after a reference-semantics change)
CP_ANYWHERE_REFERENCE_PYTHON=/path/to/reference/venv/bin/python \
  PYTHONPATH=/path/to/pinned/CellProfiler \
  /path/to/reference/venv/bin/python scripts/segmentation/generate_reference.py

Size and shape measurements#

MeasureObjectSizeShape is available for 2D label arrays in native Python and WASM. It uses the pinned CellProfiler 4.2.8 reference, with scikit-image 0.21.0 and centrosome 1.2.3.

from cp_anywhere import measure

result = measure(
    {"DNA": dna_image},
    {"Cells": cell_labels},
    families=["object_size_shape"],
    size_shape_calculate_zernikes=True,
    size_shape_calculate_advanced=False,
)

The family measures each supplied object set, without using image intensity or the image mask. Images are still required by the shared measure API. The original three default families are unchanged: request object_size_shape explicitly to add shape columns.

Columns and settings#

All columns have the AreaShape_ prefix. The module supplies:

  • 25 standard columns: area, perimeter, axes, eccentricity, orientation, center coordinates, bounding boxes, form factor, extent, solidity, compactness, Euler number, radii, convex area, Feret diameters, and equivalent diameter.

  • 30 Zernike columns, orders 0 through 9. Enabled by default, matching CellProfiler’s calculate_zernikes setting.

  • 53 advanced columns: spatial, central and normalized moments, Hu moments, the inertia tensor and its eigenvalues. Disabled by default, matching CellProfiler’s calculate_advanced setting.

The two flags select 25, 55, 78, or 108 total columns. Undefined values remain NaN or infinity, rather than becoming zero. The normal result serialization and export methods retain their existing nonfinite-value rules. An empty object set still has the selected schema with empty arrays.

The .cppipe importer accepts revision 3. Its three settings are the object list, Zernike flag, and advanced flag. Each imported module keeps its own settings and object selection, even when a pipeline measures Nuclei, Cells, and Cytoplasm with different flags.

Limits#

  • 3D size/shape measurements are not implemented. A 3D request raises a clear error rather than returning 2D substitutes.

  • The public measurement API requires consecutive object IDs starting at

    1. The real reference produces inconsistently packed columns for sparse labels. The low-level Rust routines preserve that behavior for numerical comparison, but the integrated API rejects it to prevent corrupt row alignment in exports. Other measurement families keep their existing sparse-label support.

  • Numerical checks target the pinned reference environment. Advanced moment reductions reproduce its NumPy/OpenBLAS ARMv8 order; this is not a claim of bitwise agreement with every BLAS implementation.

Verification#

Real reference fixtures cover rectangles, rings, diagonal shapes, nested objects, multiple objects, border objects, single pixels, and empty images. Tests compare every selected column under all four flag combinations. Separate real pipelines run primary, secondary, and tertiary segmentation before size/shape measurement, with different flags for each compartment.

The browser check runs both measureJson and measureWithOptions in real Playwright Chromium, not a simulated WASM runtime. The original eight-argument measure export remains available for existing callers; measureWithOptions adds diagnostic neighbor layers and shape flags.

To regenerate and check the reference data:

PYTHONPATH=/path/to/pinned/CellProfiler /path/to/reference/python \
  scripts/reference/generate_size_shape.py
PYTHONPATH=/path/to/pinned/CellProfiler /path/to/reference/python \
  scripts/cppipe/generate_size_shape_pipeline.py
uv run --frozen pytest tests/test_size_shape.py tests/cppipe/test_size_shape_pipeline.py
python scripts/wasm/build_package.py
uv run --frozen python scripts/parity/compare_wasm_size_shape.py

The browser check requires the browser example’s existing Node dependencies and Playwright Chromium installation. Building a package does not publish it.

Granularity#

MeasureGranularity is available for 2D images, in native Python and WASM. It uses the pinned CellProfiler 4.2.8 reference, with scipy 1.9.3 and scikit-image 0.21.0.

from cp_anywhere import measure_granularity

result = measure_granularity(
    dna_image,
    objects={"Cells": cell_labels},
    subsample_size=0.25,  # CellProfiler's own defaults
    image_sample_size=0.25,
    element_size=10,
    granular_spectrum_length=16,
)
result.image[0]  # GS1, the image-level granularity spectrum
result.objects["Cells"][0]  # GS1, one value per Cells object

measure_granularity is a standalone function, separate from the unified measure() family dispatch: CellProfiler’s own module measures one image at a time, with its own subsampling/structuring-element settings, and its result shape (per-scale spectra, not a flat feature table) does not fit the shared measure() contract.

Algorithm#

For each image: optionally downsample for the main granularity computation (subsample_size) and separately for background removal (image_sample_size), using CellProfiler’s own mix of bilinear (order 1) pixel resampling and cubic B-spline (order 3) mask resampling. Subtract a grayscale-opened background (erosion then dilation with a disk structuring element of element_size radius), then repeatedly erode with a radius-1 disk and reconstruct by dilation, recording the percent change in masked mean brightness at each of granular_spectrum_length scales.

Per-object spectra use the same per-scale reconstructed image, rescaled back to full resolution, averaged over each object’s unmasked pixels at full resolution – not the (possibly subsampled) working image directly.

An object with zero pixels under the mask reports NaN at every scale, matching real CellProfiler: its own baseline mean is already NaN, and NaN never recovers through the per-scale subtraction.

Limits#

  • 2D only. Real CellProfiler’s own module divides by orig_shape[dimension] - 1 while rescaling its working image back to full resolution, and raises ZeroDivisionError, unconditionally, for any image with a 1-row or 1-column dimension – confirmed live against the pinned reference. measure_granularity raises a clear ValueError for that case instead of silently producing 0/NaN; there is no real oracle value to match for it.

  • image must already be at the same precision real CellProfiler would have measured. Image(..., convert=True) rescales integer dtypes by their dtype’s max value (identify_primary_objects reproduces this automatically for segmentation inputs, but measure_granularity does not – intensity measurements expect already-scaled float input, the same contract measure()’s other intensity families use) and always downcasts through float32 even for float64 input. The second effect is usually negligible (~1e-7 relative) but is a real precision difference, not a cp-anywhere rounding bug, when reproducing an exact oracle run numerically.

  • The .cppipe importer accepts revision 4 (7 settings: images, “measure within objects?”, objects, both subsampling factors, element size, spectrum length). Every listed image is measured independently and merged into the same MeasurementResult, matching real CellProfiler’s Granularity_<scale>_<image name> feature-naming convention.

Verification#

Real reference fixtures cover CellProfiler’s own defaults, full resolution (no subsampling), uneven subsample/background factors, partial and fully-empty masks, zero/constant/negative images, and multiple simultaneous object sets – 12 cases total, generated by scripts/reference/generate_granularity.py against a live pinned CellProfiler run. A separate real two-image, two-object-set .cppipe pipeline (scripts/cppipe/generate_granularity_pipeline.py) exercises the full IdentifyPrimaryObjects -> IdentifySecondaryObjects -> IdentifyTertiaryObjects -> MeasureGranularity chain end to end.

The browser check runs measureGranularityJson in real Playwright Chromium, not a simulated WASM runtime.

To regenerate and check the reference data:

PYTHONPATH=/path/to/pinned/CellProfiler /path/to/reference/python \
  scripts/reference/generate_granularity.py
PYTHONPATH=/path/to/pinned/CellProfiler /path/to/reference/python \
  scripts/cppipe/generate_granularity_pipeline.py
uv run --frozen pytest tests/test_granularity.py tests/cppipe/test_granularity_pipeline.py
python scripts/wasm/build_package.py
uv run --frozen python scripts/parity/compare_wasm_granularity.py

The browser check requires the browser example’s existing Node dependencies and Playwright Chromium installation.

CellProfiler pipeline (.cppipe) import#

cp_anywhere.load_pipeline(path) reads a real CellProfiler .cppipe file and translates the settings of every module this project implements (IdentifyPrimaryObjects, IdentifySecondaryObjects, IdentifyTertiaryObjects, MeasureImageIntensity, MeasureObjectIntensity, MeasureObjectSizeShape, MeasureGranularity, MeasureObjectIntensityDistribution, MeasureTexture, MeasureColocalization, MeasureObjectNeighbors) into cp_anywhere’s own configuration – an existing pipeline’s settings for those modules carry over 1-for-1, with no manual re-entry.

import cp_anywhere

pipeline = cp_anywhere.load_pipeline("my_pipeline.cppipe")
segmentations, derived_objects, measurement = pipeline.run(
    {"DNA": dna_image, "Actin": actin_image}
)

The second result contains both secondary and tertiary object sets. For example, derived_objects["Cytoplasm"].labels contains the tertiary labels. The return value remains a three-item tuple.

This is a settings importer, not a full pipeline execution engine: file-matching/organization modules (Images, Metadata, NamesAndTypes, Groups, …) are recognized and skipped – pipeline.skipped_modules lists them – not executed; you supply already-loaded image arrays by name, same as cp_anywhere.measure.

Every out-of-scope setting (an older pipeline revision, “basic mode”, a non-"Adjacent" MeasureObjectNeighbors distance method, an unimplemented IdentifySecondaryObjects method, and a handful more) raises cp_anywhere.CppipeError naming the exact setting, rather than silently mis-translating it – see spec/cppipe-compatibility.md for the full settings-mapping table, every explicit gap, and how it is verified (real, oracle-generated .cppipe fixtures in fixtures/cppipe/, checked in tests/cppipe/test_cppipe_import.py against the pinned CellProfiler 4.2.8 oracle’s own segmentation output on the same test image). The CLI also accepts a pipeline directly:

cp-anywhere --pipeline my_pipeline.cppipe --image DNA=dna.tif --output result.json

Database export#

MeasurementResult can be written directly to a real Parquet file, a DuckDB database, or a SQLite database (using STRICT tables), on top of the existing to_csv/to_arrow/to_json:

result.to_parquet("result.parquet")  # same long-form rows as to_arrow
result.to_duckdb("results.duckdb")  # wide Per_Image / Per_Cells / ...
result.to_sqlite("results.sqlite")  # same, as SQLite STRICT tables

to_duckdb/to_sqlite write wide tables named Per_Image/ Per_<ObjectName> – matching real CellProfiler’s own ExportToDatabase naming convention – and default to if_exists="append", so processing many images in a loop and calling one of these once per image builds up a single combined dataset, the way CellProfiler’s own batch runs do. Pass if_exists="replace" to start fresh instead.

to_duckdb needs the optional duckdb dependency (pip install cp-anywhere[duckdb]); to_parquet and to_sqlite need nothing beyond this project’s own core dependencies.

For a dense, fixed-size feature array per object instead of long-form rows (e.g. to load straight into a (n_rows, n_features) numpy array), use to_parquet(..., mode="array"): path becomes a directory, one Per_Image.parquet/Per_<ObjectName>.parquet file is written per scope, and each file’s features column is a single fixed-size array per row. The exact ordered feature names for that array are embedded in the file’s own Parquet schema metadata (no sidecar file to go out of sync) – read them back with cp_anywhere.read_parquet_feature_names(path). The CLI exposes this as --output <directory> --parquet-mode array.

to_parquet(..., mode="wide") writes the same per-scope files as mode="array", but as the traditional tabular shape instead – one real, named column per feature, no array/metadata lookup needed (--output <directory> --parquet-mode wide).

For one combined, CytoTable -style table across every object scope – one row per object, scopes joined on object_id, feature columns prefixed by scope name – use to_parquet(..., mode="joined"). path is either a single .parquet file or a directory, in which case it’s written as a real, max_rows_per_file-paginated Parquet dataset via pyarrow.dataset.write_dataset that can accumulate across many images with if_exists="append" (the default): --output <file-or-directory> --parquet-mode joined --join outer|inner --max-rows-per-file N.

The CLI dispatches on --output’s file extension the same way:

cp-anywhere --image DNA=dna.tif --objects Cells=labels.npy \
  --output results.sqlite --if-exists append

See spec/database-export.md for the exact schema (column names and types), why SQLite specifically needs STRICT tables, and how it’s verified (tests/test_database_export.py).

CI/CD#

On every pull request (.github/workflows/run-tests.yml)#

  • pre_commit_checks: all pre-commit hooks (ruff, cargo fmt/clippy via the Rust hooks below, yamllint, codespell, mdformat, zizmor, vulture, ty). Syncs every uv dependency group (--all-groups) so ty can resolve the demo-only onnx/duckdb imports in scripts/zarr/*.py.

  • run_tests (matrix: Python 3.11-3.15 x Ubuntu/macOS): cargo fmt --check, cargo clippy -D warnings, cargo test, maturin develop + pytest, and scripts/parity/compare_native.py (golden-fixture parity; does not need the live CellProfiler oracle, so it runs on every PR across the full OS/Python matrix).

  • wasm_build_and_browser_check: builds crates/cp_anywhere_wasm with wasm-pack, builds the examples/browser-vite Vite demo, runs it in a real headless Chromium via Playwright (npm run browser-check), and checks the captured output with scripts/parity/compare_wasm.py – a real WASM build and browser smoke test on every PR, not a native-only check.

Nightly (.github/workflows/nightly-full-parity.yml)#

Installs the pinned CellProfiler 4.2.8 oracle (reference/install.sh, which needs a JDK for python-javabridge – too heavy to run on every PR) and re-runs the full suite against it: pytest with the oracle configured, compare_native.py, and a full WASM build + browser check + compare_wasm.py. This is the plan’s “full feature inventory parity across native/Python/WASM” nightly check. Also runnable on demand (workflow_dispatch).

Weekly (.github/workflows/benchmark.yml)#

Native Rust throughput benchmark (Criterion) against the committed baseline, with a tolerance-based regression gate (scripts/benchmarks/check_regression.py) – see docs/performance.md for why this runs on a schedule rather than every PR (CI runner timing noise).

On release (.github/workflows/publish-*.yml)#

See docs/release.md for the full publish pipeline (PyPI, crates.io, npm, GitHub Release assets).

Known gaps#

  • Property-based tests for invariants such as label permutation and object ordering (plan section 8) are not implemented. The current test suite checks golden-fixture correctness, not permutation-invariance properties; this would be a reasonable follow-up using proptest (Rust) or hypothesis (Python).

  • None of the new CI workflows added in M7 (wasm_build_and_browser_check, nightly-full-parity.yml) have been exercised by a real GitHub Actions run in this session – only validated locally (the exact same commands each step runs were run directly in this session and passed) and statically (actionlint/zizmor via pre-commit).