Project documentation#
Supported feature table#
Generated by scripts/docs/render_feature_table.py by joining
spec/features.yaml (the frozen empirical CellProfiler 4.2.8 oracle
inventory, M1 – never hand-edited or mutated by this script) with
the real parity reports (benchmarks/reports/native-parity.json,
benchmarks/reports/wasm-parity.json, M3-M5) and the cp_measure
mapping (spec/cp_measure-map.yaml, M2). Regenerate after any
parity re-run; do not hand-edit.
Total declared columns: 119 (see spec/compatibility.md for the full scope contract).
Module |
Feature |
Object scope |
Dims |
Rust |
WASM |
cp_measure |
|---|---|---|---|---|---|---|
MeasureImageIntensity |
Intensity_LowerQuartileIntensity_Actin |
Image |
2/3 |
pass_exact |
pass_exact |
unsupported |
MeasureImageIntensity |
Intensity_LowerQuartileIntensity_DNA |
Image |
2/3 |
pass_exact |
pass_exact |
unsupported |
MeasureImageIntensity |
Intensity_MADIntensity_Actin |
Image |
2/3 |
pass_exact |
pass_exact |
unsupported |
MeasureImageIntensity |
Intensity_MADIntensity_DNA |
Image |
2/3 |
pass_exact |
pass_exact |
unsupported |
MeasureImageIntensity |
Intensity_MaxIntensity_Actin |
Image |
2/3 |
pass_exact |
pass_exact |
unsupported |
MeasureImageIntensity |
Intensity_MaxIntensity_DNA |
Image |
2/3 |
pass_exact |
pass_exact |
unsupported |
MeasureImageIntensity |
Intensity_MeanIntensity_Actin |
Image |
2/3 |
pass_exact |
pass_exact |
unsupported |
MeasureImageIntensity |
Intensity_MeanIntensity_DNA |
Image |
2/3 |
pass_tolerance |
pass_tolerance |
unsupported |
MeasureImageIntensity |
Intensity_MedianIntensity_Actin |
Image |
2/3 |
pass_exact |
pass_exact |
unsupported |
MeasureImageIntensity |
Intensity_MedianIntensity_DNA |
Image |
2/3 |
pass_exact |
pass_exact |
unsupported |
MeasureImageIntensity |
Intensity_MinIntensity_Actin |
Image |
2/3 |
pass_exact |
pass_exact |
unsupported |
MeasureImageIntensity |
Intensity_MinIntensity_DNA |
Image |
2/3 |
pass_exact |
pass_exact |
unsupported |
MeasureImageIntensity |
Intensity_PercentMaximal_Actin |
Image |
2/3 |
pass_exact |
pass_exact |
unsupported |
MeasureImageIntensity |
Intensity_PercentMaximal_DNA |
Image |
2/3 |
pass_exact |
pass_exact |
unsupported |
MeasureImageIntensity |
Intensity_Percentile_10_Actin |
Image |
2/3 |
pass_exact |
pass_exact |
unsupported |
MeasureImageIntensity |
Intensity_Percentile_10_DNA |
Image |
2/3 |
pass_tolerance |
pass_tolerance |
unsupported |
MeasureImageIntensity |
Intensity_Percentile_50_Actin |
Image |
2/3 |
pass_exact |
pass_exact |
unsupported |
MeasureImageIntensity |
Intensity_Percentile_50_DNA |
Image |
2/3 |
pass_exact |
pass_exact |
unsupported |
MeasureImageIntensity |
Intensity_Percentile_90_Actin |
Image |
2/3 |
pass_exact |
pass_exact |
unsupported |
MeasureImageIntensity |
Intensity_Percentile_90_DNA |
Image |
2/3 |
pass_tolerance |
pass_tolerance |
unsupported |
MeasureImageIntensity |
Intensity_StdIntensity_Actin |
Image |
2/3 |
pass_tolerance |
pass_tolerance |
unsupported |
MeasureImageIntensity |
Intensity_StdIntensity_DNA |
Image |
2/3 |
pass_tolerance |
pass_tolerance |
unsupported |
MeasureImageIntensity |
Intensity_TotalArea_Actin |
Image |
2/3 |
pass_exact |
pass_exact |
unsupported |
MeasureImageIntensity |
Intensity_TotalArea_DNA |
Image |
2/3 |
pass_exact |
pass_exact |
unsupported |
MeasureImageIntensity |
Intensity_TotalIntensity_Actin |
Image |
2/3 |
pass_exact |
pass_exact |
unsupported |
MeasureImageIntensity |
Intensity_TotalIntensity_DNA |
Image |
2/3 |
pass_tolerance |
pass_tolerance |
unsupported |
MeasureImageIntensity |
Intensity_UpperQuartileIntensity_Actin |
Image |
2/3 |
pass_exact |
pass_exact |
unsupported |
MeasureImageIntensity |
Intensity_UpperQuartileIntensity_DNA |
Image |
2/3 |
pass_tolerance |
pass_tolerance |
unsupported |
MeasureObjectIntensity |
Intensity_IntegratedIntensityEdge_Actin |
Cells |
2/3 |
pass_exact |
pass_exact |
mapped |
MeasureObjectIntensity |
Intensity_IntegratedIntensityEdge_Actin |
Nuclei |
2/3 |
pass_exact |
pass_exact |
mapped |
MeasureObjectIntensity |
Intensity_IntegratedIntensityEdge_DNA |
Cells |
2/3 |
pass_exact |
pass_exact |
mapped |
MeasureObjectIntensity |
Intensity_IntegratedIntensityEdge_DNA |
Nuclei |
2/3 |
pass_exact |
pass_exact |
mapped |
MeasureObjectIntensity |
Intensity_IntegratedIntensity_Actin |
Cells |
2/3 |
pass_exact |
pass_exact |
mapped |
MeasureObjectIntensity |
Intensity_IntegratedIntensity_Actin |
Nuclei |
2/3 |
pass_exact |
pass_exact |
mapped |
MeasureObjectIntensity |
Intensity_IntegratedIntensity_DNA |
Cells |
2/3 |
pass_exact |
pass_exact |
mapped |
MeasureObjectIntensity |
Intensity_IntegratedIntensity_DNA |
Nuclei |
2/3 |
pass_exact |
pass_exact |
mapped |
MeasureObjectIntensity |
Intensity_LowerQuartileIntensity_Actin |
Cells |
2/3 |
pass_exact |
pass_exact |
mapped |
MeasureObjectIntensity |
Intensity_LowerQuartileIntensity_Actin |
Nuclei |
2/3 |
pass_exact |
pass_exact |
mapped |
MeasureObjectIntensity |
Intensity_LowerQuartileIntensity_DNA |
Cells |
2/3 |
pass_exact |
pass_exact |
mapped |
MeasureObjectIntensity |
Intensity_LowerQuartileIntensity_DNA |
Nuclei |
2/3 |
pass_exact |
pass_exact |
mapped |
MeasureObjectIntensity |
Intensity_MADIntensity_Actin |
Cells |
2/3 |
pass_exact |
pass_exact |
mapped |
MeasureObjectIntensity |
Intensity_MADIntensity_Actin |
Nuclei |
2/3 |
pass_exact |
pass_exact |
mapped |
MeasureObjectIntensity |
Intensity_MADIntensity_DNA |
Cells |
2/3 |
pass_exact |
pass_exact |
mapped |
MeasureObjectIntensity |
Intensity_MADIntensity_DNA |
Nuclei |
2/3 |
pass_exact |
pass_exact |
mapped |
MeasureObjectIntensity |
Intensity_MassDisplacement_Actin |
Cells |
2/3 |
pass_exact |
pass_exact |
mapped |
MeasureObjectIntensity |
Intensity_MassDisplacement_Actin |
Nuclei |
2/3 |
pass_exact |
pass_exact |
mapped |
MeasureObjectIntensity |
Intensity_MassDisplacement_DNA |
Cells |
2/3 |
pass_exact |
pass_exact |
mapped |
MeasureObjectIntensity |
Intensity_MassDisplacement_DNA |
Nuclei |
2/3 |
pass_exact |
pass_exact |
mapped |
MeasureObjectIntensity |
Intensity_MaxIntensityEdge_Actin |
Cells |
2/3 |
pass_exact |
pass_exact |
mapped |
MeasureObjectIntensity |
Intensity_MaxIntensityEdge_Actin |
Nuclei |
2/3 |
pass_exact |
pass_exact |
mapped |
MeasureObjectIntensity |
Intensity_MaxIntensityEdge_DNA |
Cells |
2/3 |
pass_exact |
pass_exact |
mapped |
MeasureObjectIntensity |
Intensity_MaxIntensityEdge_DNA |
Nuclei |
2/3 |
pass_exact |
pass_exact |
mapped |
MeasureObjectIntensity |
Intensity_MaxIntensity_Actin |
Cells |
2/3 |
pass_exact |
pass_exact |
mapped |
MeasureObjectIntensity |
Intensity_MaxIntensity_Actin |
Nuclei |
2/3 |
pass_exact |
pass_exact |
mapped |
MeasureObjectIntensity |
Intensity_MaxIntensity_DNA |
Cells |
2/3 |
pass_exact |
pass_exact |
mapped |
MeasureObjectIntensity |
Intensity_MaxIntensity_DNA |
Nuclei |
2/3 |
pass_exact |
pass_exact |
mapped |
MeasureObjectIntensity |
Intensity_MeanIntensityEdge_Actin |
Cells |
2/3 |
pass_exact |
pass_exact |
mapped |
MeasureObjectIntensity |
Intensity_MeanIntensityEdge_Actin |
Nuclei |
2/3 |
pass_exact |
pass_exact |
mapped |
MeasureObjectIntensity |
Intensity_MeanIntensityEdge_DNA |
Cells |
2/3 |
pass_exact |
pass_exact |
mapped |
MeasureObjectIntensity |
Intensity_MeanIntensityEdge_DNA |
Nuclei |
2/3 |
pass_exact |
pass_exact |
mapped |
MeasureObjectIntensity |
Intensity_MeanIntensity_Actin |
Cells |
2/3 |
pass_exact |
pass_exact |
mapped |
MeasureObjectIntensity |
Intensity_MeanIntensity_Actin |
Nuclei |
2/3 |
pass_exact |
pass_exact |
mapped |
MeasureObjectIntensity |
Intensity_MeanIntensity_DNA |
Cells |
2/3 |
pass_exact |
pass_exact |
mapped |
MeasureObjectIntensity |
Intensity_MeanIntensity_DNA |
Nuclei |
2/3 |
pass_exact |
pass_exact |
mapped |
MeasureObjectIntensity |
Intensity_MedianIntensity_Actin |
Cells |
2/3 |
pass_exact |
pass_exact |
mapped |
MeasureObjectIntensity |
Intensity_MedianIntensity_Actin |
Nuclei |
2/3 |
pass_exact |
pass_exact |
mapped |
MeasureObjectIntensity |
Intensity_MedianIntensity_DNA |
Cells |
2/3 |
pass_exact |
pass_exact |
mapped |
MeasureObjectIntensity |
Intensity_MedianIntensity_DNA |
Nuclei |
2/3 |
pass_exact |
pass_exact |
mapped |
MeasureObjectIntensity |
Intensity_MinIntensityEdge_Actin |
Cells |
2/3 |
pass_exact |
pass_exact |
mapped |
MeasureObjectIntensity |
Intensity_MinIntensityEdge_Actin |
Nuclei |
2/3 |
pass_exact |
pass_exact |
mapped |
MeasureObjectIntensity |
Intensity_MinIntensityEdge_DNA |
Cells |
2/3 |
pass_exact |
pass_exact |
mapped |
MeasureObjectIntensity |
Intensity_MinIntensityEdge_DNA |
Nuclei |
2/3 |
pass_exact |
pass_exact |
mapped |
MeasureObjectIntensity |
Intensity_MinIntensity_Actin |
Cells |
2/3 |
pass_exact |
pass_exact |
mapped |
MeasureObjectIntensity |
Intensity_MinIntensity_Actin |
Nuclei |
2/3 |
pass_exact |
pass_exact |
mapped |
MeasureObjectIntensity |
Intensity_MinIntensity_DNA |
Cells |
2/3 |
pass_exact |
pass_exact |
mapped |
MeasureObjectIntensity |
Intensity_MinIntensity_DNA |
Nuclei |
2/3 |
pass_exact |
pass_exact |
mapped |
MeasureObjectIntensity |
Intensity_StdIntensityEdge_Actin |
Cells |
2/3 |
pass_exact |
pass_exact |
mapped |
MeasureObjectIntensity |
Intensity_StdIntensityEdge_Actin |
Nuclei |
2/3 |
pass_exact |
pass_exact |
mapped |
MeasureObjectIntensity |
Intensity_StdIntensityEdge_DNA |
Cells |
2/3 |
pass_exact |
pass_exact |
mapped |
MeasureObjectIntensity |
Intensity_StdIntensityEdge_DNA |
Nuclei |
2/3 |
pass_exact |
pass_exact |
mapped |
MeasureObjectIntensity |
Intensity_StdIntensity_Actin |
Cells |
2/3 |
pass_exact |
pass_exact |
mapped |
MeasureObjectIntensity |
Intensity_StdIntensity_Actin |
Nuclei |
2/3 |
pass_exact |
pass_exact |
mapped |
MeasureObjectIntensity |
Intensity_StdIntensity_DNA |
Cells |
2/3 |
pass_exact |
pass_exact |
mapped |
MeasureObjectIntensity |
Intensity_StdIntensity_DNA |
Nuclei |
2/3 |
pass_exact |
pass_exact |
mapped |
MeasureObjectIntensity |
Intensity_UpperQuartileIntensity_Actin |
Cells |
2/3 |
pass_exact |
pass_exact |
mapped |
MeasureObjectIntensity |
Intensity_UpperQuartileIntensity_Actin |
Nuclei |
2/3 |
pass_exact |
pass_exact |
mapped |
MeasureObjectIntensity |
Intensity_UpperQuartileIntensity_DNA |
Cells |
2/3 |
pass_exact |
pass_exact |
mapped |
MeasureObjectIntensity |
Intensity_UpperQuartileIntensity_DNA |
Nuclei |
2/3 |
pass_exact |
pass_exact |
mapped |
MeasureObjectIntensity |
Location_CenterMassIntensity_X_Actin |
Cells |
2/3 |
pass_exact |
pass_exact |
mapped |
MeasureObjectIntensity |
Location_CenterMassIntensity_X_Actin |
Nuclei |
2/3 |
pass_exact |
pass_exact |
mapped |
MeasureObjectIntensity |
Location_CenterMassIntensity_X_DNA |
Cells |
2/3 |
pass_exact |
pass_exact |
mapped |
MeasureObjectIntensity |
Location_CenterMassIntensity_X_DNA |
Nuclei |
2/3 |
pass_exact |
pass_exact |
mapped |
MeasureObjectIntensity |
Location_CenterMassIntensity_Y_Actin |
Cells |
2/3 |
pass_exact |
pass_exact |
mapped |
MeasureObjectIntensity |
Location_CenterMassIntensity_Y_Actin |
Nuclei |
2/3 |
pass_exact |
pass_exact |
mapped |
MeasureObjectIntensity |
Location_CenterMassIntensity_Y_DNA |
Cells |
2/3 |
pass_exact |
pass_exact |
mapped |
MeasureObjectIntensity |
Location_CenterMassIntensity_Y_DNA |
Nuclei |
2/3 |
pass_exact |
pass_exact |
mapped |
MeasureObjectIntensity |
Location_CenterMassIntensity_Z_Actin |
Cells |
2/3 |
pass_exact |
pass_exact |
mapped |
MeasureObjectIntensity |
Location_CenterMassIntensity_Z_Actin |
Nuclei |
2/3 |
pass_exact |
pass_exact |
mapped |
MeasureObjectIntensity |
Location_CenterMassIntensity_Z_DNA |
Cells |
2/3 |
pass_exact |
pass_exact |
mapped |
MeasureObjectIntensity |
Location_CenterMassIntensity_Z_DNA |
Nuclei |
2/3 |
pass_exact |
pass_exact |
mapped |
MeasureObjectIntensity |
Location_MaxIntensity_X_Actin |
Cells |
2/3 |
pass_exact |
pass_exact |
mapped |
MeasureObjectIntensity |
Location_MaxIntensity_X_Actin |
Nuclei |
2/3 |
pass_exact |
pass_exact |
mapped |
MeasureObjectIntensity |
Location_MaxIntensity_X_DNA |
Cells |
2/3 |
pass_exact |
pass_exact |
mapped |
MeasureObjectIntensity |
Location_MaxIntensity_X_DNA |
Nuclei |
2/3 |
pass_exact |
pass_exact |
mapped |
MeasureObjectIntensity |
Location_MaxIntensity_Y_Actin |
Cells |
2/3 |
pass_exact |
pass_exact |
mapped |
MeasureObjectIntensity |
Location_MaxIntensity_Y_Actin |
Nuclei |
2/3 |
pass_exact |
pass_exact |
mapped |
MeasureObjectIntensity |
Location_MaxIntensity_Y_DNA |
Cells |
2/3 |
pass_exact |
pass_exact |
mapped |
MeasureObjectIntensity |
Location_MaxIntensity_Y_DNA |
Nuclei |
2/3 |
pass_exact |
pass_exact |
mapped |
MeasureObjectIntensity |
Location_MaxIntensity_Z_Actin |
Cells |
2/3 |
pass_exact |
pass_exact |
mapped |
MeasureObjectIntensity |
Location_MaxIntensity_Z_Actin |
Nuclei |
2/3 |
pass_exact |
pass_exact |
mapped |
MeasureObjectIntensity |
Location_MaxIntensity_Z_DNA |
Cells |
2/3 |
pass_exact |
pass_exact |
mapped |
MeasureObjectIntensity |
Location_MaxIntensity_Z_DNA |
Nuclei |
2/3 |
pass_exact |
pass_exact |
mapped |
MeasureObjectNeighbors |
Neighbors_AngleBetweenNeighbors_Adjacent |
Cells |
2/3 |
pass_exact |
pass_tolerance |
mapped |
MeasureObjectNeighbors |
Neighbors_FirstClosestDistance_Adjacent |
Cells |
2/3 |
pass_exact |
pass_exact |
mapped |
MeasureObjectNeighbors |
Neighbors_FirstClosestObjectNumber_Adjacent |
Cells |
2/3 |
pass_exact |
pass_exact |
mapped |
MeasureObjectNeighbors |
Neighbors_NumberOfNeighbors_Adjacent |
Cells |
2/3 |
pass_exact |
pass_exact |
mapped |
MeasureObjectNeighbors |
Neighbors_PercentTouching_Adjacent |
Cells |
2/3 |
pass_exact |
pass_exact |
mapped |
MeasureObjectNeighbors |
Neighbors_SecondClosestDistance_Adjacent |
Cells |
2/3 |
pass_exact |
pass_exact |
mapped |
MeasureObjectNeighbors |
Neighbors_SecondClosestObjectNumber_Adjacent |
Cells |
2/3 |
pass_exact |
pass_exact |
mapped |
M3 native Rust core#
Result#
The Rust library (crates/cp_anywhere) implements the complete declared
scope from spec/compatibility.md: MeasureImageIntensity,
MeasureObjectIntensity over Cells/Nuclei × DNA/Actin, and
MeasureObjectNeighbors same-set Adjacent with directed relationship
rows. crates/cp_anywhere_python is a thin PyO3/NumPy binding with no
scientific logic of its own; src/cp_anywhere/api.py validates inputs and
exposes cp_anywhere.measure(...).
scripts/parity/compare_native.py compares the Rust result against all 13
saved golden fixtures and all 119 declared columns plus the directed
relationship table: 1,560 of 1,560 rows pass (1,535 exact, 25 within
the spec/features.yaml float64 tolerance). The 25 tolerance rows are all
MeasureImageIntensity scalar reductions with errors between 1e-17 and
1e-14 — float64 summation-order variance, not a scope or semantics bug.
Full counts and reasons are recorded in spec/native-coverage.yaml and
benchmarks/reports/native-parity.json.
cargo test --manifest-path crates/cp_anywhere/Cargo.toml
env -u CONDA_PREFIX uv run maturin develop --release --uv
uv run --frozen python scripts/parity/compare_native.py
CP_ANYWHERE_REFERENCE_PYTHON=/path/to/reference/venv/bin/python \
PYTHONPATH=/path/to/pinned/CellProfiler \
uv run --frozen pytest
env -u CONDA_PREFIX is required on hosts where both CONDA_PREFIX and
VIRTUAL_ENV are set; maturin refuses to build with both present.
Hard-case source decisions#
These follow spec/reference-semantics.md, which records the exact
original behavior read from the pinned CellProfiler 4.2.8 and NumPy 1.23.5
sources, not a cleaned-up reimplementation:
Integer images are promoted to
f64without dtype-maximum rescaling, matching the reference runner’sImage(..., convert=False).Object-quantile and MAD formulas use
n * fraction(not(n-1) * fraction) andn / dimensions, matching the original module exactly.Maximum-intensity coordinate ties are resolved by a literal Rust port of NumPy 1.23.5’s introsort (
numpy_argsort/argsort_range/numpy_heapsortinlib.rs), derived fromnumpy/core/src/npysort/quicksort.cppandnpysort_heapsort.hatde82cd9468704a033702974010ee7e7efc85b393(BSD-3-Clause; seelicenses/numpy.txtandTHIRD_PARTY_NOTICES.md). A stable per-object sort does not reproduce the golden tie-break positions; this port does.Same-set adjacent neighbors use only the first two center axes even in 3D (z/y, omitting x) for nearest-distance and angle, a separate ordering for distance/angle selection versus the ID ordering, and the original
centrosome.outline.outline-equivalent first-two-axes diagonal outline (also applied unchanged in 3D). These are preserved reference quirks, not defects introduced by this port.Sparse object IDs keep their array position through the maximum label; missing slots are not compacted.
Resource and memory contract#
Inputs: 2D or 3D only, nonzero shape, checked shape-product multiplication (
checked_mul), matching channel/object/mask lengths, labels must be unsigned and<= MAX_LABEL(1,000,000; a documented limit, not an attempt to allocateu32::MAXrows).crates/cp_anywhere_python/src/lib.rsborrows C-contiguousf64image arrays andu32label arrays directly as slices — no copy for that common case.src/cp_anywhere/api.pyperforms exactly one conversion vianp.ascontiguousarrayfor other numeric dtypes or non-contiguous input; it does not claim zero-copy for those cases.The kernel runs with
py.detach(...)(PyO3’s GIL release), so Rust work does not hold the GIL. Callers must not mutate input arrays whilemeasure()is running, since the Rust side holds borrowed slices for the call’s duration.Rejected inputs: 1D or higher-than-3D shapes, boolean/float label arrays, non-real-numeric image dtypes, mismatched shapes, labels above
MAX_LABEL, and (when neighbors are selected) noCellsobject set.The library (
cp_anywhere) has no Python, filesystem, or platform dependency:Input/ResultTableare plain Rust types over borrowed slices, which keeps a future WASM target possible without a rewrite.
API#
cp_anywhere.measure(images, objects, *, mask=None, image_id="Image", families=None) returns a frozen MeasurementResult with .features
(mirroring the golden JSON structure: Image scope holds scalars, object
scopes hold NumPy arrays aligned to .object_ids), .object_ids, and
.relationships (directed edges with both image/object IDs). .rows()
gives a long table; .to_csv()/.to_json() write it and the
reference-shaped structure respectively, with nonfinite floats encoded
explicitly ({"nonfinite": "NaN"}, "+Infinity", "-Infinity") rather
than silently dropped or turned into null. families restricts
computation to a subset of ("image_intensity", "object_intensity", "neighbors") so later benchmarking can time an equivalent subset without
computing unselected families.
The CLI (cp-anywhere --image NAME=PATH --objects NAME=PATH [--mask PATH] --output PATH) loads saved NPY arrays and writes CSV or JSON depending on
the output suffix.
Wheel build and installed-wheel smoke test#
maturin build --release produced
cp_anywhere-0.1.0-cp311-abi3-macosx_11_0_arm64.whl. It was installed into
a fresh, isolated uv venv with no project checkout on sys.path. The
imported module resolved to that venv’s site-packages (not the source
tree), and cp_anywhere.measure(...) on the touching_gradient fixture
returned all 119 declared columns. This proves the installed wheel, not
just the editable development install, produces the full declared scope.
cargo fmt --check and cargo clippy --all-targets are clean for both
crates. cargo test passes the one Rust unit test (invalid shape and
over-limit label rejection). The full project pytest suite (22 tests,
including the real CellProfiler 4.2.8 replay) passes.
Known limitations#
Only the one
rejects_bad_shape_and_large_idRust unit test exists; most correctness assurance currently comes from the Python-side golden parity script, not from Rust-level property tests (e.g. proptest) on the sorting or neighbor kernels.Timing against
cp_measureand against a WASM build is explicitly out of scope for M3; seedocs/decisions/0002-rust-core.mdand M4/M6.
M4 WASM feasibility and browser integration spike#
Result#
The Rust core compiles to wasm32-unknown-unknown with its default
features, including the .cppipe executor. The core does not depend on
Python or browser bindings.
crates/cp_anywhere_wasm is a thin wasm-bindgen binding (JSON in/out for
small metadata, typed arrays for the numeric payload). The build script
scripts/wasm/build_package.py uses wasm-pack and names the npm package
cp-anywhere. The Vite example
(examples/browser-vite) that runs the module inside a dedicated web
worker.
Both measurement and segmentation run entirely client-side with
verified parity against the real CellProfiler 4.2.8 oracle. A real,
reproducible headless-Chromium run (Playwright, not a mock DOM;
examples/browser-vite/run-browser-check.mjs) executes the worker
against:
All 13 declared measurement golden fixtures (2D and 3D) via
measure(), compared byscripts/parity/compare_wasm.py– 1,560 of 1,560 rows pass (1,533 exact, 27 within the declared float64 tolerance; zero failures). 13 fixtures × 120 rows (119 features + 1 relationship table) – the identical row count the native build passes (docs/native.md), closing M5’s “100% … in native Rust, Python, and WASM” gate for the current declared manifest.All 100 segmentation golden fixtures (
fixtures/segmentation/) viaidentifyPrimaryObjects(), compared byscripts/parity/compare_wasm_segmentation.py– 100 of 100 pass exactly on labels and object count (one fixture’s scalar threshold uses the same documented, narrow, label-safe float32/float64 tolerance exception as the native/Python build; seetests/segmentation/test_reference_alignment.py’s_KNOWN_THRESHOLD_MAP_TOLERANCE_GAP) – seedocs/segmentation-coverage.md.
The 27 measurement tolerance rows are the same MeasureImageIntensity
reduction-order families already seen in the native build
(docs/native.md), plus two Neighbors_AngleBetweenNeighbors_Adjacent
rows at 1e-14-scale absolute / 1e-16-scale relative error –
consistent with a transcendental-function (acos/atan2) last-bit
difference between wasm32’s and native arm64’s libm, not a computation
bug.
cargo fmt --manifest-path crates/cp_anywhere_wasm/Cargo.toml --check
cargo clippy --manifest-path crates/cp_anywhere_wasm/Cargo.toml --target wasm32-unknown-unknown --all-targets
python scripts/wasm/build_package.py
cd examples/browser-vite
npm install
uv run --frozen python ../../scripts/wasm/export_fixtures.py # measurement fixtures (gitignored)
uv run --frozen python ../../scripts/wasm/export_segmentation_fixtures.py # segmentation fixtures (gitignored)
npm run build
npx playwright install chromium # once
npm run browser-check # writes browser-results.json
cd ../..
uv run --frozen python scripts/parity/compare_wasm.py
uv run --frozen python scripts/parity/compare_wasm_segmentation.py
Browser demo and npm package#
The documentation also includes Cell profiling in your browser. It runs illumination correction, segmentation, and measurements on the same crop as the Python notebook. To build and serve that page locally from the repository root:
python scripts/wasm/build_package.py
uv run --frozen --group docs sphinx-build -W docs/src docs/_build/html
uv run --frozen python scripts/wasm/stage_docs_demo.py docs/_build/html
python -m http.server 8000 --bind 127.0.0.1 --directory docs/_build/html
Open http://127.0.0.1:8000/browser-demo.html. The published docs build
includes these assets. It does not fetch the runtime from npm or a CDN.
examples/browser-vite is a real Vite project (not a static mockup): its
src/worker.ts imports the generated cp-anywhere package, fetches
each fixture’s raw binary arrays (see below), calls measure() and
identifyPrimaryObjects(), and posts the results back; src/main.ts
drives the worker from the page and renders status in #status.
npm run build produces a genuine production bundle:
dist/index.html 0.46 kB
dist/assets/worker-*.js ~9.0 kB
dist/assets/cp_anywhere_wasm_bg-*.wasm 377.62 kB (146.80 kB gzip-equivalent)
dist/assets/index-*.js ~2.5 kB
npm run preview (or run-browser-check.mjs’s own tiny static server)
serves this real build; nothing is dev-server-only or dynamically
recompiled per request.
Segmentation: identifyPrimaryObjects#
identifyPrimaryObjects(pixels, rows, cols, mask, config_json) shares
its exact settings surface with the Python binding
(cp_anywhere::segmentation::build_config/ConfigParams – see
spec/segmentation-modules.md), so the same config_json field names
(threshold_method, unclump_method, watershed_method, …) work
identically in both bindings, eliminating a second, driftable mapping.
Its result (SegmentationResult) is exposed as typed-array/scalar
getters (labels, smallRemovedLabels, objectCount, threshold,
thresholdMap, guideThreshold), not a JSON string: the label arrays
are the same size as the input image, unlike measure()’s much smaller
per-object feature output, so JSON-encoding them would be wasteful.
identifyPrimaryObjectsJson(input) is a JSON-string overload (pixels as
a plain array) kept for parity with measureJson, for debugging/small
inputs without constructing a typed array by hand.
examples/zarr-onnx-duckdb demonstrates this chained into a full
Zarr -> segmentation -> measurement -> Parquet pipeline in a real
browser (see docs/browser-integration.md) – this is the shape a
browser-based consumer needing both segmentation and feature
extraction from cp-anywhere would use, not a theoretical API surface.
Typed-array inputs, output conversion, memory#
Input:
measure(shape, image_names, images, object_names, objects, mask, image_id, families)takes real typed arrays (Uint32Array/Float64Array/Uint8Array) across the wasm-bindgen boundary — the plan’s “typed-array inputs” requirement.wasm-bindgencopies&[T]parameters into wasm linear memory once per call; this is the standard, documented cost of that FFI pattern (seecrates/cp_anywhere_wasm/src/lib.rs), not something hidden.Numeric fixture data is fetched as raw binary, not JSON. The first export attempt used plain JSON and failed immediately:
nonfinite’s pixel array contains literal NaN/Infinity, which are not valid JSON numbers (JSON.parsecorrectly rejects them).scripts/wasm/export_fixtures.pynow writes small JSON metadata (shape, names,image_id) plus.images.f64/.objects.u32/.mask.u8raw binary blobs; the worker fetches each as anArrayBufferand constructs a typed-array view directly over the bytes, which preserves the exact IEEE754 bit pattern and is also the more realistic typed-array transfer path for a browser demo.Output conversion:
measure()returns one JSON string matchingMeasurementResult.to_json()’s shape exactly (same{"nonfinite": "NaN"}-style markers), so the same Python comparator (scripts/parity/compare_cp_measure.py’scompare_feature/compare_relationships) verifies both the native and the WASM result without a second, independently-trusted comparison implementation.Memory:
WebAssembly.Memory.buffer.byteLengthwas 3,538,944 bytes (~3.46 MiB) after running all 13 fixtures sequentially in one worker instance — a post-hoc read, not a tracked peak (see Known limitations).
Worker execution and browser compatibility#
The module runs inside a dedicated Worker({ type: "module" }), loaded via
Vite’s new URL(...) + new Worker(...) pattern, which Vite bundles
correctly (confirmed by the production build above). Testing used real
headless Chromium (Playwright, Chrome for Testing 159.0.8154.6); Firefox
and Safari were not tested in this spike (see Known limitations).
crossOriginIsolated / COOP-COEP were not required for this
single-threaded wasm-bindgen build.
Native vs WASM comparison#
Both the native build (docs/native.md) and this WASM build pass the same
golden fixtures exactly (modulo the same class of float64
reduction/transcendental-order tolerance). Measured per-call timing
(benchmarks/reports/wasm-baseline.json, 3 warmups / 10 runs, same four
representative workloads from spec/benchmark-contract.yaml):
Fixture |
WASM median ms (worker round trip) |
|---|---|
touching_gradient |
0.15 |
real_cell_cc0 |
2.85 |
grid_medium |
11.80 |
grid_large |
174.00 |
Each sample times one measure() call, which includes the in-WASM
serde_json::to_string() encoding of the full 119-column result — a
worker-round-trip cost, not an isolated kernel time. It is not yet
comparable to the native/cp_measure Python baseline
(benchmarks/baseline/initial-cp-measure-timing.json) on an
apples-to-apples kernel-only basis; that isolation, plus a real
cross-target regression gate, is M6’s job. spec/benchmark-contract.yaml’s
browser_worker_budget, browser_memory_budget, and
browser_package_size_budget are updated from “provisional until M4” to
real first-measured numbers with a documented margin.
Known limitations#
Only Chromium was tested (via Playwright). Firefox and Safari are untested; no WASM feature used here (no SIMD, no threads, no SharedArrayBuffer) is Chromium-specific, so cross-browser support is expected but not verified in this spike.
Per-call timing includes JSON-encoding every result; kernel-only timing (e.g., via a numeric-only output mode) is left to M6.
wasm_linear_memory_bytes_after_runis a single post-run read, not a tracked peak across the run.This spike did not test a
crossOriginIsolated/threaded build; the current build is intentionally single-threaded, matching the plan’s “test… browser compatibility” bar without requiring COOP/COEP deployment headers.
Browser integration: Zarr, ONNX Runtime Web, DuckDB-WASM#
What this is#
examples/zarr-onnx-duckdb/ is a real, end-to-end browser integration
proving the pieces named in the plan work together with cp-anywhere’s own
WASM build, with no mocked stage:
zarrita.js reads a real OME-Zarr v2 array (
public/dna.zarr, written byscripts/zarr/export_demo_fixture.py) over HTTP.onnxruntime-web runs a real, checked ONNX graph (
public/threshold.onnx) against that array, producing a reference threshold mask (logged for comparison; not used downstream – see “Why the mask is only a reference” below).cp-anywhere’s own
identifyPrimaryObjects(the same CellProfiler-compatible, oracle-verified segmentation the Python binding exposes – Otsu threshold + Intensity declumping/watershed in this demo) segments the same Zarr pixels into real object labels.cp-anywhere’s
measure()(this project’s own Rust/WASM kernel, the same build verified indocs/wasm.md) measures the Zarr image using cp-anywhere’s own segmentation output as theCellsobject labels – both segmentation and feature extraction now come from cp-anywhere, not just the latter.DuckDB-WASM loads cp-anywhere’s own measurement output into a SQL table and exports it as a real Parquet file (
COPY ... FORMAT PARQUET).
scripts/zarr/validate_export.py then validates that Parquet file with
native DuckDB (not the WASM build under test) and cross-checks every
value – and the segmentation’s own object_count/threshold – against
a fresh, from-source native Python cp_anywhere.identify_primary_objects()
cp_anywhere.measure()call on the identical Zarr bytes – proving the browser pipeline (both its segmentation and its measurement) produced numerically correct output, not just that it ran without an error.
Why the mask is only a reference#
The ONNX threshold model (public/threshold.onnx, a checked
Greater(image, 0.5) -> Cast(..., INT32) graph) still runs in this demo
to exercise the real Zarr -> ONNX -> cp-anywhere -> DuckDB pipeline named
in the plan, and its foreground pixel count is logged for comparison.
Its output is not fed into measurement: main.ts segments with
cp-anywhere’s own identifyPrimaryObjects instead (step 3 above), so the
object labels measurement actually uses come from real CellProfiler-
compatible segmentation (with declumping), not a disconnected-foreground
threshold mask that would merge every touching object into a single
label. A future revision could use the ONNX path for a learned model
(e.g. StarDist) instead of the fixed-threshold demo graph; that remains
independent of cp-anywhere’s own segmentation path either way.
Run it#
uv sync --group demo-assets
uv run --frozen python scripts/zarr/export_demo_fixture.py
cd examples/zarr-onnx-duckdb
npm install
npm run build
npm run browser-check # real Playwright/Chromium run; writes
# cp_anywhere_features.parquet and browser-results.json
cd ../..
uv run --group demo-assets python scripts/zarr/validate_export.py
Verified results#
A real run (Playwright 1.65.1, Chrome for Testing, npm run browser-check)
on the touching_gradient fixture’s DNA channel (12x12, normalized to
[0, 1]):
Zarr read: shape
[12, 12], 144 pixels.ONNX inference (reference only): 72 foreground pixels (
mask > 0.5).cp-anywhere
identifyPrimaryObjects()(Otsu + Intensity declump/watershed,min_diameter=4.0): 1 object, threshold0.486328125.cp-anywhere
measure(): 21object_intensitycolumns for that object.DuckDB-WASM Parquet export: real
PAR1magic bytes at both ends of the file, independently readable by native DuckDB.scripts/zarr/validate_export.py: all 21 feature values, plus the segmentation’s own threshold, match a from-source native Pythoncp_anywhere.identify_primary_objects()+cp_anywhere.measure()run exactly (rtol=1e-6, atol=1e-9) – zero mismatches.
Known limitations#
Network required at run time. DuckDB-WASM fetches its
parquetextension fromextensions.duckdb.orgon the firstLOAD parquet; this demo does not vendor that extension. For a fully offline/CSP-restricted deployment, fetch and servehttps://extensions.duckdb.org/v1.5.4/wasm_mvp/parquet.duckdb_extension.wasmlocally and setcustom_extension_repositorybeforeLOAD parquet(verify the embedded engine version first – it does not always match the npm package version; see the exact version-pinning caveats this was derived from in~/.hermes/cache/scratch/cp-anywhere-browser-integration-research.mdif still present, or re-derive from DuckDB-WASM’s own docs).Single browser tested. Only Chromium (via Playwright) was exercised. No SharedArrayBuffer/COOP/COEP cross-origin isolation was used or is required for this single-threaded WASM MVP path.
2D only. The Zarr stage of this demo uses one 2D channel; this does not affect cp-anywhere’s own 2D/3D coverage (
docs/native.md,docs/wasm.md), which is unrelated and already complete.Not an OME-Zarr multiscale/metadata demo. The written Zarr array has an empty
.zattrs(no OME-NGFF multiscale metadata, no multichannel axes); it is a plain single-chunk array read for its pixel data only.Segmentation scope is this demo’s own fixed choice, not every setting.
main.tshard-codesthreshold_method="otsu",unclump_method="intensity",watershed_method="intensity",min_diameter=4.0;identifyPrimaryObjects’s full settings surface (every threshold method, adaptive scope, Shape/Propagate declump, hole filling, border/size filtering) is documented and independently oracle-verified indocs/segmentation-coverage.mdandspec/segmentation-modules.md, not re-demonstrated here.
M6 performance report#
Headline result#
Fixed a real O(pixels^2) bug in the neighbors kernel, found by profiling
a per-workload throughput regression the plan explicitly warns against
hiding behind an aggregate: before the fix, cp-anywhere was 3.6x slower
than the pinned cp_measure baseline on the grid_large workload (256
objects, 192x192 px), even though every smaller workload was already much
faster. After the fix, all four representative workloads beat the baseline
individually, and WASM round-trip times fell by 11x-33x at the same time.
How the bug was found#
crates/cp_anywhere/src/lib.rs’s “touching” check (used for
PercentTouching in MeasureObjectNeighbors) needs to know, for every
outline pixel of object k, whether any pixel within a small fixed radius
(1.5 px) belongs to a different object. The original implementation
tested that by scanning every pixel in the whole image for each outline
pixel – an accidental O(perimeter_pixels * total_pixels) algorithm when
O(perimeter_pixels) suffices, because the actual predicate only ever
depends on a handful of neighboring offsets.
This was invisible in the declared fixture set’s smallest three workloads
(it is a constant-factor cost there) but dominates at grid_large scale
(36,864 px, 256 objects): a cargo bench run that split the combined
object_intensity_and_neighbors timing by family
(crates/cp_anywhere/benches/throughput.rs’s by_family group)
showed neighbors alone was 151.45 ms on grid_large vs 9.68 ms on
grid_medium – roughly a 15.6x time increase for a 4x object-count / 4x
pixel-count increase, consistent with an O(pixels * objects) or O(pixels^2)
pattern, not the near-linear scaling object_intensity showed over the
same step (0.44 ms -> 2.18 ms, ~5x).
The fix#
Replaced the whole-image scan with a bounded local-neighborhood check using
the same shifted(index, shape, delta) helper the adjacent-relationship
loop just below it already used correctly: a small, fixed Cartesian product
of candidate offsets (9 for 2D, 8 for 3D) covering exactly the same set of
relative positions the original predicate tested, reusing the project’s
existing bounds-checked index arithmetic instead of adding a new primitive.
The predicate itself was not touched – including its asymmetric,
reference-preserving 3D offset range (d == 0 || d == -1, not a symmetric
radius) – only how it is evaluated changed, from O(total pixels) to O(1)
per outline pixel.
Correctness was re-verified immediately after the change, not assumed:
cargo test --manifest-path crates/cp_anywhere/Cargo.toml # pass
uv run --frozen python scripts/parity/compare_native.py # 1440/1440 pass (unchanged)
# rebuild WASM, re-export fixtures, rebuild demo, re-run in Chromium:
uv run --frozen python scripts/parity/compare_wasm.py # 1440/1440 pass (unchanged)
Both parity counts are bit-for-bit identical to before the fix – the change is a pure performance fix, not a semantics change.
Before / after timing#
Native Rust kernel only (cp_anywhere::measure, no Python/FFI,
object_intensity + neighbors families, Criterion, median of 100
samples):
Workload |
Before |
After |
Speedup |
|---|---|---|---|
touching_gradient |
38.05 us |
35.48 us |
1.07x |
real_cell_cc0 |
2.53 ms |
0.195 ms |
13.0x |
grid_medium |
10.16 ms |
0.695 ms |
14.6x |
grid_large |
154.85 ms |
7.354 ms |
21.1x |
Browser (WASM, Playwright/Chromium, median per measure() call including
in-WASM JSON encoding, 10 timed runs):
Workload |
Before |
After |
Speedup |
|---|---|---|---|
touching_gradient |
0.15 ms |
0.20 ms |
~unchanged (near timer resolution) |
real_cell_cc0 |
2.85 ms |
0.70 ms |
4.1x |
grid_medium |
11.80 ms |
2.00 ms |
5.9x |
grid_large |
174.00 ms |
11.85 ms |
14.7x |
Throughput vs the pinned cp_measure baseline#
Per spec/benchmark-contract.yaml’s native_acceptance gate: equal-weight
arithmetic mean of per-workload cp_measure_time / cp-anywhere_time ratios
on the object_intensity + neighbors scope (the only scope cp_measure
implements), minimum 1.0. Computed by
scripts/benchmarks/check_throughput_gate.py, which prints every
per-workload ratio and fails if any workload is below the minimum, so an
aggregate pass cannot hide an individual regression:
Workload |
cp_measure (ms) |
cp-anywhere Rust (ms) |
Ratio |
|---|---|---|---|
touching_gradient |
1.034 |
0.0355 |
29.1x |
real_cell_cc0 |
3.380 |
0.1952 |
17.3x |
grid_medium |
10.916 |
0.6952 |
15.7x |
grid_large |
42.933 |
7.3535 |
5.8x |
Equal-weight mean: 17.0x. Every workload individually clears the 1.0x
minimum (all_workloads_pass_individually: true in
benchmarks/reports/native-throughput-gate.json) – before the fix,
grid_large’s ratio was 0.28x (a genuine regression the aggregate of
~7.5x would have silently hidden had it not been checked per-workload).
Python binding overhead#
Per plan 5.2 (“Python binding overhead: quantified and bounded”), comparing
the installed release wheel’s cp_anywhere.measure() against the bare Rust
kernel on the same inputs (benchmarks/benchmark_native.py vs
scripts/benchmarks/report_native_rust.py):
Workload |
Rust (ms) |
Python (ms) |
Overhead |
|---|---|---|---|
touching_gradient |
0.0355 |
0.0744 |
2.09x |
real_cell_cc0 |
0.1952 |
0.2614 |
1.34x |
grid_medium |
0.6952 |
0.7626 |
1.10x |
grid_large |
7.3535 |
7.7758 |
1.06x |
The fixed per-call cost (roughly 0.04-0.07 ms: Python call dispatch plus
NumPy array validation) shrinks in relative terms as the workload grows,
and stays within the declared <=1.5x budget
(spec/benchmark-contract.yaml’s python_overhead_budget) for every
workload at or above real_cell_cc0 scale. No unnecessary full-array copy
was found in the binding for the already-C-contiguous fixture inputs.
CI regression gate#
scripts/benchmarks/check_regression.py compares a fresh
benchmarks/reports/native-rust-baseline.json against the committed
benchmarks/baseline/native-rust-committed.json, failing only when a
workload is more than 20% slower than committed – never on an
improvement, and never on noise within that margin, per the plan’s
explicit warning against hard-failing on noisy single-run CI timing.
.github/workflows/benchmark.yml runs the full benchmark -> report ->
throughput-gate -> regression-check chain on a weekly schedule and on
manual dispatch (not on every PR, for the same noise reason), uploading
the raw report as a 90-day build artifact for trend-watching.
To intentionally accept a new baseline after a verified, deliberate
change, regenerate benchmarks/reports/native-rust-baseline.json and copy
it over benchmarks/baseline/native-rust-committed.json.
Reproduce#
cargo bench --manifest-path crates/cp_anywhere/Cargo.toml
uv run --frozen python scripts/benchmarks/report_native_rust.py
env -u CONDA_PREFIX uv run maturin develop --release --uv
uv run --frozen python benchmarks/benchmark_native.py
uv run --frozen python scripts/benchmarks/check_throughput_gate.py
uv run --frozen python scripts/benchmarks/check_regression.py
Known limitations#
Single machine, single run per report regeneration (Criterion’s own statistical sampling, 100 samples per workload, is the only within-run repetition; see
benchmark-analysis’s guidance on single-rep measurements – the gaps here are large (5.8x-29x, 21x on the previously-regressed workload), well above the “order of magnitude” bar for a conclusion to survive run-to-run noise).cp_measure’s own timing snapshot (benchmarks/baseline/initial-cp-measure-timing.json) was captured once, in M2, on this same machine; it was not re-measured in M6. Re-running it would only change the comparison if this machine’s absolute performance characteristics changed between M2 and M6, which they did not (same hardware, same session).The regression-gate tolerance (20%) and the schedule (weekly, not per-PR) are both judgment calls aimed at the plan’s explicit warning against noisy single-run CI failures; they have not been tuned against real CI runner variance data, because this session cannot execute a real scheduled GitHub Actions run.
Release process#
What publishes where, and how#
Package |
Registry |
Workflow |
Trigger |
Auth |
|---|---|---|---|---|
|
PyPI |
|
|
OIDC trusted publishing ( |
|
crates.io |
|
|
|
|
npm |
|
|
OIDC trusted publishing ( |
Standalone |
GitHub Release assets |
|
|
|
The GitHub release tag is the only version you set. All three workflows
build the commit attached to that tag. tools/sync_release_version.py
updates the Cargo workspace and lockfile in each build checkout and
checks all three workspace packages before building. Do not bump versions
in source files or commit the generated changes.
For v1.2.3, all three registries get 1.2.3. For prerelease tags such
as v1.2.3-rc.1, crates.io and npm get 1.2.3-rc.1 and PyPI gets
1.2.3rc1. Use -alpha.N, -beta.N, or -rc.N for prereleases.
npm publishes prereleases under next and stable releases under latest.
cp-anywhere is the only crate configured for crates.io publishing. It includes
segmentation, illumination, measurements, and .cppipe execution. To use
the individual operations without pipeline parsing or its serialization
dependencies, set default-features = false in your Cargo dependency.
cp_anywhere_python and cp_anywhere_wasm remain internal binding crates.
The Python and browser packages ship through PyPI and npm.
One-time setup (must be done once, outside this repo, before the first release)#
Each registry has different setup steps for a first release of its package:
GitHub “release” environment. Already created (
repos/wayscience/cp-anywhere/environments/release), referenced by all three publish workflows’environment: release. Optional hardening: add required reviewers under Settings -> Environments -> release.PyPI – zero-touch, even for the first release. PyPI supports a “pending publisher” that creates the project on first use:
Sign in at https://pypi.org/manage/account/publishing/.
Add a pending GitHub publisher: PyPI project name
cp-anywhere, ownerwayscience, repositorycp-anywhere, workflow filenamepublish-pypi.yml, environment namerelease.Nothing else to do – the first tagged release creates the project and publishes to it automatically.
npm – needs one manual publish first. npm’s trusted publishing has no “pending publisher” equivalent: a trusted publisher can only be attached to a package that already exists, so the very first publish cannot go through OIDC.
Use a disposable checkout. Publish a bootstrap prerelease so the GitHub workflow can publish the first stable version. For a first stable release of
v0.1.0, use:python tools/sync_release_version.py v0.1.0-rc.1 python scripts/wasm/build_package.py npm publish ./crates/cp_anywhere_wasm/pkg --access public --tag next
Once
cp-anywhereexists on npm, go to its package settings -> Publishing access -> add a trusted publisher: GitHub Actions, ownerwayscience, repositorycp-anywhere, workflow filenamepublish-npm.yml, environmentrelease.Enable direct
npm publishfor the trusted publisher. Publish the first stable version through GitHub Releases, not by hand.
crates.io – needs a token for the first release. Trusted publishing on crates.io reached general availability in 2025, but like npm, the trusted publisher can only be configured for a crate that already exists – so the first publish needs a classic API token.
Sign in at https://crates.io/settings/tokens, create a token scoped to
publish-new(andpublish-updatefor later releases).Add it as a
CARGO_REGISTRY_TOKENsecret in the GitHubreleaseenvironment (Settings -> Environments -> release -> Environment secrets).publish-crate.ymlalready reads this secret, so no workflow change is needed.Optional, after the crate exists: migrate
publish-crate.ymlfrom the stored token to OIDC (rust-lang/crates-io-auth-action@v1, the modern equivalent of this project’s PyPI/npm setup) and delete theCARGO_REGISTRY_TOKENsecret. Not required – the token-based workflow keeps working for every future release too.
Cutting a release#
Open a PR that updates
CHANGELOG-equivalent notes if desired (this project usesrelease-drafter–.github/workflows/draft-release.ymldrafts release notes automatically from merged PR labels on every push tomain; review and edit the draft before publishing).Publish the drafted GitHub Release with a
vX.Y.Ztag. This singlerelease: publishedevent fans out to all three publish workflows above, plus the.wasm/checksum release-asset upload. The tag is the only place a version is chosen; nothing else needs editing first.Check the three publishing workflows in the Actions tab. After registry setup, no per-release version edits or manual registry uploads are needed.
Local packaging checks#
Build and check the artifacts locally before cutting a release. These commands do not upload anything:
# crates.io: build and verify the archive
cargo package --locked -p cp-anywhere
# PyPI: sdist + wheel build, then install from the wheel into a throwaway
# venv with no project source on sys.path
uv build
python3 -m venv /tmp/smoke && /tmp/smoke/bin/pip install dist/*.whl numpy
/tmp/smoke/bin/python -c "import cp_anywhere; cp_anywhere.measure(...)"
# npm: wasm-pack build, then a full publish dry run
python scripts/wasm/build_package.py
cd crates/cp_anywhere_wasm/pkg && npm publish --dry-run --access public
Use --allow-dirty with cargo package to check uncommitted changes locally.
The release workflow uses it because version sync modifies the checkout
before building.
Known gaps#
The publish workflows themselves have not been exercised by a real GitHub Actions run (that would require actually cutting a release, which this session does not do). Local packaging checks do not verify registry authentication or publishing on GitHub’s runners.
The one-time setup steps above (PyPI pending publisher, npm bootstrap publish + trusted publisher, crates.io token) require the registry account owner’s own login and have not been done yet – they are not something a CI workflow or an automated tool can perform.
M5 implementation coverage dashboard#
Declared feature inventory#
The declared manifest is spec/features.yaml: 119 columns across
MeasureImageIntensity, MeasureObjectIntensity, and same-set adjacent
MeasureObjectNeighbors (full settings and exclusions in
spec/compatibility.md). This is the project’s first release scope, not
every CellProfiler module; expanding it requires its own fixtures and
scope decision, per spec/compatibility.md’s scope decision gate.
Cross-target parity status#
Target |
Fixtures |
Rows checked |
Pass (exact + tolerance) |
Fail |
Report |
|---|---|---|---|---|---|
Native Rust |
13 |
1,560 (119 + 1 rel.) |
1,535 + 25 = 1,560 |
0 |
|
Python (PyO3) |
13 |
1,560 |
1,535 + 25 = 1,560 |
0 |
Same binding as native Rust ( |
WASM (browser) |
13 |
1,560 |
1,533 + 27 = 1,560 |
0 |
|
100% of the 119 declared features pass CellProfiler 4.2.8 parity in
native Rust, Python, and WASM, across all 13 golden fixtures (2D and 3D,
including two independent real-microscopy sources).
No fixture, feature, or target has a silent fallback or omission; any
feature that could not be verified would show as fail_*, not pass_*,
in the reports above. The “tolerance” pass count differs slightly by
target (25 native, 27 WASM) because the WASM run includes two
Neighbors_AngleBetweenNeighbors_Adjacent rows at a transcendental-function
libm last-bit difference (documented in docs/wasm.md); every such row is
still within spec/features.yaml’s declared atol/rtol, not a relaxed
or hidden threshold.
M5 task checklist against the plan#
“Implement remaining features in the declared manifest.” None remained: M3 implemented and passed all 119 declared columns before M4 started (
docs/native.md).“Support documented multichannel, 2D/3D, calibration, and object relationship semantics.” Multichannel:
DNA/Actintwo-channel intensity is in the declared scope and passes on all targets. 2D/3D:volume_3dpasses on native and (as of this update) WASM. Calibration:spec/compatibility.mdexplicitly declares “calibration-driven value conversion” out of scope for this release — there is no undocumented gap here, it is a declared exclusion. Object relationships: the directedNeighborsrelationship table is implemented and checked on every fixture and every target.“Add regression fixtures for each difference found in CellProfiler/
cp_measurecomparisons.” M2’s comparison (benchmarks/baseline/cp_measure-parity.json) found that cp_measure (not CellProfiler) mismatches CellProfiler on sparse-ID positions, theNumberOfNeighborsdtype, and nonfinite propagation. The golden fixtures that exposed those differences —sparse_ids,nonfinite— are exactly the fixtures this project’s own Rust/WASM implementation is checked against, and it passes all of them. No additional fixture was needed because the existing M1 fixture set already covered these cases; this was verified, not assumed.“Update per-feature support and numerical parity reports.” Done:
spec/features.yaml(oracle inventory),spec/cp_measure-map.yaml(cp_measure per-feature status),spec/native-coverage.yaml(native build status), and this dashboard (cross-target status) are all current.
Reproduce#
cargo test --manifest-path crates/cp_anywhere/Cargo.toml
env -u CONDA_PREFIX uv run maturin develop --release --uv
uv run --frozen python scripts/parity/compare_native.py
cd examples/browser-vite && npm install && npm run build
uv run --frozen python ../../scripts/wasm/export_fixtures.py
npm run browser-check
cd ../..
uv run --frozen python scripts/parity/compare_wasm.py
CP_ANYWHERE_REFERENCE_PYTHON=/path/to/reference/venv/bin/python \
PYTHONPATH=/path/to/pinned/CellProfiler \
uv run --frozen pytest
Segmentation coverage dashboard#
Declared scope#
cp_anywhere.identify_primary_objects targets CellProfiler 4.2.8’s
IdentifyPrimaryObjects module (and the Threshold module it shares
with every other segmentation module) as its compatibility oracle.
Full declared scope, including every explicit gap, is in
spec/segmentation-modules.md; this page is the distilled status
dashboard.
Cross-target verification status#
Both native Rust/Python and the WASM (browser) build pass all 100
fixtures exactly (see docs/wasm.md for the WASM-specific run and its
one documented tolerance exception, identical to the one below).
Tier |
Status |
|---|---|
Global threshold: Manual, Otsu (two- and three-class), Li, Robust Background |
Covered |
Adaptive (local) threshold: block-grid + B-spline interpolation, Sauvola |
Covered |
|
Covered |
|
Covered |
Border exclusion, size/diameter filtering, object-count limiting |
Covered |
Hole filling ( |
Covered |
Declumping ( |
Covered |
Watershed separation ( |
Covered |
Manual smoothing-filter-size / maxima-suppression-size overrides |
Covered |
|
Not implemented (explicit gap) |
|
Not implemented (explicit gap) |
|
Not implemented (explicit gap) |
3D/volumetric segmentation, |
Out of scope for this release |
Every “Covered” row above is independently verified against the real,
pinned CellProfiler 4.2.8 oracle — not assumed correct by code
inspection. tests/segmentation/test_reference_alignment.py replays
scripts/segmentation/generate_reference.py’s real-CellProfiler output
against cp_anywhere.identify_primary_objects on 100 fixture cases
(fixtures/segmentation/), spanning synthetic edge cases and two
independent real microscopy datasets (BBBC007, real_cell_cc0). No
fixture has a silent fallback or relaxed tolerance beyond the one
documented, narrow, label-safe exception below.
One known tolerance exception: a single adversarial fixture (adaptive
scope, three-class Otsu, log-transformed, a deliberately tiny 4-pixel
window) can land threshold_map values up to ~0.012 off the oracle,
from a float32-vs-float64 pixel-precision difference that a full
pipeline rewrite was tried and reverted for (see
spec/segmentation-modules.md’s “Adaptive (local) thresholding”
section). Object labels are unaffected in every case.
Chaining into measurement#
seg = cp_anywhere.identify_primary_objects(dna_image, threshold="otsu")
result = cp_anywhere.measure({"DNA": dna_image}, {"Cells": seg.labels})
Reproduce#
env -u CONDA_PREFIX uv run maturin develop --release --uv
CP_ANYWHERE_REFERENCE_PYTHON=/path/to/reference/venv/bin/python \
PYTHONPATH=/path/to/pinned/CellProfiler \
uv run --frozen pytest tests/segmentation
# regenerate fixtures from the real oracle (after a reference-semantics change)
CP_ANYWHERE_REFERENCE_PYTHON=/path/to/reference/venv/bin/python \
PYTHONPATH=/path/to/pinned/CellProfiler \
/path/to/reference/venv/bin/python scripts/segmentation/generate_reference.py
Size and shape measurements#
MeasureObjectSizeShape is available for 2D label arrays in native Python
and WASM. It uses the pinned CellProfiler 4.2.8 reference, with
scikit-image 0.21.0 and centrosome 1.2.3.
from cp_anywhere import measure
result = measure(
{"DNA": dna_image},
{"Cells": cell_labels},
families=["object_size_shape"],
size_shape_calculate_zernikes=True,
size_shape_calculate_advanced=False,
)
The family measures each supplied object set, without using image intensity
or the image mask. Images are still required by the shared measure API.
The original three default families are unchanged: request
object_size_shape explicitly to add shape columns.
Columns and settings#
All columns have the AreaShape_ prefix. The module supplies:
25 standard columns: area, perimeter, axes, eccentricity, orientation, center coordinates, bounding boxes, form factor, extent, solidity, compactness, Euler number, radii, convex area, Feret diameters, and equivalent diameter.
30 Zernike columns, orders 0 through 9. Enabled by default, matching CellProfiler’s
calculate_zernikessetting.53 advanced columns: spatial, central and normalized moments, Hu moments, the inertia tensor and its eigenvalues. Disabled by default, matching CellProfiler’s
calculate_advancedsetting.
The two flags select 25, 55, 78, or 108 total columns. Undefined values remain NaN or infinity, rather than becoming zero. The normal result serialization and export methods retain their existing nonfinite-value rules. An empty object set still has the selected schema with empty arrays.
The .cppipe importer accepts revision 3. Its three settings are the
object list, Zernike flag, and advanced flag. Each imported module keeps
its own settings and object selection, even when a pipeline measures
Nuclei, Cells, and Cytoplasm with different flags.
Limits#
3D size/shape measurements are not implemented. A 3D request raises a clear error rather than returning 2D substitutes.
The public measurement API requires consecutive object IDs starting at
The real reference produces inconsistently packed columns for sparse labels. The low-level Rust routines preserve that behavior for numerical comparison, but the integrated API rejects it to prevent corrupt row alignment in exports. Other measurement families keep their existing sparse-label support.
Numerical checks target the pinned reference environment. Advanced moment reductions reproduce its NumPy/OpenBLAS ARMv8 order; this is not a claim of bitwise agreement with every BLAS implementation.
Verification#
Real reference fixtures cover rectangles, rings, diagonal shapes, nested objects, multiple objects, border objects, single pixels, and empty images. Tests compare every selected column under all four flag combinations. Separate real pipelines run primary, secondary, and tertiary segmentation before size/shape measurement, with different flags for each compartment.
The browser check runs both measureJson and measureWithOptions in real
Playwright Chromium, not a simulated WASM runtime. The original
eight-argument measure export remains available for existing callers;
measureWithOptions adds diagnostic neighbor layers and shape flags.
To regenerate and check the reference data:
PYTHONPATH=/path/to/pinned/CellProfiler /path/to/reference/python \
scripts/reference/generate_size_shape.py
PYTHONPATH=/path/to/pinned/CellProfiler /path/to/reference/python \
scripts/cppipe/generate_size_shape_pipeline.py
uv run --frozen pytest tests/test_size_shape.py tests/cppipe/test_size_shape_pipeline.py
python scripts/wasm/build_package.py
uv run --frozen python scripts/parity/compare_wasm_size_shape.py
The browser check requires the browser example’s existing Node dependencies and Playwright Chromium installation. Building a package does not publish it.
Granularity#
MeasureGranularity is available for 2D images, in native Python and
WASM. It uses the pinned CellProfiler 4.2.8 reference, with scipy 1.9.3
and scikit-image 0.21.0.
from cp_anywhere import measure_granularity
result = measure_granularity(
dna_image,
objects={"Cells": cell_labels},
subsample_size=0.25, # CellProfiler's own defaults
image_sample_size=0.25,
element_size=10,
granular_spectrum_length=16,
)
result.image[0] # GS1, the image-level granularity spectrum
result.objects["Cells"][0] # GS1, one value per Cells object
measure_granularity is a standalone function, separate from the
unified measure() family dispatch: CellProfiler’s own module measures
one image at a time, with its own subsampling/structuring-element
settings, and its result shape (per-scale spectra, not a flat feature
table) does not fit the shared measure() contract.
Algorithm#
For each image: optionally downsample for the main granularity
computation (subsample_size) and separately for background removal
(image_sample_size), using CellProfiler’s own mix of bilinear
(order 1) pixel resampling and cubic B-spline (order 3) mask
resampling. Subtract a grayscale-opened background (erosion then
dilation with a disk structuring element of element_size radius),
then repeatedly erode with a radius-1 disk and reconstruct by
dilation, recording the percent change in masked mean brightness at
each of granular_spectrum_length scales.
Per-object spectra use the same per-scale reconstructed image, rescaled back to full resolution, averaged over each object’s unmasked pixels at full resolution – not the (possibly subsampled) working image directly.
An object with zero pixels under the mask reports NaN at every scale,
matching real CellProfiler: its own baseline mean is already NaN,
and NaN never recovers through the per-scale subtraction.
Limits#
2D only. Real CellProfiler’s own module divides by
orig_shape[dimension] - 1while rescaling its working image back to full resolution, and raisesZeroDivisionError, unconditionally, for any image with a 1-row or 1-column dimension – confirmed live against the pinned reference.measure_granularityraises a clearValueErrorfor that case instead of silently producing 0/NaN; there is no real oracle value to match for it.imagemust already be at the same precision real CellProfiler would have measured.Image(..., convert=True)rescales integer dtypes by their dtype’s max value (identify_primary_objectsreproduces this automatically for segmentation inputs, butmeasure_granularitydoes not – intensity measurements expect already-scaled float input, the same contractmeasure()’s other intensity families use) and always downcasts throughfloat32even for float64 input. The second effect is usually negligible (~1e-7 relative) but is a real precision difference, not a cp-anywhere rounding bug, when reproducing an exact oracle run numerically.The
.cppipeimporter accepts revision 4 (7 settings: images, “measure within objects?”, objects, both subsampling factors, element size, spectrum length). Every listed image is measured independently and merged into the sameMeasurementResult, matching real CellProfiler’sGranularity_<scale>_<image name>feature-naming convention.
Verification#
Real reference fixtures cover CellProfiler’s own defaults, full
resolution (no subsampling), uneven subsample/background factors,
partial and fully-empty masks, zero/constant/negative images, and
multiple simultaneous object sets – 12 cases total, generated by
scripts/reference/generate_granularity.py against a live pinned
CellProfiler run. A separate real two-image, two-object-set .cppipe
pipeline (scripts/cppipe/generate_granularity_pipeline.py) exercises
the full IdentifyPrimaryObjects -> IdentifySecondaryObjects ->
IdentifyTertiaryObjects -> MeasureGranularity chain end to end.
The browser check runs measureGranularityJson in real Playwright
Chromium, not a simulated WASM runtime.
To regenerate and check the reference data:
PYTHONPATH=/path/to/pinned/CellProfiler /path/to/reference/python \
scripts/reference/generate_granularity.py
PYTHONPATH=/path/to/pinned/CellProfiler /path/to/reference/python \
scripts/cppipe/generate_granularity_pipeline.py
uv run --frozen pytest tests/test_granularity.py tests/cppipe/test_granularity_pipeline.py
python scripts/wasm/build_package.py
uv run --frozen python scripts/parity/compare_wasm_granularity.py
The browser check requires the browser example’s existing Node dependencies and Playwright Chromium installation.
CellProfiler pipeline (.cppipe) import#
cp_anywhere.load_pipeline(path) reads a real CellProfiler .cppipe
file and translates the settings of every module this project
implements (IdentifyPrimaryObjects, IdentifySecondaryObjects,
IdentifyTertiaryObjects,
MeasureImageIntensity, MeasureObjectIntensity, MeasureObjectSizeShape,
MeasureGranularity, MeasureObjectIntensityDistribution,
MeasureTexture, MeasureColocalization,
MeasureObjectNeighbors) into cp_anywhere’s own configuration – an
existing pipeline’s settings for those modules carry over 1-for-1,
with no manual re-entry.
import cp_anywhere
pipeline = cp_anywhere.load_pipeline("my_pipeline.cppipe")
segmentations, derived_objects, measurement = pipeline.run(
{"DNA": dna_image, "Actin": actin_image}
)
The second result contains both secondary and tertiary object sets.
For example, derived_objects["Cytoplasm"].labels contains the tertiary
labels. The return value remains a three-item tuple.
This is a settings importer, not a full pipeline execution engine:
file-matching/organization modules (Images, Metadata,
NamesAndTypes, Groups, …) are recognized and skipped –
pipeline.skipped_modules lists them – not executed; you supply
already-loaded image arrays by name, same as cp_anywhere.measure.
Every out-of-scope setting (an older pipeline revision, “basic mode”,
a non-"Adjacent" MeasureObjectNeighbors distance method, an
unimplemented IdentifySecondaryObjects method, and a handful more)
raises cp_anywhere.CppipeError naming the exact setting, rather than
silently mis-translating it – see spec/cppipe-compatibility.md for
the full settings-mapping table, every explicit gap, and how it is
verified (real, oracle-generated .cppipe fixtures in
fixtures/cppipe/, checked in tests/cppipe/test_cppipe_import.py
against the pinned CellProfiler 4.2.8 oracle’s own segmentation output
on the same test image). The CLI also accepts a pipeline directly:
cp-anywhere --pipeline my_pipeline.cppipe --image DNA=dna.tif --output result.json
Database export#
MeasurementResult can be written directly to a real Parquet file, a
DuckDB database, or a SQLite database (using STRICT tables), on top
of the existing to_csv/to_arrow/to_json:
result.to_parquet("result.parquet") # same long-form rows as to_arrow
result.to_duckdb("results.duckdb") # wide Per_Image / Per_Cells / ...
result.to_sqlite("results.sqlite") # same, as SQLite STRICT tables
to_duckdb/to_sqlite write wide tables named Per_Image/
Per_<ObjectName> – matching real CellProfiler’s own
ExportToDatabase naming convention – and default to
if_exists="append", so processing many images in a loop and calling
one of these once per image builds up a single combined dataset, the
way CellProfiler’s own batch runs do. Pass if_exists="replace" to
start fresh instead.
to_duckdb needs the optional duckdb dependency
(pip install cp-anywhere[duckdb]); to_parquet and to_sqlite need
nothing beyond this project’s own core dependencies.
For a dense, fixed-size feature array per object instead of long-form
rows (e.g. to load straight into a (n_rows, n_features) numpy
array), use to_parquet(..., mode="array"): path becomes a
directory, one Per_Image.parquet/Per_<ObjectName>.parquet file is
written per scope, and each file’s features column is a single
fixed-size array per row. The exact ordered feature names for that
array are embedded in the file’s own Parquet schema metadata (no
sidecar file to go out of sync) – read them back with
cp_anywhere.read_parquet_feature_names(path). The CLI exposes this
as --output <directory> --parquet-mode array.
to_parquet(..., mode="wide") writes the same per-scope files as
mode="array", but as the traditional tabular shape instead – one
real, named column per feature, no array/metadata lookup needed
(--output <directory> --parquet-mode wide).
For one combined, CytoTable
-style table across every object scope – one row per object, scopes
joined on object_id, feature columns prefixed by scope name – use
to_parquet(..., mode="joined"). path is either a single .parquet
file or a directory, in which case it’s written as a real,
max_rows_per_file-paginated Parquet dataset via
pyarrow.dataset.write_dataset that can accumulate across many images
with if_exists="append" (the default): --output <file-or-directory> --parquet-mode joined --join outer|inner --max-rows-per-file N.
The CLI dispatches on --output’s file extension the same way:
cp-anywhere --image DNA=dna.tif --objects Cells=labels.npy \
--output results.sqlite --if-exists append
See spec/database-export.md for the exact schema (column names and
types), why SQLite specifically needs STRICT tables, and how it’s
verified (tests/test_database_export.py).
CI/CD#
On every pull request (.github/workflows/run-tests.yml)#
pre_commit_checks: all pre-commit hooks (ruff, cargo fmt/clippy via the Rust hooks below, yamllint, codespell, mdformat, zizmor, vulture,ty). Syncs everyuvdependency group (--all-groups) sotycan resolve the demo-onlyonnx/duckdbimports inscripts/zarr/*.py.run_tests(matrix: Python 3.11-3.15 x Ubuntu/macOS):cargo fmt --check,cargo clippy -D warnings,cargo test,maturin develop+pytest, andscripts/parity/compare_native.py(golden-fixture parity; does not need the live CellProfiler oracle, so it runs on every PR across the full OS/Python matrix).wasm_build_and_browser_check: buildscrates/cp_anywhere_wasmwithwasm-pack, builds theexamples/browser-viteVite demo, runs it in a real headless Chromium via Playwright (npm run browser-check), and checks the captured output withscripts/parity/compare_wasm.py– a real WASM build and browser smoke test on every PR, not a native-only check.
Nightly (.github/workflows/nightly-full-parity.yml)#
Installs the pinned CellProfiler 4.2.8 oracle (reference/install.sh,
which needs a JDK for python-javabridge – too heavy to run on every
PR) and re-runs the full suite against it: pytest with the oracle
configured, compare_native.py, and a full WASM build + browser check +
compare_wasm.py. This is the plan’s “full feature inventory parity
across native/Python/WASM” nightly check. Also runnable on demand
(workflow_dispatch).
Weekly (.github/workflows/benchmark.yml)#
Native Rust throughput benchmark (Criterion) against the committed
baseline, with a tolerance-based regression gate
(scripts/benchmarks/check_regression.py) – see docs/performance.md
for why this runs on a schedule rather than every PR (CI runner timing
noise).
On release (.github/workflows/publish-*.yml)#
See docs/release.md for the full publish pipeline (PyPI, crates.io,
npm, GitHub Release assets).
Known gaps#
Property-based tests for invariants such as label permutation and object ordering (plan section 8) are not implemented. The current test suite checks golden-fixture correctness, not permutation-invariance properties; this would be a reasonable follow-up using
proptest(Rust) orhypothesis(Python).None of the new CI workflows added in M7 (
wasm_build_and_browser_check,nightly-full-parity.yml) have been exercised by a real GitHub Actions run in this session – only validated locally (the exact same commands each step runs were run directly in this session and passed) and statically (actionlint/zizmorvia pre-commit).