Regression Suite

The repository carries a versioned regression suite in regression/. Each entry pairs a set of observations — the 2D–3D target correspondences, robot poses and image sizes a calibration consumes, extracted once from the images — with the reference result calibration-rs produces from them. Every change to the library is checked against these results, and because the observations are plain JSON, other calibration tools can run on exactly the same data.

Entries

The real-image entries come from their publishers:

• kuka_1 is dataset kuka_1 of RWHE-Calib, from I. Ali, O. Suominen, A. Gotchev and E. Ruiz Morales, Methods for Simultaneous Robot-World-Hand–Eye Calibration: A Comparative Study, Sensors 19(12), 2837 (2019), https://doi.org/10.3390/s19122837. The images and robot poses are published at https://github.com/ihtishamaliktk/RWHE-Calib as open-access data; the authors state no licence and ask users to cite the paper.
• The stereo entries use Chessboard Pictures for Stereocamera Calibration by Daniel (Kaggle user danielwe14), version 3, https://www.kaggle.com/datasets/danielwe14/stereocamera-chessboard-pictures, which states no licence.
• ds8 is dataset 8 of A. Tabb and K. M. Ahmad Yousef, Solving the robot-world hand-eye(s) calibration problem with iterative methods, Machine Vision and Applications 28 (2017); the images are published at https://doi.org/10.15482/USDA.ADC/1340592 in the U.S. public domain.
• opencv_stereo is the stereo calibration sample of OpenCV (samples/data/left*.jpg, right*.jpg in https://github.com/opencv/opencv), under OpenCV's Apache-2.0 licence. OpenCV does not publish the printed square size, so the board is declared with squares of 1.
• The fisheye entries use calib-3_stereo_from_JY, the stereo fisheye set of OpenCV's own fisheye calibration tests, from OpenCV's test data repository https://github.com/opencv/opencv_extra (testdata/cv/cameracalibration/fisheye/), which states no licence for it. The set ships the chessboard corners with the images, and the entries are frozen from those corners, so calibration-rs and OpenCV calibrate the same points (see Fisheye Calibration: OpenCV's Test Set).

regression/suite.toml records each entry's licence (an SPDX identifier, NOASSERTION where the publisher states none), citation and source.

The synthetic entries come with their ground truth, so they check accuracy as well as stability, and the laser entries are the suite's laser coverage. Their data was generated once with fixed seeds and is frozen like the rest: a change to the generator cannot change it.

Layout

regression/
├── suite.toml                          entries: description, licence, config
├── SHA256SUMS                          checksums of the observation and truth files
├── observations/<id>.json              the entry's observations
├── truth/<id>.json                     a synthetic entry's true parameters
├── results/<id>/calibration-rs.json    the entry's reference result
├── results/<id>/<tool>.json            another library's result, for comparison
├── fetch/<id>.sha256                   the files `suite fetch` downloads, with checksums
└── schema/                             JSON Schemas of the three file formats

SHA256SUMS is in the format of sha256sum, so the data can be verified without any of our tools:

cd regression && sha256sum -c SHA256SUMS      # shasum -a 256 -c on macOS

Running it

cargo run --release -p vision-calibration-bench --bin calib-bench -- suite check

suite check verifies each observation file against SHA256SUMS (a missing or mismatched checksum fails the entry; it is never skipped), calibrates the entry with its config and compares every number in the result with the reference. Each entry reports PASS or FAIL with the values that moved, and the largest change of an estimated parameter in units of the reference's standard deviation (see Parameter Uncertainty); the command exits non-zero when any entry fails. --only <id> selects entries.

--sigma <k> compares only the estimated parameters, each within k standard deviations of the reference, for results that may legitimately differ in their last digits. --backend factrs (or tiny_solver) runs every solver stage on that backend:

calib-bench suite check --backend factrs --sigma 1

Two more subcommands maintain the suite:

• suite bless recalibrates the selected entries and overwrites their reference results. Run it when a change is meant to move the results, and review the diff of results/ like any other code change.
• suite freeze re-extracts an entry's observations from the images its dataset manifest names (built with --features tier-b), reads them from the corner files the publisher released, or regenerates a synthetic entry's observations and truth from its scene, and rewrites SHA256SUMS. Observations change only when the detector does. The images are read from the manifest's folder, or with --images <dir> from the cache suite fetch fills (below); corner files only from that cache.

Fetching the images

The observations are all suite check needs. To re-extract them, or to run another tool on the images themselves, suite fetch downloads each entry's images from its publisher and checks every file against the entry's fetch/<id>.sha256:

cargo run --release -p vision-calibration-bench --features tier-b \
    --bin calib-bench -- suite fetch
cargo run --release -p vision-calibration-bench --features tier-b \
    --bin calib-bench -- suite freeze --images target/suite-images

The files land in target/suite-images/<id>/ (--images <dir> picks another folder), laid out as the dataset manifest expects. A file already there that matches its checksum is not downloaded again, so the command is cheap to repeat; a downloaded file that does not match fails the entry, as it means the publisher's files changed. suite freeze --images <dir> verifies the cached files again, freezes the entries suite fetch downloads (or those named with --only) and rewrites SHA256SUMS. On Apple-silicon macOS, where the observations are frozen, a clean checkout reproduces them byte for byte. Corner detection is not bit-identical across platforms: on x86-64 Linux the same images give corners up to about 0.001 px apart, so the files differ there while every view and corner is the same.

suite.toml gives each fetchable entry a fetch source: files, each file downloaded from a base_url followed by its path, or zip, one archive whose root folder is the dataset's. An entry frozen from published corners names them under corners: the file format (opencv_storage, the cv::FileStorage XML of OpenCV's calibration tests, one node image_<i> per view), the problem (planar_intrinsics or rig_extrinsics), the object file of target points and, per camera, its points file and image_size, which corner files do not record. A scheduled CI job fetches the images from the publishers every week, re-freezes these entries and fails when the observations no longer match SHA256SUMS.

The image entries are also the public datasets of the benchmark commands that run detection: calib-bench run --dataset <id> and calib-bench feature-cov read each entry's dataset manifest, and --images <dir> reads the fetchable entries from the suite fetch cache.

The observation format

{
  "version": 1,
  "cameras": [{ "id": "cam0", "image_size": [1928, 1208] }],
  "problem": "single_cam_handeye",
  "input": {
    "views": [
      {
        "obs": {
          "points_3d": [[0.0, 0.0, 0.0], [0.02, 0.0, 0.0]],
          "points_2d": [[109.811, 92.798], [140.950, 91.702]]
        },
        "meta": {
          "base_se3_gripper": {
            "rotation": [0.0, 0.0, 0.0, 1.0],
            "translation": [1.205, -0.808, 0.524]
          }
        }
      }
    ]
  }
}

• problem names the problem type and input is its input, the same payload CalibrationSession::set_input takes in Rust and the run_* functions take in Python (see Data Types Quick Reference). Rig inputs hold num_cameras and, per view, one observation per camera in cameras order, null where that camera did not see the target.
• cameras lists the cameras in input order with their image size [width, height] in pixels.
• points_3d are target points in metres in the target frame; planar targets lie in z = 0. points_2d are the matching image points in pixels, with integer coordinates at pixel centres (the OpenCV convention).
• A pose is {"rotation": [qx, qy, qz, qw], "translation": [x, y, z]}: a unit quaternion, scalar last, and a translation in metres. a_se3_b maps coordinates in frame b to frame a, so base_se3_gripper takes gripper coordinates to robot-base coordinates — OpenCV's R_gripper2base, t_gripper2base.
• version is the format version; it changes only with an incompatible format change.

schema/observation_set.json is the format's JSON Schema, generated from the types the library reads, with the input of every problem type; truth.json and external_result.json describe the truth files and the results of other libraries below.

Ground truth

truth/<id>.json names each true value by the label the result's covariance uses (see Parameter Uncertainty):

{
  "parameters": { "fx": 1000.0, "fy": 990.0, "k1": -0.12, "plane.nx": 0.287 },
  "poses": {
    "gripper_se3_camera": { "rotation": [0.05, -0.02, 0.01, 0.998], "translation": [0.05, -0.03, 0.08] }
  }
}

Scalars sit under parameters: intrinsics, distortion coefficients, the sensor tilt and laser-plane components, prefixed cam<i>. in a rig with more than one camera. Poses sit under poses by their export names. suite check compares every estimated parameter the truth names with its true value, in units of the result's own standard deviation (a pose by the right perturbation T_true⁻¹ · T), and fails the entry when one is 5σ or more off. Each entry reports its largest error against the truth.

Comparing with other libraries

The reference results are the problem's export, minus the per-feature residual lists and the projected laser lines (they follow from the parameters and the observations; the residual histograms and the reprojection statistics stay). When comparing them with another tool's output:

• Intrinsics k holds fx, fy, cx and cy in pixels, in the same pixel convention as the observations.
• Distortion dist holds the Brown-Conrady coefficients by name; OpenCV's five-element distCoeffs is (k1, k2, p1, p2, k3) of the same model. Most configs fix k3 = 0. A fisheye camera's KannalaBrandt4 coefficients k1 … k4 are the four elements of cv::fisheye's D.
• Reprojection error: mean_reproj_error and per_cam_reproj_errors are the mean Euclidean distance in pixels between observed and reprojected points. OpenCV's calibrateCamera returns the root-mean-square distance, which is larger for the same fit.
• Poses follow the a_se3_b convention above.
• Scheimpflug tilt: tilt_x, tilt_y are OpenCV's τx, τy of CALIB_TILTED_MODEL, with the same sign and the same place in the projection (after distortion, before K).

OpenCV

tools/compare/opencv/compare.py calibrates every entry OpenCV has a method for and writes results/<id>/opencv.json: the parameters under the labels of our results, the poses under our export names, OpenCV's standard deviations and its mean reprojection error. The OpenCV version is pinned in the harness's uv project:

uv run --project tools/compare/opencv tools/compare/opencv/compare.py
cargo run --release -p vision-calibration-bench --bin calib-bench -- suite compare

OpenCV has no laser-plane calibration, so the laser entries have no comparison. suite compare sets each result against our reference: differences in units of our standard deviation, OpenCV's standard deviations against ours, the mean reprojection error by the same definition, and for a synthetic entry each library's largest error against the truth, in units of our σ:

Reading it:

• On plain and Scheimpflug intrinsics the two libraries agree within 0.02σ. Both start from the classical covariance; ours is scaled by its view scatter when the views scatter significantly more than the noise predicts, so on the real stereo cameras our standard deviations are 1.9–2.2 times OpenCV's. On synth_planar they are 1.6 times: its views happen to pass that test (about one calibration in forty does on data that fit the noise model exactly). The Scheimpflug fit needs the same parameter set on both sides: with the tangential terms p1, p2 free, a tilt and a decentring distortion explain the same images, and OpenCV's fit wanders along that valley (its own σ of cy is 40 px).
• Hand-eye results differ by design. Our hand-eye workflows refine the robot poses within their priors, so on kuka_1 the reprojection error is the corner noise (0.13 px) rather than the robot's pose error (2.3 px with the measured poses). OpenCV takes the robot poses as exact for the hand-eye pose, but calibrates the intrinsics with a free pose per view, as the refined robot poses effectively are here: our intrinsics standard deviations are 1.5–1.7 times OpenCV's, the view scatter's scale. The synthetic hand-eye entry's robot poses are exact, and our solve estimates a robot error near zero from the corrections: it lands 1.2σ from the truth with standard deviations a third to two thirds of OpenCV's, whose hand-eye, from the same poses, is 10σ from the truth in our units.
• Rig results refine each camera's intrinsics jointly with the rig, from every view of every camera. On stereo_rig OpenCV's stereoCalibrate refines them too, but from the views both cameras see; the two estimates differ by up to 4σ, along the coupling between camera 0's focal length and camera 1's depth offset. On the synthetic rig the joint estimate lands within 2.1σ of the truth.
• ds8 is a three-camera rig; OpenCV calibrates each camera's intrinsics alone and camera 0's hand-eye alone, while ours refines all three cameras with the rig and the hand-eye, so the difference includes what the joint solve moves.
• On OpenCV's fisheye set the two libraries agree within 0.01σ, on each camera alone and on the stereo pair, where both refine the intrinsics with the rig; OpenCV's results are the values its own tests expect. cv::fisheye reports no standard deviations, so there is no σ ratio.

Drift tolerance

A number passes when it is within abs + rel · max(|reference|, |result|) of its reference, with rel = 1e-6 and abs = 1e-9 unless the entry sets its own tolerance. That flags any change in the solution while allowing the last-bit differences between platforms and compilers. Strings, flags and the shape of the result must match exactly.

With --sigma k, a parameter passes when its change is within k of the reference's standard deviations. A pose's change is measured as the right perturbation T_ref⁻¹ · T in the coordinates of its covariance. The two solver backends agree within 0.02σ on every entry.

Adding an entry

A synthetic entry names a scene of the solver benchmark's full matrix under synthetic, by its id <problem>/<camera>/<scale>/n<σ>/<clean|huber|cauchy> (as calib-bench solver run --preset full reports them), with the config the scene needs, and carries the CC0 licence. Run suite freeze --only <id> and suite bless --only <id> and commit the observation, truth and result files with SHA256SUMS. For an entry with images:

1. Check that the data's licence allows publishing observations derived from it, and record the licence (and citation, if the authors ask for one) in the entry.
2. Write a dataset manifest (dataset.toml, the file the desktop app's Run workspace reads; data/kuka_1/dataset.toml is an example) and an [[entry]] in suite.toml that points manifest at it.
3. Run suite freeze --only <id>, then suite bless --only <id>, and commit the observation file, SHA256SUMS and the reference result.
4. If the publisher serves the files without a login, give the entry a fetch source and list every file the manifest reads, with its SHA-256, in fetch/<id>.sha256 (the sha256sum format, paths relative to the manifest's folder). suite fetch --only <id> and suite freeze --only <id> --images target/suite-images must then reproduce the observation file.

When the publisher releases the target corners with the images, as OpenCV does for its test sets, the entry can name them under corners instead of a dataset manifest; it then needs a fetch source whose list holds the corner files.