Introduction

calibration-rs is a Rust library for camera calibration — the process of estimating the internal parameters (intrinsics, lens distortion) and external parameters (pose, rig geometry, hand-eye transforms) of camera systems from observed correspondences between known 3D points and their 2D projections.

API reference

Three Ways In

• The desktop app runs a calibration from a folder of images and shows the result, its residuals and its uncertainty, without code (The Desktop App).
• Python: pip install vision-calibration, one call per workflow (Getting Started).
• Rust: the vision-calibration crate, with the same workflows plus step-by-step sessions (Getting Started).

Not sure which workflow fits your hardware? Choose Your Workflow maps your setup to a workflow and a camera model and says what to capture.

Who This Book Is For

Engineers and researchers in machine vision, robotics and 3D reconstruction who want to:

• calibrate single cameras, stereo rigs or multi-camera systems;
• perform hand-eye calibration for cameras mounted on robot arms;
• calibrate laser triangulation devices (camera + laser plane);
• understand the mathematics behind calibration algorithms.

Running a calibration needs no mathematical background. The theory chapters assume familiarity with linear algebra (SVD, eigendecomposition, least squares) and basic projective geometry (homogeneous coordinates, projection matrices), and introduce Lie group theory (SO(3), SE(3)) when needed.

What calibration-rs Provides

The library covers the full calibration pipeline:

1. Linear initialization — closed-form solvers (Zhang's method, DLT, PnP, Tsai-Lenz hand-eye, epipolar geometry) that produce approximate parameter estimates
2. Non-linear refinement — Levenberg-Marquardt bundle adjustment that minimizes reprojection error to sub-pixel accuracy
3. Session framework — a high-level API with step functions, configuration, and JSON checkpointing
4. Data handling and diagnostics — target detectors with a detection cache, a dataset manifest, per-feature residuals and parameter uncertainty for every result

Workspace Structure

calibration-rs is organized as a layered Rust workspace with a layered architecture:

vision-calibration (facade)
    → vision-calibration-pipeline (sessions, workflows, dataset runner)
        → vision-calibration-optim (non-linear refinement)
        → vision-calibration-linear (initialization)
        → vision-calibration-dataset / -detect (manifest, detectors)
    → vision-mvg (N-view geometry, rectification, dense stereo)
        → vision-geometry (two-view solvers)
            → vision-calibration-core (primitives, camera models, RANSAC)

The key dependency rule: the linear and optimization crates are peers — they both depend on vision-calibration-core but not on each other. This separation keeps initialization algorithms independent of the optimization backend.

Relation to OpenCV

Readers familiar with OpenCV's calibration module will find analogous functionality throughout:

calibration-rs differs from OpenCV in several ways: it is written in pure Rust, uses a composable camera model with generic type parameters, provides a backend-agnostic optimization IR, and offers a session framework with JSON checkpointing for production workflows.

Book Organization

The book is the one user guide for the library. It is organized by what you want to do first and by the mathematics behind it last:

• Choose Your Workflow: from your hardware to a workflow, a camera model and a capture plan.
• Getting Started: install the library and run a first calibration in Rust and Python, then a stereo rig on real data.
• Calibration Workflows: one chapter per problem type (planar intrinsics, hand-eye, rigs, laser devices), plus choosing a distortion model, seeding the initialization with prior knowledge and reading per-feature residuals.
• The Session Framework: the CalibrationSession API behind every workflow.
• Data and Detection: calibration targets and their detectors, the dataset manifest that describes a folder of images, laser datasets and the generated manifest field reference.
• Uncertainty: the covariance every result carries, how to read it, and the projection uncertainty of a camera.
• Multiple-View Geometry: pose recovery, triangulation, bundle adjustment, rectification and dense stereo for calibrated cameras.
• The Desktop App: running calibrations and inspecting the results without writing code.
• Theory: the camera model, geometric primitives and robust estimation, the linear initialization algorithms and the non-linear optimization machinery, each with a formal problem statement and derivation.
• Reference and Development: the regression suite, extending the library, synthetic data and a type reference.

Each algorithm chapter in the theory parts includes a formal problem statement, objective function, assumptions, and a full derivation leading to the implementation.

The chapters show Rust unless a section says otherwise; Getting Started puts Rust and Python side by side.

Choose Your Workflow

Three questions settle how to calibrate a system: how you want to run it, which workflow matches your hardware, and which camera model matches your lenses. This chapter answers them in that order and ends with what to capture.

1. How you run it

All three run the same solvers and produce the same JSON export, so a result from one opens in the others.

2. Which workflow

The workflow follows from the hardware: how many cameras there are, whether a robot moves the camera or the target, and whether a laser projects a line for triangulation.

Each workflow also calibrates the intrinsics of every camera it uses: a rig does not need a planar-intrinsics run per camera first. The hand-eye workflows cover both mountings, the camera on the gripper (eye-in-hand, the default) and the camera fixed in the scene (eye-to-hand); the configuration says which.

The two rig laser workflows differ in staging. rig_laserline_device fits the laser planes against a rig hand-eye result you computed earlier and keep fixed; rig_handeye_laserline estimates everything at once.

A single laser head on a robot can be calibrated in two workflows, the laserline device and then the single-camera hand-eye. If you do this, hold the first workflow's camera fixed in the second. The laser plane is in that camera's frame, and a hand-eye found with a re-estimated camera does not compose with it. rig_handeye_laserline with one camera avoids the question.

3. Which camera model

The camera model is a setting of every workflow, not a workflow of its own. A fisheye stereo pair is a rig_extrinsics calibration with a fisheye camera model; a tilted-sensor laser profiler is a laserline_device calibration with a Scheimpflug sensor. The model has three parts:

The details are in Choosing a Distortion Model, Fisheye Projection and Sensor Models and Scheimpflug Tilt. A rig may mix models: each camera can override the shared one.

4. What to capture

Every workflow observes a planar calibration target: a chessboard, a ChArUco board, a PuzzleBoard or a ring grid (Calibration Targets and Detection). Measure the square or ring pitch of the printed board; the calibration's scale comes from it.

• Views. At least 3 per camera for any workflow to run; 15 to 30 for a result worth keeping. Tilt the board 15–30° about both of its axes, make it a third to a half of the image wide, and move it into the corners of the image, where distortion is largest. Capturing Calibration Images measures what each of these buys.
• Rigs. Every camera needs its own 3 or more views of the board, and the rig's cameras need views in which they see the board at the same time, so the board links them.
• Robots. One robot pose per view, from the controller, with the frame convention and units written down: the manifest asks for them and does not guess. Rotate the gripper about at least two axes between views; a hand-eye calibration from translations alone is undetermined.
• Lasers. Two frames per view: one with the board lit and the laser off, one with the laser on (Laser Datasets).

Every result carries its parameter uncertainty, which shows how well the views constrained it (Parameter Uncertainty). For a single camera, the library also ranks candidate board poses by how much each would help as the next view (Planning the Next View).

Next

Describe your image folder in a dataset manifest. The folder sniffer writes a first draft from the file and folder names and leaves what only you know, such as the target and the robot's pose convention, marked for you to fill in; the desktop app edits the manifest in a form. Then follow the chapter for your workflow from the table above, and judge the result with Judging a Result.

Architecture Overview

calibration-rs is organized as a layered workspace of Rust crates plus Python bindings. This chapter explains the dependency structure, data flow, and key design patterns.

Crate Dependency Graph

vision-calibration (facade: re-exports everything)
    │
    ├── vision-calibration-pipeline   (sessions, problem types, dataset runner)
    │       ├── vision-calibration-optim     (non-linear optimization)
    │       ├── vision-calibration-linear    (closed-form initialization)
    │       ├── vision-calibration-dataset   (dataset manifest)
    │       ├── vision-calibration-detect    (target detectors, detection cache)
    │       └── vision-geometry
    │
    └── vision-mvg                    (N-view geometry, rectification, dense stereo)
            └── vision-geometry       (two-view solvers)

vision-calibration-core is the base layer: linear, optim, vision-geometry
and vision-mvg all depend on it.

Key rule: vision-calibration-linear and vision-calibration-optim are peers. They both depend on vision-calibration-core but never on each other. This keeps initialization algorithms free of optimization dependencies and vice versa. vision-calibration-linear may also use vision-geometry; vision-calibration-pipeline does not depend on vision-mvg (the facade re-exports it directly).

Crate Responsibilities

vision-calibration-core

The foundation layer providing:

• Math types — nalgebra-based aliases: Pt2, Pt3, Vec3, Mat3, Iso3, Real (= f64)
• Camera models — composable trait-based pipeline: ProjectionModel, DistortionModel, SensorModel, IntrinsicsModel
• Observation types — CorrespondenceView (2D-3D point pairs), View<Meta>, PlanarDataset, RigDataset
• RANSAC engine — generic Estimator trait with configurable options
• Synthetic data utilities — grid generation, pose sampling, projection
• Reprojection error computation — single-camera and multi-camera rig

vision-geometry

Deterministic two-view and pose solvers, all free functions in per-topic modules:

The facade re-exports the main solvers at the root of its mvg module (vision_calibration::mvg::dlt_homography, …) and the PnP solvers as init::pnp; the full set is in the vision-geometry crate.

vision-mvg

N-view geometry on top of vision-geometry: calibrated relative-pose recovery, N-view triangulation, bundle adjustment from a calibration, stereo rectification from a calibration, and dense stereo matching. Facade path: vision_calibration::mvg.

vision-calibration-dataset and vision-calibration-detect

vision-calibration-dataset defines DatasetSpec, the on-disk manifest describing images, robot poses, and target metadata. vision-calibration-detect provides the target detectors (chessboard, ChArUco, PuzzleBoard, ring grid) and a filesystem detection cache. The pipeline's dataset_runner turns a manifest into calibration inputs.

vision-calibration-linear

Closed-form initialization solvers (the two-view solvers live in vision-geometry). Each produces an approximate estimate suitable for seeding non-linear optimization:

vision-calibration-optim

Non-linear refinement with a backend-agnostic architecture:

1. IR (Intermediate Representation) — ProblemIR with ParamBlock and ResidualBlock types that describe optimization problems independently of any solver
2. Factors — generic residual functions parameterized over RealField for automatic differentiation
3. Backends — tiny-solver and factrs: two linearization engines driven by one Levenberg–Marquardt loop with sparse linear solvers
4. Problem builders — domain-specific functions that construct IR from calibration data

vision-calibration-pipeline

The session framework providing production-ready workflows:

• CalibrationSession<P: ProblemType> — generic state container with config, input, state, output, exports
• Step functions — free functions operating on &mut CalibrationSession<P> (e.g., step_init, step_optimize)
• Pipeline functions — convenience wrappers chaining all steps
• JSON checkpointing — full serialization for session persistence
• Seven problem types: PlanarIntrinsicsProblem, SingleCamHandeyeProblem, LaserlineDeviceProblem, RigExtrinsicsProblem, RigHandeyeProblem, RigLaserlineDeviceProblem, RigHandeyeLaserlineProblem. Planar intrinsics takes the camera model as config (camera.model), so a tilted sensor is a value of its sensor slot; the rig problems take camera plus per-camera camera_overrides, so each rig camera can have its own distortion and sensor.

vision-calibration

Unified facade crate that re-exports everything through a clean module hierarchy:

#![allow(unused)]
fn main() {
use vision_calibration::prelude::*;            // Minimal planar hello-world imports
use vision_calibration::planar_intrinsics::*;  // Planar workflow
use vision_calibration::core::*;               // Math types
use vision_calibration::mvg::dlt_homography;   // Two-view solvers
use vision_calibration::mvg::rectification;    // N-view geometry
use vision_calibration::optim::RobustLoss;     // Optimization vocabulary (hand-picked)
}

vision-calibration-py

Python bindings crate (maturin/PyO3) exposing high-level session workflows; published to PyPI.

Data Flow

Every calibration workflow follows the same pattern:

Observations (2D-3D correspondences)
    │
    ▼
Linear Initialization (vision-calibration-linear)
    │  Closed-form solvers: ~5-40% accuracy
    ▼
Non-Linear Refinement (vision-calibration-optim)
    │  Levenberg-Marquardt: <2% accuracy, <1 px reprojection
    ▼
Calibrated Parameters (K, distortion, poses, ...)

The session framework wraps this flow with configuration, state tracking, and checkpointing:

CalibrationSession::new()
    │
    ▼
session.set_input(dataset)
    │
    ▼
step_init(&mut session, None)      ← linear initialization
    │
    ▼
step_optimize(&mut session, None)  ← non-linear refinement
    │
    ▼
session.export()                   ← calibrated parameters

Design Patterns

Composable Camera Model

The camera projection pipeline is built from four independent traits composed via generics:

pixel = K(sensor(distortion(projection(direction))))

Each stage can be mixed and matched. For example, a standard camera uses Pinhole + BrownConrady5 + IdentitySensor + FxFyCxCy, while a laser profiler might use Pinhole + BrownConrady5 + ScheimpflugParams + FxFyCxCy.

Backend-Agnostic Optimization

Problems are defined as an intermediate representation (IR) that is independent of any specific solver. The IR is then compiled to a solver-specific form:

Problem Builder  →  ProblemIR  →  OptimBackend::solve()
                  (generic)       (solver-specific)

This allows swapping the optimization backend without changing problem definitions.

Step Functions

Calibration workflows are decomposed into discrete steps implemented as free functions. This allows:

• Inspection of intermediate state between steps
• Resumption from any point (via JSON checkpointing)
• Customization of per-step options
• Composition of steps from different problem types

Getting Started

This chapter calibrates a camera twice, each time in Rust and in Python. The first run uses synthetic data: a planar board seen from several poses, the two-step calibration (linear initialization, then non-linear refinement), and the intrinsics and reprojection error it recovers. The second run, Calibrate Your Own Images, starts from a folder of board images and ends with the camera, the uncertainty of each parameter and a quality verdict.

Both interfaces drive the same solvers and expose the same workflow: a session per problem type with its step functions, and a one-call runner per workflow (run_calibration in Rust; run_planar_intrinsics, run_rig_extrinsics, ... in Python). The Rust program below steps through a session; the Python program uses the runner, and Step by Step in Python shows the session.

Install

Rust

[dependencies]
vision-calibration = { version = "0.16", features = ["synthetic"] }
anyhow = "1"

vision-calibration is the facade crate: it re-exports the calibration workflows, the dataset manifest, the target detectors' cache and the multiple-view geometry tools. The synthetic feature adds the synthetic-data generators the example below calibrates; a calibration from real images does not need it.

Python

pip install vision-calibration

Python 3.10 or newer. To build the extension from a checkout instead, run maturin develop -m crates/vision-calibration-py/Cargo.toml. The package is vision_calibration; the examples below import it as vc.

Command line

cargo install vision-calibration-cli

The Python package installs the same vcal program. It calibrates a folder of images without code: The Command Line.

A Minimal Calibration

Each program builds a synthetic camera with known intrinsics and Brown-Conrady distortion, projects an 8x6 board (40 mm squares) from several poses, and recovers the camera from those pixels alone.

Rust

use anyhow::Result;
use vision_calibration::core::make_pinhole_camera;
use vision_calibration::planar_intrinsics::{step_init, step_optimize};
use vision_calibration::prelude::*;
use vision_calibration::synthetic::planar;

fn main() -> Result<()> {
    // 1. Define the ground-truth camera
    let k_gt = FxFyCxCy {
        fx: 800.0, fy: 780.0, cx: 640.0, cy: 360.0
    };
    let dist_gt = BrownConrady5 {
        k1: 0.05, k2: -0.02, k3: 0.0,
        p1: 0.001, p2: -0.001, iters: 8,
    };
    let camera = make_pinhole_camera(k_gt, dist_gt);

    // 2. Generate synthetic observations
    let board = planar::grid_points(8, 6, 0.04); // 8x6 grid, 40 mm squares
    let poses = planar::poses_yaw_y_z(6, -0.2, 0.08, 0.5, 0.05);
    let views = planar::project_views_all(&camera, &board, &poses)?;

    // 3. Create a session and set the input
    let dataset = PlanarDataset::new(
        views.into_iter().map(View::without_meta).collect()
    )?;
    let mut session = CalibrationSession::<PlanarIntrinsicsProblem>::new();
    session.set_input(dataset)?;

    // 4. Run the calibration
    step_init(&mut session, None)?;      // linear initialization
    step_optimize(&mut session, None)?;  // non-linear refinement

    // 5. Inspect the result
    let export = session.export()?;
    let k = export.params.camera.intrinsics;
    println!("fx={:.1}, fy={:.1}, cx={:.1}, cy={:.1}",
             k.fx, k.fy, k.cx, k.cy);
    println!("Mean reprojection error: {:.4} px",
             export.mean_reproj_error);

    Ok(())
}

The library ships this program as an example:

cargo run -p vision-calibration --example planar_synthetic

Python

import math
import vision_calibration as vc

# 1. The ground-truth camera and a planar board (8x6 grid, 40 mm squares)
fx, fy, cx, cy = 800.0, 780.0, 640.0, 360.0
k1, k2, p1, p2 = 0.05, -0.02, 0.001, -0.001
board = [(i * 0.04, j * 0.04, 0.0) for i in range(8) for j in range(6)]


def project(pt, yaw, tilt, z):
    """Pixel of a board point for a board turned by (yaw, tilt) at depth z."""
    x, y, _ = pt
    cyw, syw, ct, st = math.cos(yaw), math.sin(yaw), math.cos(tilt), math.sin(tilt)
    xc = cyw * x + syw * st * y          # R = Ry(yaw) * Rx(tilt)
    yc = ct * y
    zc = -syw * x + cyw * st * y + z
    xn, yn = xc / zc, yc / zc
    r2 = xn * xn + yn * yn
    radial = 1.0 + k1 * r2 + k2 * r2 * r2
    xd = xn * radial + 2 * p1 * xn * yn + p2 * (r2 + 2 * xn * xn)
    yd = yn * radial + p1 * (r2 + 2 * yn * yn) + 2 * p2 * xn * yn
    return (fx * xd + cx, fy * yd + cy)


# 2. Synthetic observations: varied yaw, tilt and distance
poses = [(-0.3 + 0.1 * v, 0.25 * math.sin(v), 0.45 + 0.04 * v) for v in range(8)]
views = [
    vc.PlanarView(
        observation=vc.Observation(
            points_3d=board,
            points_2d=[project(p, yaw, tilt, z) for p in board],
        )
    )
    for yaw, tilt, z in poses
]

# 3. Build the input and run the calibration (initialization + refinement)
dataset = vc.PlanarIntrinsicsInput(views=views)
result = vc.run_planar_intrinsics(dataset)

# 4. Inspect the result
k = result.camera.intrinsics
print(f"fx={k.fx:.1f}, fy={k.fy:.1f}, cx={k.cx:.1f}, cy={k.cy:.1f}")
print(f"Mean reprojection error: {result.mean_reproj_error:.4f} px")

Real data replaces step 2 in both languages: detect the board corners in each image and pair every corner with its position on the board. Calibrate Your Own Images below does that for a folder of images; Planar Intrinsics with Real Data shows the detection step by hand.

Step by Step in Python

The runner above is a session run in one call. The session gives the steps, their results and the init log:

session = vc.PlanarIntrinsicsSession()
session.set_input(dataset)
init = session.step_init()                 # linear initialization
print(f"init fx={init.camera.intrinsics.fx:.1f}")
session.step_optimize()                    # non-linear refinement
result = session.export()                  # the same PlanarIntrinsicsExport

cam = vc.CameraModel(result.camera)        # project with the calibrated camera
print(cam.project([0.0, 0.0, 1.0]))        # the principal point

Calibrate Your Own Images

A folder of photographs of a calibration board is enough to start. Without code, vcal init <folder> and vcal run <folder>/dataset.toml do what this section does (The Command Line). The program below takes three steps. It drafts a dataset manifest that describes the folder, waits for you to fill in what only you know, and then detects the board, calibrates and judges the result. It works for any workflow: the manifest's topology picks it.

1. Draft the manifest

The folder sniffer reads file and folder names, never image content. It groups images into cameras, finds a robot-pose file if there is one and picks the topology. What it cannot know, the board above all, it leaves as a placeholder and lists under _unresolved. For a folder of board images from one camera, the draft is:

{
  "version": 1,
  "topology": "planar_intrinsics",
  "cameras": [{ "id": "cam0", "images": { "kind": "glob", "pattern": "*.png" } }],
  "target": { "kind": "chessboard", "rows": 0, "cols": 0, "square_size_m": 0.0 },
  "_unresolved": ["target"]
}

2. Fill in what only you know

Describe the board and delete _unresolved. For a chessboard, rows and cols count the inner corners, where four squares meet, and square_size_m is the measured side of a printed square in metres:

"target": { "kind": "chessboard", "rows": 7, "cols": 11, "square_size_m": 0.03 }

The calibration refuses a manifest whose _unresolved list is not empty, so a placeholder never reaches a result. ChArUco boards, PuzzleBoards and ring grids are described in Calibration Targets and Detection, and robot poses and the other fields in The Dataset Manifest.

3. Calibrate and judge

Rust

[dependencies]
vision-calibration = "0.16"
anyhow = "1"
serde_json = "1"

The real-image route needs no feature flags: detection, the manifest and the dataset runner are part of the default build.

use std::path::PathBuf;

use anyhow::{Result, bail};
use vision_calibration::analysis::{CheckStatus, QualityThresholds, assess};
use vision_calibration::common::CalibratedCameras;
use vision_calibration::core::Uncertainty;
use vision_calibration::dataset::{DatasetSpec, sniff_folder};
use vision_calibration::control::RunControl;
use vision_calibration::dataset_runner::{Detection, calibrate_dataset};
use vision_calibration::detect::FsDetectionCache;

fn main() -> Result<()> {
    let dir = PathBuf::from("my_board_images");
    let image_size = [1024, 576]; // width, height in pixels

    // 1. Draft the manifest and stop until it is filled in.
    let manifest = dir.join("dataset.json");
    if !manifest.exists() {
        let draft = sniff_folder(&dir)?;
        std::fs::write(&manifest, serde_json::to_string_pretty(&draft)?)?;
        bail!("fill in {:?} in {}, delete `_unresolved`", draft.unresolved, manifest.display());
    }

    // 2. Detect (cached for the next run) and calibrate.
    let spec = DatasetSpec::from_path(&manifest)?;
    let cache = FsDetectionCache::new(dir.join(".detections"));
    let run = calibrate_dataset(&spec, &dir, None, Detection::Cached(&cache), None, &RunControl::default())?;
    println!("{} of {} views used", run.usable_views, run.total_views);

    // 3. The camera and the uncertainty of its parameters.
    let export = &run.export;
    let k = &export.camera_params()[0].intrinsics;
    println!("fx={:.2} fy={:.2} cx={:.2} cy={:.2}", k.fx, k.fy, k.cx, k.cy);
    if let Uncertainty::Available(cov) = export.parameter_uncertainty() {
        for label in ["fx", "fy", "cx", "cy"] {
            println!("  σ({label}) = {:.3} px", cov.std_dev(label).unwrap_or(f64::NAN));
        }
    }

    // 4. The verdict, and the checks that did not pass.
    let report = assess(export, &[image_size], &QualityThresholds::default())?;
    println!("verdict: {:?}", report.verdict);
    for check in report.checks.iter().filter(|c| c.status != CheckStatus::Pass) {
        println!("  {:?} {:?}: {}", check.kind, check.status, check.message);
    }

    // 5. Keep the export: JSON with a published schema.
    std::fs::write(dir.join("calibration.json"), serde_json::to_string_pretty(export)?)?;
    Ok(())
}

calibrate_dataset reads the manifest's topology and runs that workflow with its default configuration; pass Some(CalibrationConfig::...) to change it (see Configuring a Run). Its result holds the export, which every analysis accepts whichever workflow produced it, and the detections it left out with the reason for each. The library ships this program, generalized to several cameras, as the calibrate_folder example:

cargo run --release -p vision-calibration --example calibrate_folder -- my_board_images 1024 576

Python

import json
import sys
from pathlib import Path

import vision_calibration as vc

folder = Path("my_board_images")
manifest = folder / "dataset.json"

# 1. Draft the manifest and stop until it is filled in.
if not manifest.exists():
    draft = vc.sniff_folder(folder)
    manifest.write_text(json.dumps(draft, indent=2))
    sys.exit(f"fill in {draft['_unresolved']} in {manifest}, delete _unresolved")

# 2. Detect (cached for the next run) and calibrate.
cache = vc.FsDetectionCache(folder / ".detections")
result = vc.calibrate_dataset(manifest, cache=cache)

# 3. The camera and the uncertainty of its parameters.
k, u = result.camera.intrinsics, result.uncertainty
print(f"fx={k.fx:.2f} ± {u.std_dev('fx'):.3f}  fy={k.fy:.2f} ± {u.std_dev('fy'):.3f}")
print(f"cx={k.cx:.2f} ± {u.std_dev('cx'):.3f}  cy={k.cy:.2f} ± {u.std_dev('cy'):.3f}")

# 4. The verdict, and the checks that did not pass.
report = vc.assess(result, image_sizes=[(1024, 576)])
print("verdict:", report.verdict)
for check in report.checks:
    if check.status != "pass":
        print(" ", check.kind, check.status, check.message)

Reading the output

On the 20 left-camera images of the stereo set in Stereo Rig with Real Data (a 7×11 chessboard with 30 mm squares, 1024×576 pixels), both programs print the same numbers; the Python program's output is:

fx=713.85 ± 7.672  fy=724.67 ± 7.821
cx=523.41 ± 1.688  cy=285.95 ± 1.433
verdict: warn
  variance_factor not_applicable no expected feature noise was stated (expected_feature_noise_px)
  projection_sigma warn projection σ up to 7.065 px over the 1024×576 image (centre 1.665 px, corners 7.065 px)

Read it from the bottom up. The verdict is warn, not pass: where a ray lands in the image is uncertain by up to 7 px in the corners, against 1.7 px in the centre. The views did not cover the image corners well enough to pin the focal length down to better than about 1 % (σ(fx) ≈ 7.7 px). More views with the board near the corners and tilted would shrink it; Planning the Next View ranks candidate poses by how much each would help. The variance check is not applicable until you state the detector's noise (expected_feature_noise_px). Judging a Result explains every check, and Reading the Covariance the σ.

No images at hand: a sample dataset

The library downloads real calibration datasets with a manifest already filled in. Each one comes from its publisher at a pinned revision, every file is checked against its SHA-256, and a second fetch reuses the cache:

Each sample carries the citation its publisher asks for (citation in the catalogue). In Rust, fetching needs the fetch feature, which adds an HTTPS client:

vision-calibration = { version = "0.16", features = ["fetch"] }

use vision_calibration::dataset::{DatasetSpec, samples};
use vision_calibration::control::RunControl;
use vision_calibration::dataset_runner::{Detection, calibrate_dataset};

fn main() -> anyhow::Result<()> {
    let manifest = samples::fetch("opencv_left", &samples::default_cache_dir()?)?;
    let spec = DatasetSpec::from_path(&manifest)?;
    let dir = manifest.parent().expect("the manifest sits in the sample's folder");
    let run = calibrate_dataset(&spec, dir, None, Detection::Uncached, None, &RunControl::default())?;
    println!("{} of {} views used", run.usable_views, run.total_views);
    Ok(())
}

manifest = vc.samples.fetch("opencv_left")
result = vc.calibrate_dataset(manifest)

The cache is vision-calibration in the platform's cache directory; the VISION_CALIBRATION_DATA environment variable, or a directory passed to fetch, puts it elsewhere. The example program above runs on a fetched sample's folder as on your own. The fisheye sample needs a fisheye camera model in its configuration (Fisheye Projection).

Configuring a Run

Every workflow has a *Config type with the same grouped shape (solver, camera, init, ...). Rust updates it on the session or hands it to calibrate_dataset wrapped in CalibrationConfig; Python passes it to the runner, to calibrate_dataset or to the session's set_config.

Rust

#![allow(unused)]
fn main() {
use vision_calibration::optim::RobustLoss;
use vision_calibration::planar_intrinsics::run_calibration;

session.update_config(|c| {
    c.solver.max_iters = 80;
    c.solver.robust_loss = RobustLoss::Huber { scale: 1.0 };
})?;
run_calibration(&mut session)?; // the same two steps in one call
}

From a manifest, the configuration names its workflow, and a configuration for a different topology than the manifest's is an error raised before any image is read:

#![allow(unused)]
fn main() {
use vision_calibration::control::RunControl;
use vision_calibration::dataset_runner::{CalibrationConfig, Detection, calibrate_dataset};
use vision_calibration::planar_intrinsics::PlanarIntrinsicsConfig;

let mut config = PlanarIntrinsicsConfig::default();
config.solver.max_iters = 80;
let config = CalibrationConfig::PlanarIntrinsics(config);
let run = calibrate_dataset(&spec, &dir, Some(config), Detection::Cached(&cache), None, &RunControl::default())?;
}

Python

config = vc.PlanarIntrinsicsConfig(
    solver=vc.SolverConfig(max_iters=80, robust_loss=vc.robust_huber(1.0)),
)
result = vc.run_planar_intrinsics(dataset, config)
result = vc.calibrate_dataset("my_board_images/dataset.json", config)  # from a manifest

What Happens in Each Step

Step 1: Linear Initialization

step_init (session.step_init() in Python; run_planar_intrinsics runs it first):

1. Computes a homography from each view's 2D-3D correspondences using DLT (direct linear transform)
2. Estimates intrinsics $K$ from the homographies using Zhang's method
3. Estimates lens distortion coefficients from homography residuals
4. Iterates steps 2-3 to refine the joint $K$ + distortion estimate
5. Recovers each view's camera pose from its homography and $K$

After this step, intrinsics are typically within 10-40% of the true values: good enough to start the non-linear optimization.

Step 2: Bundle Adjustment

step_optimize runs Levenberg-Marquardt, minimizing the total reprojection error:

$$\underset{K,\mathbf{d},\{T_i\}}{\min }\sum _{i=1}^N\sum _{j=1}^{M_i}{\Vert \pi (K,\mathbf{d},T_i,{\mathbf{P}}_j)-{\mathbf{p}}_{ij}\Vert }^2$$

where $\pi$ is the camera projection function, $T_i$ is the pose of view $i$, $\mathbf{d}$ are distortion coefficients, ${\mathbf{P}}_j$ are known 3D board points, and ${\mathbf{p}}_{ij}$ are observed pixel coordinates.

After optimization, expect under 2% error on intrinsics and under 1 px mean reprojection error on real data. The synthetic data above is noise-free, so both programs recover the ground truth almost exactly.

Where to Go Next

• Stereo Rig with Real Data: a two-camera rig from detected corners, in under a minute.
• Planar Intrinsics Calibration: the full mathematical treatment of this workflow.
• Parameter Uncertainty: every result carries the covariance of its parameters.
• Architecture Overview: how the crates fit together.
• The Dataset Manifest: every field of the manifest, robot poses and image pairing included.

The Command Line

vcal calibrates a folder of images without a line of code: it drafts the dataset manifest, checks it, detects the target, calibrates, and writes the result and its quality report as JSON. It runs the same library calls as Calibrate Your Own Images, so the numbers are the same.

cargo install vision-calibration-cli   # or: pip install vision-calibration

The Rust crate and the Python package install the same program.

vcal <command> --help lists each command's options.

A folder of images

vcal init data/my_camera

wrote data/my_camera/dataset.toml
fill in, then delete `_unresolved`:
  target

The draft lists what the file names cannot tell, the board above all, under _unresolved. Describe the board (The Dataset Manifest) and delete the list:

[target]
kind = "chessboard"
rows = 6            # inner corners down
cols = 9            # inner corners across
square_size_m = 0.025

vcal check then reads the manifest and each image's header. It reports a manifest that is incomplete or invalid, a camera whose images it cannot find, and a camera whose images differ in size, all before any detection:

$ vcal check data/my_camera/dataset.toml
data/my_camera/dataset.toml: planar_intrinsics
  camera cam0: 640 × 480 px
ok

vcal run calibrates. On the 13 views of the opencv_left sample, fetched to ~/.cache/vision-calibration, it prints:

$ vcal run ~/.cache/vision-calibration/opencv_left/dataset.toml
planar_intrinsics: 13 of 13 views used
camera 0 (left), 640 × 480 px:
  fx = 534.108 ± 0.930 px
  fy = 534.155 ± 0.966 px
  cx = 341.029 ± 1.075 px
  cy = 233.586 ± 1.144 px
verdict: warn
  not_applicable variance_factor: no expected feature noise was stated (expected_feature_noise_px)
  warn projection_sigma camera 0: projection σ up to 5.490 px over the 640×480 image (centre 1.146 px, corners 5.490 px)
wrote ~/.cache/vision-calibration/opencv_left/calibration.json and ~/.cache/vision-calibration/opencv_left/quality.json

Each parameter comes with its σ, and the verdict with the checks that did not pass (Judging a Result). Here the 13 views leave the image corners uncertain by 5 px; Capturing Calibration Images says which views would fix that. A view left out of the calibration is listed with its reason, such as a chessboard whose corners do not reach every edge of the board.

calibration.json is the workflow's export, the same JSON the library and the desktop app write, and quality.json is the quality report. --report adds report.html, all of it as one page with the residual and projection-σ figures (The Calibration Report). -o <dir> writes them elsewhere. Detections are cached in .detections next to the manifest, so a second run skips detection; --redetect detects again.

Configuration

Without a configuration file the workflow runs with its defaults: a pinhole camera with Brown-Conrady distortion. -c <file> changes the defaults the file names and keeps the rest. The file is TOML or JSON and holds the workflow's *Config fields. A fisheye camera needs a fisheye model:

# fisheye.toml
camera.model = { projection = "equidistant", distortion = "kannala_brandt4", sensor = "identity" }

vcal run "$(vcal sample fetch opencv_fisheye_stereo)" -c fisheye.toml

On that sample the default model leaves each focal length uncertain by 14 px; the fisheye model brings σ(fx) to 1.1 px. A field the workflow does not have is an error, so a misspelt name never passes silently. The fields are those of Configuring a Run and of each workflow's chapter.

Sample datasets

vcal sample list
vcal run "$(vcal sample fetch kuka_1)" -o out/kuka

fetch downloads a sample from its publisher, verifies every file's SHA-256 and prints the path of its manifest, so it composes with run. A second fetch reuses the cache: VISION_CALIBRATION_DATA, else vision-calibration in the platform's cache directory, or --cache <dir>.

Exit status

A script can stop on a failed calibration with vcal run ... || exit.

Progress and Ctrl-C

On a terminal, vcal run keeps one line of standard error up to date: the images detected (detecting 12/40), then each step and its solver iterations (optimize: iteration 7/50, cost 1.2345e2). Piped or redirected, it prints no progress. Ctrl-C stops the run at the next image or iteration, prints interrupted and exits with status 130; a second Ctrl-C exits at once.

Limits

The laser workflows (laserline_device, rig_laserline_device, rig_handeye_laserline) need a laser-line extractor, which the library takes from code (Laser Datasets); vcal run refuses them before any detection. Calibrate them from Rust or Python.

Capturing Calibration Images

A calibration is only as good as the views it sees. The solver does not tell a good capture from a poor one by the residuals, which can be small either way; the views decide how well the focal length, the principal point and the distortion are determined. This chapter measures what matters, so a capture can be planned before the first image is taken.

The short version, for one camera and a planar board:

• 15 to 30 views, each with the whole board in the image.
• Tilt the board 15–30° out of the fronto-parallel pose, and turn the tilt direction from view to view: about the board's horizontal axis, then its vertical axis, both ways.
• Make the board large in the image: a third to a half of the image width.
• Move the board over the whole image, into every corner, not only the centre.

A rig needs this for every camera, plus views that two cameras see at once. A robot moves the camera instead of the board, so the same rules apply to the gripper's poses, rotated about at least two axes (Choose Your Workflow).

What each rule buys

The numbers below come from the capture_planning example, which you can run and change:

cargo run --release -p vision-calibration --example capture_planning

It simulates one camera (1280 × 960 pixels, fx = 1000 px, barrel distortion) and a chessboard of 30 mm squares with 0.2 px corner noise, and calibrates each capture plan. The baseline is 16 views of a board with 12 × 9 interior corners, 0.65 m away, centred on each image quadrant in turn and tilted by 20°. Each table changes one thing. A view whose board leaves the image is dropped, as a chessboard detector would drop it ("used").

The columns are the stated standard deviations of the focal length, the principal point and the first radial coefficient, and the projection σ (Projection Uncertainty): how far, in pixels, the calibration's uncertainty moves where a ray lands, at the image centre and at its worst anywhere in the image. The quality check warns when the worst exceeds 1 px (Judging a Result).

Number of views

The uncertainty falls roughly as one over the square root of the number of views: doubling the views gains about 30 %. More views are cheap insurance, but they do not make up for a poor tilt or a small board.

Tilt

The focal length is measured by perspective: how the board's far edge shrinks against its near edge. A board facing the camera squarely shows none, and its distance and the focal length trade off exactly, so the initialization fails (see When the views cannot work). A few degrees leave the focal length tens of pixels uncertain. From about 15° on it is well determined, and more tilt helps little. Past 30° a large board no longer fits in the image when it sits near a corner, and views are lost: here only 4 of 16 survive at 40°.

Tilt direction

Views tilted the same way are parallel planes and tell the camera nothing more than one of them; the initialization fails. Tilting about the board's horizontal axis, both ways, gives two board orientations: they determine the vertical focal length and leave the horizontal one almost free. Turn the board about both axes.

Board size in the image

A larger board in the image has more corners and, more importantly, more perspective across it. Going from a fifth to two fifths of the image width cuts every σ by a factor of six or seven. Bring the board closer, or use a larger board, until it fills a third to a half of the width; the limit is that it still fits whole in the image when tilted near a corner.

Placement

Distortion grows toward the image corners. Views that keep the board in the centre determine the focal length as well as any, but leave the distortion coefficients to be extrapolated to the corners, where the projection σ reaches 14 px. Moving the board into each quadrant, its outer corners near the image corners, brings the worst σ under a pixel.

A rig needs its joint error too

The tables above judge each parameter by its own σ. A rig adds parameters that trade off with the cameras' own: one camera's rotation in the rig and the two principal points all move the image of a board that sits in the middle of both images the same way. Views that leave them to trade off still give every parameter a σ that holds, but the error along the combination the views do pin down (here camera 1's yaw against the difference of the principal points) comes from larger errors that cancel, and through the second order those still reach it. Anything that uses the whole covariance at once — a calibration prior in bundle adjustment, a test of two calibrations against each other — then trusts that combination several times too much.

The result's uncertainty predicts this (second_order, see Reading the Covariance), and the quality report's covariance_linearity check warns above 1.2. For a synthetic stereo pair (eight views of an 8 × 6 board with 30 mm squares at 0.6–0.8 m, 0.3 px noise), over 300 calibrations each:

Tilt and placement each help, and only both together bring the combination within its σ (the last row's measured worst is below what 300 calibrations resolve). The effect grows with the noise variance: the first row at 0.1 px predicts 1.35, at 0.03 px 1.04. So for a rig, put the board in the image corners as well as tilting it, in views both cameras see.

Check while you capture

You do not have to trust the rules: calibrate after the first ten or so views and read the result. The quality verdict flags poor coverage, a large projection σ and, for a rig, a joint error beyond the covariance, and the projection map shows where in the image the calibration is weak:

#![allow(unused)]
fn main() {
use vision_calibration::analysis::{QualityThresholds, assess};
use vision_calibration::common::CalibratedCameras;

let export = session.export()?;
let report = assess(&export, &[[1280, 960]], &QualityThresholds::default())?;
println!("{:?}", report.verdict);
let map = export.projection_uncertainty(0, [1280, 960])?;
println!("σ centre {:.2} px, worst {:.2} px", map.center, map.max);
}

report = vc.assess(result, image_sizes=[(1280, 960)])
print(report.verdict)
p = vc.projection_uncertainty(result, (1280, 960), camera=0)
print(f"σ centre {p.center:.2f} px, worst {p.max:.2f} px")

For a single camera, rank_candidates then says which board pose to take next: it predicts, without a re-solve, how much each candidate view would reduce the worst projection σ (Planning the Next View).

When the views cannot work

When the board planes are parallel in every view, fronto-parallel or tilted the same way throughout, no calibration can recover the focal length, and the initialization stops before any solve with an error that says so and names the fix:

the 16 views do not determine the camera intrinsics: ...; tilt the board
about both of its axes, by 15° to 30°, from view to view

The initialization measures how far the board orientations spread across the views, as seen through a camera whose focal length is the image extent the corners cover, and rejects a spread under 1°. A spread of a few degrees still initializes, with the large σ the tables above show. With strong lens distortion, parallel boards no longer look parallel, and the same error comes from Zhang's method finding no camera that fits the views.

In Rust it is a typed error to match on; in Python it is a ValueError with the same message:

#![allow(unused)]
fn main() {
use vision_calibration::{Error, init};

match run_calibration(&mut session) {
    Err(Error::Linear(init::Error::DegenerateViews { .. })) => {
        println!("recapture: tilt the board about both of its axes");
    }
    other => other?,
}
}

The image itself

The geometry above assumes every corner is found to a fraction of a pixel. That needs:

• Focus and sharpness: the board in focus over its whole tilted extent, with an exposure short enough that it does not blur when held by hand.
• Even lighting without glare on the board; a matte print.
• A flat, rigid board, its square size measured on the print (the calibration's scale comes from it), not taken from the file.
• Fixed camera settings: focus, zoom and resolution unchanged through the capture and afterwards. A refocus changes the intrinsics.

The detection noise scales every σ above in proportion: a detector that finds corners to 0.1 px instead of 0.2 px halves them.

Stereo Rig with Real Data

This chapter walks through the stereo_session example, which calibrates a stereo camera rig from synchronized left/right views with RigExtrinsicsProblem. It is the quickest way to see a complete multi-camera calibration on real data: the corners are already detected and committed to the repository, so no images are needed.

Run It First

cargo run -p vision-calibration --example stereo_session --release

The program loads 20 views of a 7x11 chessboard (30 mm squares) seen by both cameras and prints, among other lines:

--- Step 2: Per-Camera Intrinsics Optimization ---
  Camera 0: fx=713.8, fy=724.7, cx=523.4, cy=286.0, reproj_err=0.234px
  Camera 1: fx=724.8, fy=735.6, cx=513.9, cy=293.0, reproj_err=0.244px

--- Step 4: Rig Bundle Adjustment ---
  Rig BA mean reprojection error: 0.2461 px
    Camera 0: 0.2413 px
    Camera 1: 0.2509 px
  Rig baseline (after BA): |c1 - c0| = 0.1898 m (189.84 mm)

--- Uncertainty ---
  Baseline: 189.84 ± 0.489 mm
  Corner noise σ ≈ 0.188 px (6160 residuals)

The ± comes from the export's uncertainty: the covariance of the estimated parameters, here of camera 1's cam_se3_rig. The rig solve refines both cameras' intrinsics with the rig, so it includes their uncertainty, and it is scaled by the view scatter, about 4 here: the error the views do not share with each other (see Parameter Uncertainty).

The Session in Five Calls

A calibration is a session plus a sequence of step functions invoked in order. Each step reads what it needs from the session, writes its own output back, and returns a typed result, so you can inspect the state between any two steps:

Session::new()
  → set_input(observations)
  → step_intrinsics_init_all      ← Zhang's method per camera
  → step_intrinsics_optimize_all  ← per-camera refinement
  → step_rig_init                 ← linear cam_se3_rig + rig_se3_target
  → step_rig_optimize             ← joint bundle adjustment
  → export()                      ← stable, JSON-serializable result

The interesting part of the example is ten lines:

#![allow(unused)]
fn main() {
let mut session = CalibrationSession::<RigExtrinsicsProblem>::new();
session.set_input(input)?;

step_intrinsics_init_all(&mut session, None)?;
step_intrinsics_optimize_all(&mut session, None)?;
step_rig_init(&mut session, None)?;
step_rig_optimize(&mut session, None)?;

let export = session.export()?;
let mean_reproj = export.mean_reproj_error;
let centre = |i: usize| export.cam_se3_rig[i].inverse().translation.vector;
let baseline = (centre(1) - centre(0)).norm();
}

RigExtrinsicsProblem is a marker type: through the ProblemType trait it fixes the input, configuration and export types, so the session is fully typed. Each step_* takes an Option<*Options> for per-call tweaks (max_iters, verbosity, ...); None uses the session's config. export() returns a RigExtrinsicsExport, the stable view of the result and the handoff to other tools.

The same calibration from Python is one call (crates/vision-calibration-py/examples/stereo_session.py loads the same observation file):

import vision_calibration as vc

# `views`: one vc.RigExtrinsicsView per frame, with a vc.Observation (or None)
# per camera; `dataset` wraps them.
dataset = vc.RigExtrinsicsInput(num_cameras=2, views=views)
result = vc.run_rig_extrinsics(dataset)
print(result.mean_reproj_error, result.per_cam_reproj_errors)

Other camera setups use other problem types with the same pattern: only the input shape and the step list change. Step Functions vs Pipeline Functions tabulates them, and each problem type's facade module (vision_calibration::rig_extrinsics, ...) re-exports its types and steps.

Building the Rig Dataset

A rig view holds one optional observation per camera. A camera that does not see the target in a given frame contributes None; the calibration copes with such views as long as each camera has enough usable views overall (at least 3).

#![allow(unused)]
fn main() {
use vision_calibration::core::{NoMeta, RigDataset, RigView, RigViewObs};

views.push(RigView {
    meta: NoMeta,
    obs: RigViewObs {
        cameras: vec![left, right], // each an Option<CorrespondenceView>
    },
});

let input = RigDataset::new(views, 2)?; // 2 cameras
}

Each CorrespondenceView is built from the detected corners with CorrespondenceView::new(points_3d, points_2d)?, exactly as in Planar Intrinsics with Real Data.

Reading the Step Results

Each step returns its own result struct; keep them to report progress:

#![allow(unused)]
fn main() {
use vision_calibration::rig_extrinsics::{
    RigExtrinsicsProblem, step_intrinsics_init_all, step_intrinsics_optimize_all,
    step_rig_init, step_rig_optimize,
};

step_intrinsics_init_all(&mut session, None)?;
let intr_opt = step_intrinsics_optimize_all(&mut session, None)?;
let rig_init = step_rig_init(&mut session, None)?;
let rig_opt = step_rig_optimize(&mut session, None)?;

println!("Rig BA mean reprojection error: {:.4} px", rig_opt.mean_reproj_error);
for (i, err) in rig_opt.per_cam_reproj_errors.iter().enumerate() {
    println!("  Camera {i}: {err:.4} px");
}
}

run_calibration(&mut session) (vision_calibration::rig_extrinsics::run_calibration) runs the same four steps with default options in one call.

Every step also appends an entry to the session log, which records what the initialization produced and which values came from manual seeds (see Seeding with Prior Knowledge):

#![allow(unused)]
fn main() {
for entry in session.log() {
    println!("[{}] {:?} {}",
        entry.operation, entry.success, entry.notes.as_deref().unwrap_or(""));
}
}

CalibrationSession is JSON-serializable: persist it after step_rig_init (session.to_json()) and resume later with a fresh step_rig_optimize under different solver options (see CalibrationSession).

The Example Dataset

The example reads regression/observations/stereo_rig.json: the corners of a 7×11 chessboard with 30 mm squares, detected in 20 image pairs of a 1024×576 stereo camera. The file is part of the regression suite; its input field is a serialized RigExtrinsicsInput, so it loads with serde_json:

#![allow(unused)]
fn main() {
let text = std::fs::read_to_string("regression/observations/stereo_rig.json")?;
let mut file: serde_json::Value = serde_json::from_str(&text)?;
let input: RigExtrinsicsInput = serde_json::from_value(file["input"].take())?;
}

With your own images, detect the board in each camera's image of a pair and build the RigView as above. Pair the images by a shared index or timestamp; a camera whose detection fails contributes None. The detection step is the same as in Planar Intrinsics with Real Data. A ChArUco board identifies its corners by their markers, so a partly visible board still yields correspondences:

#![allow(unused)]
fn main() {
use calib_targets::aruco::builtins;
use calib_targets::charuco::{CharucoBoardSpec, CharucoParams, MarkerLayout};
use calib_targets::detect;

let board = CharucoBoardSpec::new(
    22,                       // rows
    22,                       // columns
    0.00135,                  // cell size, metres
    0.75,                     // marker size relative to a cell
    builtins::DICT_4X4_1000,  // dictionary
)
.with_marker_layout(MarkerLayout::OpenCvCharuco);
let charuco_params = CharucoParams::for_board(board);

let detection = detect::detect_charuco(&img, &charuco_params);
for corner in detection.corners {
    let target = corner.target_position; // board-frame position, metres
    points_3d.push(Pt3::new(target.x as f64, target.y as f64, 0.0));
    points_2d.push(Pt2::new(corner.position.x as f64, corner.position.y as f64));
}
}

Command-Line Options

cargo run -p vision-calibration --example stereo_session --release -- --max-views=10

--max-views=N uses the first N views. The Python counterpart is crates/vision-calibration-py/examples/stereo_session.py.

Interpreting Results

The example prints the number of views each camera sees, per-camera intrinsics with per-camera reprojection errors, the rig BA reprojection error (about 0.25 px on this dataset), and the baseline (about 189 mm).

• Per-camera intrinsics: cameras with the same lens should have similar focal lengths. Differences in principal point are normal.
• Baseline: the norm of the translation between camera centers should match the physical distance between the cameras. For a horizontal stereo pair, expect a translation primarily along X and a rotation primarily around Y.
• Reprojection errors should be within your application tolerance.

Verification

• Epipolar constraint: corresponding points should lie on epipolar lines. Measure the average distance from points to their epipolar lines (see Epipolar Geometry).
• Rectification: the pair should rectify cleanly, with horizontal scanlines in the rectified images corresponding.
• Triangulation: triangulate known board corners and verify the 3D distances match the board geometry (see Linear Triangulation).

Data Collection Tips

• Both cameras should see the board in most views; views seen by only one camera constrain only that camera's intrinsics.
• Vary board orientation and distance — same advice as single-camera calibration.
• Ensure synchronization — if images are not captured simultaneously, the board may have moved between left and right captures.
• Overlap region — the board should be in the overlapping field of view of both cameras.

Planar Intrinsics Calibration

This is the most common calibration workflow: estimate camera intrinsics and lens distortion from multiple views of a planar calibration board. It combines Zhang's linear initialization with Levenberg-Marquardt bundle adjustment.

Problem Formulation

Parameters:

• Intrinsics: $K=(f_x,f_y,c_x,c_y)$ — 4 scalars
• Distortion: $\mathbf{d}=(k_1,k_2,k_3,p_1,p_2)$ — 5 scalars
• Per-view poses: $\{T_v{\}}_{v=1}^M$ — $M$ SE(3) transforms (6 DOF each)

Total: $9+6M$ parameters.

Observations: For each view $v$ and board point $j$:

• Known 3D position ${\mathbf{P}}_j$ (on the board, at $Z=0$)
• Observed pixel ${\mathbf{p}}_{vj}$

Objective: Minimize total reprojection error:

$$\underset{K,\mathbf{d},\{T_v\}}{\min }\sum _{v=1}^M\sum _{j=1}^{N_v}{\Vert \pi (K,\mathbf{d},T_v,{\mathbf{P}}_j)-{\mathbf{p}}_{vj}\Vert }^2$$

where $\pi$ is the full camera projection pipeline: SE(3) transform → pinhole → distortion → intrinsics.

With robust loss $\rho$:

$$\underset{K,\mathbf{d},\{T_v\}}{\min }\sum _{v=1}^M\sum _{j=1}^{N_v}\rho \left(\Vert \pi (K,\mathbf{d},T_v,{\mathbf{P}}_j)-{\mathbf{p}}_{vj}\Vert \right)$$

Two-Step Pipeline

Step 1: Linear Initialization (step_init)

1. Homographies: Compute $H_v$ for each view via DLT (from board points at $Z=0$ to observed pixels)
2. Intrinsics: Estimate $K$ from homographies using Zhang's method, iteratively refined with distortion estimation (see Iterative Intrinsics)
3. Distortion: Estimate $(k_1,k_2,p_1,p_2)$ from homography residuals (see Distortion Fit)
4. Poses: Decompose each homography to recover $T_v$ (see Pose from Homography)

After initialization, intrinsics are typically within 10-40% of the true values.

Step 2: Non-Linear Refinement (step_optimize)

Constructs the optimization problem as IR:

• Parameter blocks: "cam" (4D, Euclidean), "dist" (the distortion model's coefficients, Euclidean; absent for no distortion), "sensor" (tilt_x, tilt_y, Scheimpflug sensor only), "pose/0"..."pose/M-1" (7D, SE3)
• Residual blocks: One ReprojPoint { model: PINHOLE4_DIST5, chain: SinglePose, .. } per observation (2D residual)
• Solver: the Levenberg–Marquardt loop on the backend solver.backend selects (tiny-solver by default; see Solver Backends)

After optimization, expect <2% intrinsics error and <1 px mean reprojection error.

Configuration

init and solver are shared sub-structs reused by every intrinsics-bearing problem type; camera names the camera model and its fixed parameters:

#![allow(unused)]
fn main() {
pub struct PlanarIntrinsicsConfig {
    // Linear-initialization stage settings.
    pub init: IntrinsicsInitConfig,
    // { iterations: usize = 2, fix_k3: bool = true }; it fits only the
    //   distortion terms the camera's solves estimate

    // Non-linear solve stage settings.
    pub solver: SolverConfig,
    // { max_iters: usize = 120, verbosity: usize = 0, robust_loss: RobustLoss = None }

    // The camera model and the parameters held fixed.
    pub camera: CameraConfig,
    // { model: CameraModelSpec { projection, distortion, sensor },
    //   fix: CameraFix { intrinsics, projection, distortion, sensor } }

    pub fix_poses: Vec<usize>,              // Fix specific view poses
}
}

solver.max_iters is sized for a tilted sensor's tilt valley; the solver stops at convergence, so the cap only bounds runs that do not converge.

Camera Model

camera.model names one kind per slot. The default is a pinhole projection, Brown-Conrady distortion and an identity (untilted) sensor.

camera.model.distortion selects the lens model that is fitted: Brown-Conrady (default), rational (Rational8), thin-prism (ThinPrism9), division (Division1) or none.

#![allow(unused)]
fn main() {
use vision_calibration::core::DistortionKind;

session.update_config(|c| c.camera.model.distortion = DistortionKind::Rational8)?;
}

camera.fix lists, per slot (intrinsics, distortion, sensor), the parameter names held fixed during the non-linear solve; a slot left None keeps the model's default (k3 for Brown-Conrady). The models, their packed layouts, the defaults and when a richer model helps are in Choosing a Distortion Model.

camera.model.sensor = SensorKind::Scheimpflug calibrates a tilted sensor as well; see Tilted (Scheimpflug) Sensor below.

Tilted (Scheimpflug) Sensor

With camera.model.sensor = SensorKind::Scheimpflug the same workflow also estimates the sensor tilt tilt_x, tilt_y (radians). The tilt trades off against the principal point and radial distortion, so the sensor changes three defaults and both steps:

• The default fix list leaves only the low-order radial terms free.
• The linear stage holds the tangential terms while the tilt is estimated: a free tangential term is ill-posed next to the tilt there.
• A Scheimpflug sensor needs a distortion model (DistortionKind::None is a configuration error).

step_init_with_seed takes an optional sensor seed in IntrinsicsManualInit (a typed error for an untilted camera). With a seeded tilt — the nominal mount angle, plus a coarse focal — step_optimize trusts that basin: it sweeps the leading radial coefficient over {0, -0.20, -0.40} with intrinsics and tilt held, then runs a joint solve bounded around the seed (focals [0.75, 1.5]×, principal point ±0.15·f, tilt ±0.35 rad: a mount's nominal tilt can be off by more than 0.1 rad). A single camera's views often pin the tilt poorly, as it trades with the principal point; the result's covariance shows it, and a rig's joint solve (refine_sensor) determines it far better. Without a tilt seed (cold start, experimental) it runs a staged multi-start: a cheap tilt sweep with the principal point frozen, a refine of the winning basin, and a bounded joint refine. The init log warns when the intrinsics are not seeded, because a from-scratch start can settle in a wrong tilt/focal basin.

#![allow(unused)]
fn main() {
use vision_calibration::core::{FxFyCxCy, ScheimpflugParams, SensorKind};
use vision_calibration::planar_intrinsics::{
    PlanarIntrinsicsConfig, PlanarIntrinsicsProblem, IntrinsicsManualInit, step_init_with_seed,
    step_optimize,
};

let mut config = PlanarIntrinsicsConfig::default();
config.camera.model.sensor = SensorKind::Scheimpflug;
let mut session = CalibrationSession::<PlanarIntrinsicsProblem>::new();
session.set_config(config)?;
session.set_input(dataset)?;

let mut seed = IntrinsicsManualInit::default();
seed.intrinsics = Some(FxFyCxCy { fx: 1050.0, fy: 1050.0, cx: 360.0, cy: 270.0 });
seed.sensor = Some(ScheimpflugParams { tilt_x: -0.087, tilt_y: 0.0 }); // nominal mount tilt
step_init_with_seed(&mut session, seed, None)?;
step_optimize(&mut session, None)?;
}

The export is the same PlanarIntrinsicsExport; params.camera.sensor holds the calibrated tilt and the covariance reports tilt_x ± σ. A device spec derives the seed with device_seed::scheimpflug_seed.

Complete Example

#![allow(unused)]
fn main() {
use vision_calibration::prelude::*;
use vision_calibration::planar_intrinsics::{step_init, step_optimize};

let mut session = CalibrationSession::<PlanarIntrinsicsProblem>::new();
session.set_input(dataset)?;

// Optional: customize configuration
session.update_config(|c| {
    c.solver.max_iters = 80;
    c.solver.robust_loss = RobustLoss::Huber { scale: 2.0 };
})?;

// Run pipeline
// Inspect initialization
let init = step_init(&mut session, None)?;
println!("Init fx={:.1}, fy={:.1}", init.camera.intrinsics.fx, init.camera.intrinsics.fy);

step_optimize(&mut session, None)?;

// Export results
let export = session.export()?;
let k = export.params.camera.intrinsics;
println!("Final fx={:.1}, fy={:.1}", k.fx, k.fy);
println!("Reprojection error: {:.4} px", export.mean_reproj_error);
}

Filtering (Optional Step)

After optimization, views or individual observations with high reprojection error can be filtered out:

#![allow(unused)]
fn main() {
use vision_calibration::planar_intrinsics::{step_filter, IntrinsicsFilterOptions};

let mut filter_opts = IntrinsicsFilterOptions::default();
filter_opts.max_reproj_error = 2.0;      // Remove observations > 2 px
filter_opts.min_points_per_view = 10;    // Minimum points to keep a view
filter_opts.remove_sparse_views = true;  // Drop views below threshold
step_filter(&mut session, Some(filter_opts))?;

// Filtering edits the input, which clears the computed state: run both steps again
step_init(&mut session, None)?;
step_optimize(&mut session, None)?;

// Or, in one call: run_calibration_with_filtering(&mut session, Some(filter_opts))?
}

OpenCV equivalence: cv::calibrateCamera performs both initialization and optimization internally. calibration-rs separates these steps for inspection and customization.

Accuracy Expectations

Input Requirements

• Minimum 3 views (for Zhang's closed form)
• Minimum 4 points per view (for homography estimation)
• View diversity: Views should include rotation around both axes and vary in distance
• Board coverage: Points should span the full image area for good distortion estimation

Planar Intrinsics with Real Data

This chapter walks through the planar_real example, which calibrates a camera from real chessboard images.

Dataset

The example uses the images of the kuka_1 sample, which it downloads on the first run (sample datasets):

• Pattern: 17×28 chessboard, 20 mm squares (the size is read from squaresize.txt)
• Images: 30 PNG files 01.png … 30.png, 1928×1208 px

Workflow

1. Corner Detection

The example detects corners with the calib_targets crate (a dev-dependency of the examples; the calibration library itself takes plain point correspondences). Each image is processed independently; failed detections are skipped and the workflow continues with the views that succeed.

#![allow(unused)]
fn main() {
use calib_targets::chessboard::{ChessboardDetection, ChessboardParams};
use calib_targets::detect::{self, default_chess_config};

const SQUARE_SIZE_M: f64 = 0.02; // 20 mm squares

let board_params = ChessboardParams::default();

let img = image::ImageReader::open(path)?.decode()?.to_luma8();
let Ok(detection) = detect::detect_chessboard(&img, &default_chess_config(), &board_params)
else {
    return Ok(None); // no board found in this image
};
}

2. Correspondence Construction

Each detected corner carries its grid index (corner.grid.u, corner.grid.v) and its pixel position. The board lies in the plane $Z=0$, so the 3D point follows directly from the grid index and the square size:

#![allow(unused)]
fn main() {
fn detection_to_view(detection: ChessboardDetection) -> Result<CorrespondenceView> {
    let mut points_3d = Vec::new();
    let mut points_2d = Vec::new();

    for corner in detection.corners {
        let grid = corner.grid;
        points_3d.push(Pt3::new(
            grid.u as f64 * SQUARE_SIZE_M,
            grid.v as f64 * SQUARE_SIZE_M,
            0.0,
        ));
        points_2d.push(Pt2::new(corner.position.x as f64, corner.position.y as f64));
    }

    Ok(CorrespondenceView::new(points_3d, points_2d)?)
}
}

CorrespondenceView::new validates that both point lists have the same length.

3. Dataset and Calibration

The remaining pipeline is identical to the synthetic case:

#![allow(unused)]
fn main() {
let dataset = PlanarDataset::new(views.into_iter().map(View::without_meta).collect())?;

let mut session = CalibrationSession::<PlanarIntrinsicsProblem>::new();
session.set_input(dataset)?;
let init = step_init(&mut session, None)?;
let opt = step_optimize(&mut session, None)?;
let export = session.export()?;

let k = export.params.camera.intrinsics;
println!("fx={:.2} fy={:.2} cx={:.2} cy={:.2}", k.fx, k.fy, k.cx, k.cy);
println!("Mean reprojection error: {:.4} px", export.mean_reproj_error);
}

4. Parameter Uncertainty

The export carries the covariance of the estimated parameters (see Parameter Uncertainty):

#![allow(unused)]
fn main() {
use vision_calibration::core::Uncertainty;

if let Uncertainty::Available(cov) = &export.uncertainty {
    println!("fx = {:.2} ± {:.2} px", k.fx, cov.std_dev("fx").unwrap());
    println!("corr(fx, cx) = {:+.3}", cov.correlation("fx", "cx").unwrap());
}
}

On this dataset the example reports a mean reprojection error of about 0.13 px and fx ≈ 2058 px with a standard deviation of about 0.3 px.

5. Outlier Filtering

Real data usually contains a few poorly detected corners. run_calibration_with_filtering solves, removes observations above a reprojection-error threshold, and solves again:

#![allow(unused)]
fn main() {
use vision_calibration::planar_intrinsics::{IntrinsicsFilterOptions, run_calibration_with_filtering};

let mut filter_opts = IntrinsicsFilterOptions::default();
filter_opts.max_reproj_error = 1.0;
filter_opts.min_points_per_view = 10;
filter_opts.remove_sparse_views = true;
run_calibration_with_filtering(&mut session, Some(filter_opts))?;
}

Interpreting Results

Key outputs to examine:

• Focal lengths ($f_x$, $f_y$): Should match the expected value based on sensor size and lens focal length.
• Principal point ($c_x$, $c_y$): Should be near the image center. Large offsets may indicate a decentered lens.
• Distortion ($k_1$, $k_2$): Negative $k_1$ indicates barrel distortion (common). $|k_1|>0.3$ suggests a wide-angle lens.
• Reprojection error: <1 px is good. >2 px suggests problems with corner detection or insufficient view diversity.

Comparison with Synthetic Data

Practical Tips

• Use 15-30 images with diverse viewpoints
• Cover the full image area — points only near the center poorly constrain distortion
• Include tilted views — not just frontal. Rotation around both axes constrains all intrinsic parameters
• Check corner ordering — mismatched 2D-3D correspondences cause calibration failure
• Keep fix_k3 on (the default) — only estimate $k_3$ if reprojection error is high and you suspect higher-order radial distortion

Running the Example

cargo run -p vision-calibration --example planar_real --release

The example prints per-step results including initialization accuracy, optimization convergence, the final calibrated parameters and their standard deviations. Its Python counterpart is crates/vision-calibration-py/examples/planar_real.py.

Choosing a Distortion Model

Every workflow that estimates intrinsics picks a lens distortion model and a per-parameter fix list. Get it wrong and you either overfit sparse data (a free k3 on 20 views chases noise) or underfit a genuinely distorted wide-angle lens. This chapter lists the available models, the default fix lists and why they are the defaults, and when reaching for a richer model actually buys anything.

The Models

DistortionKind (vision_calibration::core::DistortionKind) fills the distortion slot of the camera model (camera.model.distortion in every *Config) and selects the packed coefficient layout the factors compile against:

Every workflow calibrates every kind with a straight or a tilted (Scheimpflug) sensor: planar intrinsics, single-camera hand-eye and the laserline device through camera, the rigs per camera (camera plus camera_overrides). The rig laser workflows take each camera of the upstream rig in its own model.

Read the coefficients from export.params.camera.distortion, a DistortionParams with one variant per model (see Serialization and Runtime-Dynamic Types).

The Fix Lists

camera.fix (a CameraFix) names, per slot, the parameters the non-linear solve holds fixed:

• intrinsics: any of fx, fy, cx, cy;
• distortion: the model's own coefficient names;
• sensor: tilt_x, tilt_y.

A slot left None keeps the camera model's default; an explicit list replaces it; an unknown name is a configuration error raised before any work. In JSON: "camera": {"model": {...}, "fix": {"intrinsics": ["cx", "cy"], "distortion": ["k3", "p1", "p2"]}}.

The defaults:

• BrownConrady5: k3 fixed. k3 often overfits sparse or noisy corner data; relax it only for wide-angle lenses or high-quality data. The same rule applies to the linear stage's IntrinsicsInitConfig::fix_k3, which defaults to true for the same reason.
• The extended models: nothing fixed, since they are chosen for their extra terms.
• Any model under a Scheimpflug sensor: everything but the low-order radial terms (k1, k2; the division model's lambda) fixed. See below.

Scheimpflug-Specific Defaults

A Scheimpflug sensor (camera.model.sensor = SensorKind::Scheimpflug) changes two things, both because the sensor tilt couples with distortion:

• The default distortion fix list keeps only k1, k2 (or lambda) free: the tilt can absorb tangential-distortion-like signal, so leaving p1 and p2 free overfits.
• The linear stage holds p1 and p2 at zero while the tilt is estimated, even when the fix list frees them: a free tangential term is ill-posed next to the tilt in the linear fit. The non-linear solve, which separates them, then moves them.

In every workflow the linear stage fits only the distortion terms the camera's solves estimate: a term the fix list holds stays at zero from the start, rather than at a linear fit the later solves would keep.

The tilt itself is the sensor slot of the same fix list (camera.fix.sensor = Some(vec!["tilt_y".into()]) holds tilt_y), since the tilt is a sensor parameter, not a distortion one.

Configuring a Model

Do nothing

PlanarIntrinsicsConfig::default() selects BrownConrady5 with k3 fixed and an untilted sensor. Most datasets should start here.

Choose an extended model

#![allow(unused)]
fn main() {
use vision_calibration::core::DistortionKind;
use vision_calibration::planar_intrinsics::PlanarIntrinsicsConfig;

let mut config = PlanarIntrinsicsConfig::default();
config.camera.model.distortion = DistortionKind::Rational8;
// The extended models fix nothing by default: k1..k6, p1, p2 are all refined.
// Name what to hold, e.g. the higher-order radial block:
config.camera.fix.distortion = Some(vec!["k4".into(), "k5".into(), "k6".into()]);
}

Fix intrinsics or distortion terms

#![allow(unused)]
fn main() {
// Fix cx, cy but optimize fx, fy
session.update_config(|c| {
    c.camera.fix.intrinsics = Some(vec!["cx".into(), "cy".into()]);
})?;

// Fix k3 and the tangential terms
session.update_config(|c| {
    c.camera.fix.distortion = Some(vec!["k3".into(), "p1".into(), "p2".into()]);
})?;
}

A tilted sensor, seeded

#![allow(unused)]
fn main() {
use vision_calibration::core::{DistortionKind, SensorKind};
use vision_calibration::planar_intrinsics::{
    IntrinsicsManualInit, PlanarIntrinsicsConfig, PlanarIntrinsicsProblem, step_init_with_seed,
    step_optimize,
};
use vision_calibration::session::CalibrationSession;

let mut config = PlanarIntrinsicsConfig::default();
config.camera.model.sensor = SensorKind::Scheimpflug;    // radial-only default fix list
config.camera.model.distortion = DistortionKind::ThinPrism9;

let mut session = CalibrationSession::<PlanarIntrinsicsProblem>::new();
session.set_input(dataset)?;
session.set_config(config)?;

let mut seed = IntrinsicsManualInit::default();
seed.intrinsics = Some(nominal_k);   // datasheet focal length / pixel pitch
seed.sensor = Some(mount_tilt);      // mechanical mount angle
step_init_with_seed(&mut session, seed, None)?;
step_optimize(&mut session, None)?;
}

The linear stage always produces Brown-Conrady coefficients; they are embedded into the chosen model with any extra degrees of freedom zeroed before the non-linear refinement. A distortion seed is a value in the chosen model itself (its kind must match camera.model.distortion) and replaces the fitted coefficients as given. See Seeding with Prior Knowledge for the seeding mechanism and Planar Intrinsics Calibration for why a tilted camera starts from a seed.

Does a Richer Model Help?

Under a Scheimpflug sensor, selecting a richer distortion model is not on its own a change to what is optimized, and that surprises people. Mean reprojection error was measured under each model on two six-camera Scheimpflug rigs. Under the radial-only fix list (k1, k2 free), the configuration the shipped baselines use, the model choice alone does not move the floor:

• Rig A: BrownConrady5 = Rational8 = ThinPrism9 = 0.3316 px median mean reprojection error (the extra k4-k6 and s1-s4 coefficients stay at their zero seed under the radial-only list and never move); Division1 = 0.3358 px (stable, but its worst camera sits alone at 0.838 px).
• Rig B: 0.4739 px for the three polynomial models, 0.4745 px for Division1.

This is expected: the radial-only list frees only k1 and k2 (or lambda), so Rational8 and ThinPrism9 are Brown-Conrady in everything but name.

Freeing every coefficient does move the numbers, in both directions. On one rig it improved the hardest camera slightly (0.5008 -> 0.4988 px); on the other it diverged on the two cameras with the thinnest data. An all-free configuration needs a per-camera warm start to be safe, which is why it is not the default.

Practical Guidance

• Default to BrownConrady5 with k3 fixed (or, under a Scheimpflug sensor, the radial-only list) unless you have concrete evidence of higher-order distortion, such as a very wide field of view or a fisheye-adjacent lens. Every committed benchmark baseline stays on BrownConrady5 because, under the production mask, switching the model is a no-op.
• If you reach for an extended model, free its new coefficients deliberately via camera.fix.distortion when the sensor's default holds them. Under a Scheimpflug sensor, selecting the model alone does not unlock any new degrees of freedom.
• Freeing every extra coefficient is experimental: it needs a per-camera warm start to avoid diverging on thin-data views. Reproduce the comparison above on your own dataset before doing it on production data.

Special Cases

• Wide-angle or fisheye-adjacent lenses: set camera.fix.distortion = Some(vec![]) and init.fix_k3 = false so the linear stage also fits k3. This is the one knob the default advice tells you not to flip without a reason.
• DistortionKind::None: pinhole-only, for pre-rectified or synthetic inputs. It is not accepted with a Scheimpflug sensor: the tilt is estimated jointly with the radial terms, so configuration validation rejects None with a typed error.
• Hand-eye and laser workflows: the same camera.model.distortion choice as planar intrinsics. A rig sets it per camera, and the rig laser workflows inherit each camera's model from the upstream rig.

Fisheye Calibration: OpenCV's Test Set

This walkthrough calibrates the fisheye stereo pair OpenCV uses to test its own fisheye module, and sets the result next to OpenCV's. The camera is the Kannala-Brandt fisheye: an equidistant projection with KannalaBrandt4 distortion, the model cv::fisheye calibrates.

The data

The set is calib-3_stereo_from_JY of OpenCV's test data repository (opencv_extra, testdata/cv/cameracalibration/fisheye/): 34 synchronized image pairs, 1280×800, of an 8×6 chessboard with 24.4 mm squares. It ships the chessboard corners with the images, in cv::FileStorage files (left.xml, right.xml and object.xml), and OpenCV's tests calibrate those corners rather than detecting them again. The publisher states no licence for the data.

The regression suite freezes the same corners, so calibration-rs and OpenCV calibrate exactly the same points. Its three entries are opencv_fisheye_left, opencv_fisheye_right (planar_intrinsics) and opencv_fisheye_stereo (rig_extrinsics); the observations are in regression/observations/<id>.json. suite fetch downloads the corner files from a pinned commit of opencv_extra and checks them against their SHA-256 (see Regression Suite).

Rust

The observation file's input is the problem's input, so it deserializes straight into a PlanarDataset. The only setting that differs from a pinhole calibration is the camera model:

#![allow(unused)]
fn main() {
use vision_calibration::core::{
    CameraModelSpec, DistortionKind, PlanarDataset, ProjectionKind, SensorKind,
};
use vision_calibration::planar_intrinsics::{
    PlanarIntrinsicsConfig, PlanarIntrinsicsProblem, step_init, step_optimize,
};
use vision_calibration::session::CalibrationSession;

let text = std::fs::read_to_string("regression/observations/opencv_fisheye_left.json")?;
let mut file: serde_json::Value = serde_json::from_str(&text)?;
let input: PlanarDataset = serde_json::from_value(file["input"].take())?;

let mut config = PlanarIntrinsicsConfig::default();
config.camera.model = CameraModelSpec::new(
    ProjectionKind::Equidistant,
    DistortionKind::KannalaBrandt4,
    SensorKind::Identity,
);
let mut session = CalibrationSession::<PlanarIntrinsicsProblem>::new();
session.set_config(config)?;
session.set_input(input)?;

let init = step_init(&mut session, None)?; // bearing-based fisheye initializer
step_optimize(&mut session, None)?;
let export = session.export()?;
println!("{:?}", export.params.camera.intrinsics);
println!("{:?}", export.params.camera.distortion.values()); // k1..k4
}

step_init runs the bearing-based initializer, which needs no focal-length guess or image size: here it starts at fx = fy = 556.65, 0.3 % from the final value. The stereo pair is a rig_extrinsics calibration with the same config.camera.model; the rig refines both cameras' intrinsics jointly with the extrinsics, as OpenCV's fisheye::stereoCalibrate does. The complete program, left camera and stereo pair, is an example:

cargo run -p vision-calibration --example fisheye_opencv

Python

import json

import vision_calibration as vc

with open("regression/observations/opencv_fisheye_left.json") as f:
    views = json.load(f)["input"]["views"]
dataset = vc.PlanarIntrinsicsInput(
    views=[
        vc.PlanarView(
            observation=vc.Observation(
                points_3d=v["obs"]["points_3d"], points_2d=v["obs"]["points_2d"]
            )
        )
        for v in views
    ]
)
fisheye = vc.CameraModelSpec(projection="equidistant", distortion="kannala_brandt4")
config = vc.PlanarIntrinsicsConfig(camera=vc.CameraConfig(model=fisheye))
result = vc.run_planar_intrinsics(dataset, config)

k, d = result.camera.intrinsics, result.camera.distortion
print(f"fx={k.fx:.3f} fy={k.fy:.3f} cx={k.cx:.3f} cy={k.cy:.3f}")
print(f"k1..k4 = {d.k1:+.6f} {d.k2:+.6f} {d.k3:+.6f} {d.k4:+.6f}")
print(f"mean reprojection error {result.mean_reproj_error:.4f} px")

The stereo pair takes a vc.RigExtrinsicsInput of vc.RigExtrinsicsViews (one vc.Observation per camera) and a vc.RigExtrinsicsConfig(camera=vc.CameraConfig(model=fisheye)), run with vc.run_rig_extrinsics.

Against OpenCV

OpenCV's fisheyeTest.Calibration calibrates the left camera from the same corners with CALIB_RECOMPUTE_EXTRINSIC | CALIB_CHECK_COND | CALIB_FIX_SKEW and checks the result against stored values. The suite's OpenCV harness runs cv2.fisheye.calibrate with the same flags:

The two libraries agree to within 0.01 of our standard deviation on every parameter, and OpenCV's result is the one its own test expects. The distortion coefficients differ in the third digit, but they are strongly correlated (k2, k3 and k4 each have a σ larger than its value), and both fits reach the same mean reprojection error.

The stereo pair is compared with fisheye::stereoCalibrate, which OpenCV's fisheyeTest.stereoCalibrate runs with the same flags. Both refine the two cameras' intrinsics with the rig and land on the same solution: the translation from the left camera to the right is T = (−0.099403, 0.002708, 0.001293) m (a 99.45 mm baseline) in both, the focal lengths and principal points agree to 0.001 px, and the distortion coefficients within 0.01σ.

calib-bench suite compare prints these comparisons for every entry; the table is in Regression Suite.

Single-Camera Hand-Eye Calibration

Hand-eye calibration estimates the rigid transform between a camera and a robot's gripper (or base). This workflow combines intrinsics calibration with hand-eye estimation in a 4-step pipeline.

Problem Formulation

Transformation Chain (Eye-in-Hand)

For a camera mounted on the robot gripper:

$$T_{C,T}=T_{C,G}\cdot T_{G,B}\cdot T_{B,T}$$

where:

• $T_{C,G}=T_{\text{handeye}}^{-1}$: camera-to-gripper (the calibrated hand-eye transform, inverted)
• $T_{G,B}=T_{\text{robot}}^{-1}$: gripper-to-base (known from robot kinematics, inverted)
• $T_{B,T}$: base-to-target (the target's pose in the robot base frame, also calibrated)

Equivalently:

$$T_{C,T}=(T_{\text{robot}}\cdot T_{\text{handeye}}{)}^{-1}\cdot T_{B,T}$$

Eye-to-Hand Variant

For a fixed camera observing a target on the gripper:

$$T_{C,T}=T_{C,B}\cdot T_{B,G}\cdot T_{G,T}$$

where $T_{C,B}$ is the camera-to-base transform and $T_{G,T}$ is the target-to-gripper transform.

Optimization Objective

Minimize reprojection error across all views, jointly over all parameters:

$$\underset{K,\mathbf{d},X,T_{B,T}}{\min }\sum _{v=1}^M\sum _{j=1}^{N_v}{\Vert \pi \left(K,\mathbf{d},T_{C,T}^{(v)},{\mathbf{P}}_j\right)-{\mathbf{p}}_{vj}\Vert }^2$$

where $T_{C,T}^{(v)}$ depends on the known robot pose for view $v$, the hand-eye transform $X$, and the target pose $T_{B,T}$.

4-Step Pipeline

Step 1: Intrinsics Initialization (step_intrinsics_init)

Runs the Iterative Intrinsics solver to estimate $K$ and distortion from the calibration board observations, ignoring robot poses entirely.

Step 2: Intrinsics Optimization (step_intrinsics_optimize)

Runs planar intrinsics bundle adjustment to refine $K$, distortion, and per-view camera-to-target poses. This gives accurate per-view poses $T_{C,T}^{(v)}$ needed for hand-eye initialization.

Step 3: Hand-Eye Initialization (step_handeye_init)

Solves the hand-eye and the target pose in closed form with the method of handeye_init.method: Tsai-Lenz by default, or Park-Martin, Daniilidis or Shah (see Hand-Eye Linear):

1. Compute relative motions from robot poses and camera poses
2. Filter pairs by rotation magnitude (reject small rotations)
3. Estimate the hand-eye rotation and translation
4. Estimate $T_{B,T}$: through the first view, or over all views with Shah's method, which estimates it together with the hand-eye

Step 4: Hand-Eye Optimization (step_handeye_optimize)

Joint non-linear optimization of all parameters:

Parameter blocks:

• "cam" (4D, Euclidean): intrinsics
• "dist" (Euclidean): the distortion coefficients of the configured model (absent without distortion)
• "sensor" (2D, Euclidean): the sensor tilt, for a Scheimpflug camera
• "handeye" (7D, SE3): gripper-to-camera transform
• "target" (7D, SE3): base-to-target transform

Residual blocks: one ReprojPoint { model, chain: Handeye { .. }, .. } per observation, model being the camera's CameraModelSpec.

Without robot-pose refinement the robot poses are not optimized: they are known constants from the robot kinematics (passed as per-residual data).

Robot Pose Refinement

Robot poses carry errors of their own, usually larger than the corner noise once projected through the lever arm to the board: 0.05° at 1 m moves the camera's view by almost a millimetre. Taken as exact, these errors end up in the camera parameters. robot_poses.refine (on by default) gives every view an SE(3) correction of its measured pose:

Additional parameter blocks: "robot_delta/v" (6D, Euclidean se(3) tangent [ω, v]) per view, applied in the gripper frame as base_se3_gripper · exp(δ): ω is the tool's orientation error and v its position error

Additional residuals: a zero-mean Se3TangentPrior per view, with the robot's rotation and translation standard deviations σ_r (rot_sigma) and σ_t (trans_sigma), weighed against the corner residuals at their pixel noise σ_px:

$${\mathbf{r}}_{\text{prior}}={\sigma }_{px}\left[\begin{array}{c}\mathbf{\omega }/{\sigma }_r\\ \mathbf{v}/{\sigma }_t\end{array}\right]$$

The objective is then the maximum-likelihood one, in units of σ_px²: a correction costs as much as the corner evidence it explains away. σ_px is not known in advance, so the solve runs once at 1 px, takes σ_px from the corner residuals and runs again from that solution. Every view's correction is free; the priors are what tell the corrections apart from a change of the hand-eye or target pose that would move every view alike.

The priors' standard deviations decide how far the result trusts the robot. Too loose, and the hand-eye and target standard deviations come out larger than the data warrants; too tight, and the robot's errors leak into the camera parameters while the standard deviations come out too small. So by default (robot_poses.estimate_sigma) the solve estimates them: rot_sigma and trans_sigma (0.5°, 1 mm) are only its starting values. Each prior's variance factor is the sum of its squared normalized corrections over its redundancy, ŝ² = Σ (δₖ/σ)² / Σ (1 − Var(δₖ)/σ²), the share of each correction the corner data do not already determine. σ is scaled by ŝ and the solve repeats until both factors are 1. The export reports the estimate as robot_pose_sigma, the robot's orientation and position error in this calibration. On synthetic scenes it recovers the simulated error from a prior ten times too tight or too loose, and the normalized errors of the hand-eye and target come out with an RMS of 1.02–1.13. With estimate_sigma off, the configured values are taken as the robot's true accuracy.

A correction common to every view looks exactly like a change of the transform carried by the gripper: the hand-eye in eye-in-hand, the target's pose in the gripper in eye-to-hand. That transform is therefore known only as well as the robot's mean error over the views: its rotation's standard deviation is at least about σ_r / √views, its translation's σ_t / √views.

Configuration

Settings are grouped by stage; the shared sub-structs (IntrinsicsInitConfig, CameraConfig, HandeyeInitConfig, SolverConfig, RobotPoseConfig) are reused across every hand-eye problem type:

#![allow(unused)]
fn main() {
pub struct SingleCamHandeyeConfig {
    // Per-camera linear-initialization stage settings.
    pub intrinsics: IntrinsicsInitConfig,
    // { iterations: usize = 2, fix_k3: bool = true }; it fits only the
    //   distortion terms the camera's solves estimate

    // The camera model (any distortion kind, an identity or Scheimpflug
    // sensor) and the parameters its solves hold fixed, named per slot.
    pub camera: CameraConfig, // { model: CameraModelSpec, fix: CameraFix }

    // Hand-eye linear-initialization stage settings (Tsai-Lenz DLT).
    pub handeye_init: HandeyeInitConfig,
    // { handeye_mode: HandeyeMode, method: HandeyeMethod = TsaiLenz,
    //   min_motion_angle_deg: f64 = 5.0 }

    // Non-linear solve stage settings.
    pub solver: SolverConfig, // { max_iters, verbosity, robust_loss }

    // Robot-pose refinement settings for the hand-eye bundle adjustment.
    pub robot_poses: RobotPoseConfig,
    // { refine: bool = true, rot_sigma: f64, trans_sigma: f64 }
}
}

Input Format

Each view is a View<RobotPoseMeta> (aliased as SingleCamHandeyeView); RobotPoseMeta is in vision_calibration::common, shared with the rig hand-eye workflows:

#![allow(unused)]
fn main() {
pub struct RobotPoseMeta {
    pub base_se3_gripper: Iso3,  // Robot pose (from kinematics)
}

// Type alias
pub type SingleCamHandeyeView = View<RobotPoseMeta>;
// Where View<Meta> has fields: obs: CorrespondenceView, meta: Meta
}

Mounting Modes

Poses follow the frame_se3_frame naming: base_se3_gripper reads "gripper pose expressed in the base frame" ($T_{B,G}$). The unknowns are named the same way. The mounting mode is HandeyeInitConfig::handeye_mode:

• HandeyeMode::EyeInHand (the default): the camera rides on the gripper and observes a fixed target. The calibration solves for gripper_se3_camera ($T_{G,C}$) and base_se3_target ($T_{B,T}$).
• HandeyeMode::EyeToHand: the camera is fixed in the scene and observes a target mounted on the gripper. The calibration solves for camera_se3_base ($T_{C,B}$) and gripper_se3_target ($T_{G,T}$).

The export's four pose fields are mutually exclusive by mode: the active mode fills its pair and leaves the other pair None.

To switch to eye-to-hand:

#![allow(unused)]
fn main() {
use vision_calibration::optim::HandeyeMode;
use vision_calibration::single_cam_handeye::SingleCamHandeyeConfig;

let mut config = SingleCamHandeyeConfig::default();
config.handeye_init.handeye_mode = HandeyeMode::EyeToHand;
session.set_config(config)?;
}

In eye-to-hand mode the pixel-to-3D chain for a fixed camera observing a gripper-mounted target composes as $T_{C,T}=T_{C,B}\cdot T_{B,G}\cdot T_{G,T}$, the same identity step_handeye_init_with_seed uses to derive camera_se3_base when only gripper_se3_target is seeded (see Seeding with Prior Knowledge).

Complete Example

#![allow(unused)]
fn main() {
use vision_calibration::prelude::*;
use vision_calibration::single_cam_handeye::*;

let views: Vec<SingleCamHandeyeView> = /* load data */;
let input = SingleCamHandeyeInput::new(views)?;

let mut session = CalibrationSession::<SingleCamHandeyeProblem>::new();
session.set_input(input)?;

step_intrinsics_init(&mut session, None)?;
step_intrinsics_optimize(&mut session, None)?;
step_handeye_init(&mut session, None)?;
step_handeye_optimize(&mut session, None)?;

let export = session.export()?;
println!("Camera: {:?}", export.camera); // CameraParams, in the configured model
println!("Hand-eye and target: {:?}", export.handeye);
println!("Reprojection error: {:.4} px", export.mean_reproj_error);
}

Reading the Export

#![allow(unused)]
fn main() {
use vision_calibration::single_cam_handeye::SingleCamHandeyeSolution;

match export.handeye {
    SingleCamHandeyeSolution::EyeInHand { gripper_se3_camera, base_se3_target } => {
        // gripper_se3_camera: T_G_C, base_se3_target: T_B_T
    }
    SingleCamHandeyeSolution::EyeToHand { camera_se3_base, gripper_se3_target } => {
        // camera_se3_base: T_C_B, gripper_se3_target: T_G_T
    }
}
}

export.handeye holds exactly the two poses of the mode, so an export cannot carry the wrong pose for its mode. In JSON it is the handeye object, tagged by mode:

"handeye": {
  "mode": "EyeInHand",
  "gripper_se3_camera": { "rotation": [0, 0, 0, 1], "translation": [0.05, 0, 0.1] },
  "base_se3_target":    { "rotation": [0, 0, 0, 1], "translation": [0, 0, 1] }
}

In Python, result.handeye is a CameraEyeInHand or a CameraEyeToHand, and result.handeye.mode names the mode.

export.camera is the calibrated camera (CameraParams) in the configured model, export.robot_deltas the per-view robot-pose corrections when refinement is on, and export.diagnostics.per_feature_residuals the per-corner reprojection records (see Per-Feature Residuals).

The SingleCamHandeyeInput::new constructor rejects an empty set of views and any view with fewer than 4 correspondences (the minimum for a homography). Each stage has a *_with_seed sibling and a typed step result (IntrinsicsInitResult, HandeyeOptimizeResult, ...); see Seeding with Prior Knowledge.

From a Dataset Manifest

Instead of assembling the views by hand, a manifest with topology = "single_cam_handeye" describes where the images, the target and the robot-pose file live. vision_calibration::dataset_runner::build_single_cam_handeye_input runs the (cached) corner detection and returns the SingleCamHandeyeInput; see The Dataset Manifest.

Camera Models

The camera model is the config's camera.model: any distortion kind (Brown-Conrady by default, rational, thin-prism, division, or none) and an identity or Scheimpflug sensor. The intrinsics stage dispatches on it like Planar Intrinsics: Zhang's method embedded into the model's distortion, and for a tilted sensor the linear tilt sweep followed by the staged tilt multi-start. A known tilt goes in the intrinsics manual-init sensor seed (step_intrinsics_init_with_seed); a tilt seed for an untilted camera must be zero. camera.fix names the parameters both solves hold.

Practical Considerations

• Rotation diversity is critical for Tsai-Lenz initialization. Include rotations around at least 2 axes.
• 5-10 views minimum, with 10-20 recommended for robustness.
• Robot accuracy: If the robot's reported poses have errors > 1mm or > 0.1°, enable robot pose refinement.
• Initialization failure: If Tsai-Lenz produces a poor estimate (large reprojection error after optimization), the most likely cause is insufficient rotation diversity.

OpenCV equivalence: cv::calibrateHandEye provides the linear initialization step. calibration-rs adds joint non-linear refinement and optional robot pose correction.

Hand-Eye with KUKA Robot

This chapter walks through the handeye_session example, which performs eye-in-hand calibration using a real KUKA robot dataset.

Dataset

The kuka_1 sample, which the example downloads on the first run (sample datasets):

• Images: 30 PNG files (01.png through 30.png)
• Robot poses: RobotPosesVec.txt — one row-major $4\times 4$ homogeneous matrix per line (16 whitespace-separated values), each the pose $T_{B,G}$ of the gripper in the robot base frame, in metres. Image NN.png pairs with pose row NN.
• Board square size: squaresize.txt (20mm)
• Calibration board: 17×28 chessboard, 20 mm squares

Data Loading

The example's support module (examples/support/handeye_io.rs) reads the square size and the robot poses, detects chessboard corners in each image with the calib_targets chessboard detector, and pairs each successful detection with its robot pose. Images where detection fails are skipped.

Each pose line is converted to an Iso3:

#![allow(unused)]
fn main() {
let r = Matrix3::new(
    values[0], values[1], values[2],
    values[4], values[5], values[6],
    values[8], values[9], values[10],
);
let t = Vector3::new(values[3], values[7], values[11]);
let rot = Rotation3::from_matrix_unchecked(r);
let pose = Iso3::from_parts(Translation3::from(t), UnitQuaternion::from_rotation_matrix(&rot));
}

The detected views become the calibration input:

#![allow(unused)]
fn main() {
use vision_calibration::common::RobotPoseMeta;
use vision_calibration::single_cam_handeye::{
    SingleCamHandeyeInput, SingleCamHandeyeProblem, SingleCamHandeyeView,
};

let views: Vec<SingleCamHandeyeView> = samples
    .into_iter()
    .map(|s| SingleCamHandeyeView {
        obs: s.view,                                          // CorrespondenceView
        meta: RobotPoseMeta { base_se3_gripper: s.robot_pose }, // T_B_G
    })
    .collect();

let input = SingleCamHandeyeInput::new(views)?;
}

Calibration

#![allow(unused)]
fn main() {
use vision_calibration::single_cam_handeye::{
    step_handeye_init, step_handeye_optimize, step_intrinsics_init, step_intrinsics_optimize,
};

let mut session = CalibrationSession::<SingleCamHandeyeProblem>::new();
session.set_input(input)?;

step_intrinsics_init(&mut session, None)?;
let intr_opt = step_intrinsics_optimize(&mut session, None)?;
step_handeye_init(&mut session, None)?;
let he_opt = step_handeye_optimize(&mut session, None)?;

let export = session.export()?;
}

Running the Example

cargo run -p vision-calibration --example handeye_session

The example reports:

1. Dataset summary (total images, used views, skipped views)
2. Per-step results (initialization, optimization)
3. Final calibrated parameters (intrinsics, distortion, hand-eye transform, target pose)

Output on kuka_1 (30 chessboard views, eye-in-hand, the default mode), abridged:

--- Step 2: Intrinsics Optimization ---
  Intrinsics: fx=2057.8, fy=2058.4, cx=962.1, cy=609.9
  Reprojection error: 0.1346 px

--- Step 3: Hand-Eye Initialization (Tsai-Lenz) ---
  Hand-eye |t|: 0.2806m

--- Step 4: Hand-Eye Optimization ---
  Final reprojection error: 0.1346 px

Robot pose refinements (30 views):
  Max rotation delta: 0.0022 rad (0.12°)
  Max translation delta: 0.0013m (1.31mm)
Estimated robot error: 0.043° / 0.45mm per axis

The joint hand-eye error matches the intrinsics-only error: every view is explained through one shared gripper_se3_camera and one shared base_se3_target, and the small per-view robot-pose corrections (0.12° and 1.3 mm at most) absorb the kinematic noise, and the solve estimates the arm's error from them: 0.043° and 0.45 mm per axis. A correction acts in the gripper frame, so its rotation and translation are the tool's orientation and position errors. A hand-eye error far above the intrinsics-only error points at robot-pose noise, mechanical play or a wrong pose convention.

Interpreting Results

Key outputs of the export (eye-in-hand):

• Hand-eye transform (gripper_se3_camera): translation magnitude should match the physical camera-to-gripper distance. Rotation should reflect the mounting orientation.
• Target in base (base_se3_target): the calibration board's position in the robot base frame. Verify against the known physical setup.
• Reprojection error (mean_reproj_error): under 1 px indicates a good calibration; above 3 px suggests problems.

Common Failure Modes

1. Insufficient rotation diversity: All robot poses rotate around the same axis (e.g., only wrist rotation). The Tsai-Lenz initialization will fail or produce a poor estimate.

2. Incorrect pose convention: Robot poses must be $T_{B,G}$ (base_se3_gripper). If the convention is inverted, the calibration will diverge.

3. Mismatched corner ordering: If the chessboard detector assigns corners in a different order than expected, the 2D-3D correspondences are wrong.

4. Robot pose timestamps: If images and robot poses are not synchronized, the calibration will fail.

Data Collection Recommendations

• Rotation diversity: Include poses with significant roll, pitch, and yaw. Avoid pure translations or rotations around a single axis.
• Target visibility: Ensure the calibration board is fully visible in all images. Partial visibility causes corner detection failure.
• Stable poses: Take images when the robot is stationary. Motion blur degrades corner detection.
• Coverage: Vary the distance to the board and the position within the image.

Multi-Camera Rig Extrinsics

A multi-camera rig is a set of cameras rigidly attached to a common frame. Rig extrinsics calibration estimates the relative pose of each camera within the rig, in addition to per-camera intrinsics and distortion.

Problem Formulation

Parameters

• Per-camera intrinsics: $\{K_k\}$ — one per camera
• Per-camera distortion: $\{{\mathbf{d}}_k\}$ — one per camera
• Per-camera extrinsics: $\{T_{R,C_k}\}$ — camera-to-rig transforms (one per camera, reference camera = identity)
• Per-view rig poses: $\{T_{R,T}^{(v)}\}$ — rig-to-target transforms (one per view)

Transform Chain

For camera $k$ in view $v$:

$$T_{C_k,T}^{(v)}=T_{C_k,R}\cdot T_{R,T}^{(v)}$$

where $T_{C_k,R}=(T_{R,C_k}{)}^{-1}$.

Objective

$$\min \sum _{k=1}^C\sum _{v=1}^M\sum _{j=1}^{N_{kv}}{\Vert \pi \left(K_k,{\mathbf{d}}_k,T_{C_k,R}\cdot T_{R,T}^{(v)},{\mathbf{P}}_j\right)-{\mathbf{p}}_{kvj}\Vert }^2$$

This jointly optimizes all cameras' parameters and all rig geometry. Each observation depends on two SE(3) transforms composed together (a ReprojPoint factor with the TwoSe3 chain).

4-Step Pipeline

Step 1: Per-Camera Intrinsics Initialization (step_intrinsics_init_all)

For each camera independently:

1. Compute homographies from the views where that camera has observations
2. Run iterative intrinsics estimation (Zhang + distortion fit)
3. Recover per-camera, per-view poses

Step 2: Per-Camera Intrinsics Optimization (step_intrinsics_optimize_all)

For each camera independently:

• Run planar intrinsics bundle adjustment
• Refine $K_k$, ${\mathbf{d}}_k$, and per-view poses

Step 3: Rig Extrinsics Initialization (step_rig_init)

Uses the linear initialization from Multi-Camera Rig Initialization:

1. Define the rig frame by the reference camera ($T_{R,C_0}=I$)
2. For each non-reference camera, compute $T_{R,C_k}$ by averaging across views
3. Compute rig-to-target poses from the reference camera's poses

Step 4: Rig Bundle Adjustment (step_rig_optimize)

Joint optimization of all parameters:

Parameter blocks:

• "cam/k" (4D): per-camera intrinsics
• "dist/k" (5D for Brown-Conrady): per-camera distortion
• "extrinsics/k" (7D, SE3): per-camera camera-to-rig transforms
• "rig_pose/v" (7D, SE3): per-view rig-to-target poses

Residual blocks: one ReprojPoint { model: <camera's CameraModelSpec>, chain: TwoSe3, .. } per observation. A tilted camera adds a "sensor/k" block (2D); "dist/k" has the size of the camera's distortion model.

Fixed parameters: The reference camera's extrinsics are fixed at identity (gauge freedom).

Configuration

Settings are grouped by stage; the shared sub-structs (IntrinsicsInitConfig, RigConfig, SolverConfig) are reused across every rig-family problem type:

#![allow(unused)]
fn main() {
pub struct RigExtrinsicsConfig {
    // Per-camera linear-initialization stage settings.
    pub intrinsics: IntrinsicsInitConfig, // { iterations, fix_k3 }
    // (fits only the distortion terms each camera's solves estimate)

    // Camera model shared by every camera (projection, distortion, sensor)
    // and its named fix lists; default Brown-Conrady, identity sensor.
    pub camera: CameraConfig, // { model: CameraModelSpec, fix: CameraFix }

    // Per-camera replacements of `camera`, for rigs that mix lenses or sensors.
    pub camera_overrides: Vec<CameraOverride>, // { camera, model, fix }

    // Rig frame options.
    pub rig: RigConfig, // { reference_camera_idx, refine_intrinsics, refine_sensor }

    // Non-linear solve stage settings.
    pub solver: SolverConfig, // { max_iters, verbosity, robust_loss }
}
}

Every camera uses camera unless camera_overrides names it. An override replaces the model and the fix lists of one camera, so a rig can mix distortion kinds and sensors. A camera index out of range or listed twice is a validation error. A slot left out of fix keeps the model's default. With rig.refine_intrinsics set, each camera's fix lists are held during the rig bundle adjustment.

#![allow(unused)]
fn main() {
use vision_calibration::common::config::CameraOverride;
use vision_calibration::core::{CameraModelSpec, DistortionKind, SensorKind};

// Camera 1 has a tilted sensor and a rational lens; the others are straight Brown-Conrady.
session.update_config(|c| {
    c.camera_overrides = vec![CameraOverride::new(
        1,
        CameraModelSpec::pinhole(DistortionKind::Rational8, SensorKind::Scheimpflug),
    )];
})?;
}

A rig of tilted cameras sets config.camera = CameraConfig::new(CameraModelSpec::pinhole(DistortionKind::BrownConrady5, SensorKind::Scheimpflug)) instead. Tilt seeds go through IntrinsicsManualInit::per_cam_sensors and step_intrinsics_init_all_with_seed; zero tilts need no seed (see Scheimpflug and Laser Rigs). The export's cameras is one CameraParams per camera, stating the calibrated distortion and sensor.

The reference camera (rig.reference_camera_idx) is fixed at identity, which removes the rig's 6-DOF gauge freedom; no separate pose-gauge setting is needed.

Input Format

#![allow(unused)]
fn main() {
pub struct RigDataset<Meta> {
    pub num_cameras: usize,
    pub views: Vec<RigView<Meta>>,
}

pub struct RigView<Meta> {
    pub obs: RigViewObs,  // Per-camera observations (Option per camera)
    pub meta: Meta,
}
}

Each view has an Option<CorrespondenceView> per camera — cameras that don't see the target in a given view have None.

Per-Camera Reprojection Error

After optimization, per-camera reprojection errors are computed independently:

#![allow(unused)]
fn main() {
let export = session.export()?;
for (cam_idx, error) in export.per_cam_reproj_errors.iter().enumerate() {
    println!("Camera {}: {:.4} px", cam_idx, error);
}
println!("Overall: {:.4} px", export.mean_reproj_error);
}

Stereo Baseline

For a 2-camera rig, the baseline (distance between camera centers) is:

$$\text{baseline}=\Vert T_{R,C_1}.\text{translation}\Vert$$

since $T_{R,C_0}=I$ (camera 0 is the reference).

Complete Example

#![allow(unused)]
fn main() {
use vision_calibration::prelude::*;
use vision_calibration::rig_extrinsics::*;

let mut session = CalibrationSession::<RigExtrinsicsProblem>::new();
session.set_input(rig_dataset)?;

step_intrinsics_init_all(&mut session, None)?;
step_intrinsics_optimize_all(&mut session, None)?;
step_rig_init(&mut session, None)?;
step_rig_optimize(&mut session, None)?;

let export = session.export()?;
let baseline = export.cam_se3_rig[1].translation.vector.norm();
println!("Stereo baseline: {:.1} mm", baseline * 1000.0);
}

OpenCV equivalence: cv::stereoCalibrate for 2-camera rigs. calibration-rs generalizes to $N$ cameras.

Multi-Camera Rig Hand-Eye

This is the most complex calibration workflow: a multi-camera rig mounted on a robot arm. It combines per-camera intrinsics calibration, rig extrinsics estimation, and hand-eye calibration in a 6-step pipeline.

Problem Formulation

Transformation Chain

Frames follow the frame_se3_frame convention: T_{A,B} maps coordinates of frame $B$ into frame $A$. For camera $k$ in view $v$ with robot pose $T_{B,G}^{(v)}$ (eye-in-hand):

$$T_{C_k,T}^{(v)}=T_{C_k,R}\cdot T_{G,R}^{-1}\cdot (T_{B,G}^{(v)}{)}^{-1}\cdot T_{B,T}$$

where:

• $T_{C_k,R}$ (cam_se3_rig): rig to camera $k$ (rig extrinsics)
• $T_{G,R}$ (gripper_se3_rig): rig to gripper (the hand-eye transform)
• $T_{B,G}^{(v)}$ (base_se3_gripper): known robot pose for view $v$
• $T_{B,T}$ (base_se3_target): target in the robot base frame (calibrated)

In eye-to-hand mode the export instead carries rig_se3_base and gripper_se3_target.

Parameters

• Per-camera intrinsics: $\{K_k\}$ ($4C$ scalar parameters)
• Per-camera distortion: $\{{\mathbf{d}}_k\}$ ($5C$ scalar parameters)
• Per-camera extrinsics: $\{T_{C_k,R}\}$ ($6(C-1)$ DOF, reference camera = identity)
• Hand-eye: $T_{G,R}$ (6 DOF)
• Target pose: $T_{B,T}$ (6 DOF)
• Optionally: per-view robot corrections $\{\Delta T_v\}$ ($6M$ DOF, regularized)

6-Step Pipeline

Steps 1-2: Per-Camera Intrinsics

Same as Rig Extrinsics: initialize and optimize each camera's intrinsics independently.

Steps 3-4: Rig Extrinsics

Same as Rig Extrinsics: initialize camera-to-rig transforms via SE(3) averaging, then jointly optimize the rig geometry.

Step 5: Hand-Eye Initialization (step_handeye_init)

Uses the closed-form method of handeye_init.method (Tsai-Lenz by default; see Hand-Eye Linear) with the rig's poses and the robot poses:

1. Extract relative camera motions from rig-to-target poses
2. Extract relative robot motions from base-to-gripper poses
3. Solve $AX=XB$ for $X=T_{G,R}$ (gripper_se3_rig)
4. Estimate the target in the base frame, $T_{B,T}$

Step 6: Hand-Eye Optimization (step_handeye_optimize)

Joint optimization of all parameters:

Parameters: per-camera intrinsics and distortion, per-camera camera-to-rig extrinsics (SE3), the hand-eye transform (SE3), and the target pose (SE3).

Factor: one ReprojPoint with a rig hand-eye chain per observation, which composes the full transform chain.

Configuration

RigHandeyeConfig groups the settings by stage:

#![allow(unused)]
fn main() {
pub struct RigHandeyeConfig {
    pub intrinsics: IntrinsicsInitConfig,   // per-camera linear init
    pub manual_init: Option<IntrinsicsManualInit>, // optional per-camera seeds
    pub camera: CameraConfig,               // camera model + fix lists shared by every camera
    pub camera_overrides: Vec<CameraOverride>, // per-camera replacement of `camera`
    pub rig: RigConfig,                     // reference_camera_idx, refine_intrinsics, refine_sensor
    pub handeye_init: HandeyeInitConfig,    // handeye_mode, method, min_motion_angle_deg
    pub solver: SolverConfig,               // max_iters, verbosity, robust_loss
    pub handeye_optimize: HandeyeOptimizeConfig, // robot_poses, refine_cam_se3_rig, refine_sensor
}
}

#![allow(unused)]
fn main() {
session.update_config(|c| {
    c.handeye_init.handeye_mode = HandeyeMode::EyeInHand;
    c.solver.robust_loss = RobustLoss::Huber { scale: 2.0 };
    c.handeye_optimize.robot_poses.refine = true;
})?;
}

Set camera.model.sensor to SensorKind::Scheimpflug for a rig of tilted-sensor cameras, or name single cameras in camera_overrides to mix lenses (see Scheimpflug and Laser Rigs). The export's cameras holds one CameraParams per camera.

Complete Example

#![allow(unused)]
fn main() {
use vision_calibration::prelude::*;
use vision_calibration::rig_handeye::*;
use vision_calibration::common::HandeyeSolution;
use vision_calibration::optim::{HandeyeMode, RobustLoss};

let mut session = CalibrationSession::<RigHandeyeProblem>::new();
session.set_input(rig_dataset_with_robot_poses)?;

// Per-camera calibration
step_intrinsics_init_all(&mut session, None)?;
step_intrinsics_optimize_all(&mut session, None)?;

// Rig geometry
step_rig_init(&mut session, None)?;
step_rig_optimize(&mut session, None)?;

// Hand-eye
step_handeye_init(&mut session, None)?;
step_handeye_optimize(&mut session, None)?;

let export = session.export()?;
match export.handeye {
    HandeyeSolution::EyeInHand { gripper_se3_rig, base_se3_target } => {
        println!("gripper_se3_rig: {gripper_se3_rig:?}, base_se3_target: {base_se3_target:?}");
    }
    HandeyeSolution::EyeToHand { rig_se3_base, gripper_se3_target } => {
        println!("rig_se3_base: {rig_se3_base:?}, gripper_se3_target: {gripper_se3_target:?}");
    }
}
println!("Baseline: {:.1} mm",
    export.cam_se3_rig[1].translation.vector.norm() * 1000.0);
println!("Per-camera errors: {:?}", export.per_cam_reproj_errors);
}

Gauge Freedom

The system has a gauge freedom: the rig frame origin and the hand-eye transform are coupled. Fixing the reference camera's extrinsics at identity resolves this by defining the rig frame to coincide with camera 0.

Practical Considerations

All the advice from Single-Camera Hand-Eye applies, plus:

• All cameras must observe the target in at least some views for the rig extrinsics to be well-constrained
• Views where only some cameras see the target are handled (missing observations are skipped)
• The hand-eye transform (gripper_se3_rig) relates the gripper and the rig frame, not the gripper and an individual camera. The per-camera offset comes from the rig extrinsics.

Laserline Device Calibration

Laserline calibration jointly estimates camera parameters and a laser plane from observations of both a calibration board and laser line projections on the board. This is used in laser triangulation systems where a camera and laser are rigidly mounted together.

Problem Formulation

Parameters

• Camera intrinsics: $K=(f_x,f_y,c_x,c_y)$
• Distortion: the coefficients of the configured model, e.g. $\mathbf{d}=(k_1,k_2,k_3,p_1,p_2)$ for Brown-Conrady
• Sensor tilt: $({\tau }_x,{\tau }_y)$ (Scheimpflug cameras only)
• Per-view poses: $\{T_v\}$ (camera-to-target SE(3))
• Laser plane: normal $\hat{\mathbf{n}}\in S^2$, distance $d\in \mathbb{R}$

Observations

Each view provides two types of observations:

1. Calibration points: 2D-3D correspondences from the chessboard (same as planar intrinsics)
2. Laser pixels: 2D pixel positions where the laser line appears on the target

Objective

The cost function has two terms:

$$F=w_c\sum _{\text{calib}}{\rho }_c\left(\Vert \pi (K,\mathbf{d},S,T_v,{\mathbf{P}}_j)-{\mathbf{p}}_{vj}\Vert \right)+w_l\sum _{\text{laser}}{\rho }_l\left(r_{\text{laser}}(\cdot )\right)$$

where $w_c,w_l$ are weights, ${\rho }_c,{\rho }_l$ are (possibly different) robust loss functions, and $r_{\text{laser}}$ is the laser residual.

Laser Residual Types

Two approaches are implemented for the laser residual, selectable via configuration.

PointToPlane (3D Distance)

Algorithm:

1. Undistort laser pixel to normalized coordinates
2. Back-project to a ray in camera frame
3. Intersect the ray with the target plane (at $Z_{\text{target}}=0$, transformed by pose $T_v$) to get a 3D point ${\mathbf{P}}_c$
4. Compute signed distance from ${\mathbf{P}}_c$ to the laser plane:

$$r=\sqrt{w}\cdot ({\hat{\mathbf{n}}}^T{\mathbf{P}}_c-d)$$

Residual dimension: 1 (meters)

LineDistNormalized (2D Line Distance) — Default

Algorithm:

1. Compute the 3D intersection line of the laser plane and the target plane (in camera frame)
2. Project this 3D line onto the $Z=1$ normalized camera plane
3. Undistort the laser pixel to normalized coordinates (done once, not per-iteration)
4. Measure the perpendicular distance from the undistorted pixel to the projected line
5. Scale by $\sqrt{f_x\cdot f_y}$ for pixel-comparable units:

$$r=\sqrt{w}\cdot d_{\perp }\cdot \sqrt{f_x{f}_y}$$

Residual dimension: 1 (effective pixels)

Comparison

Both approaches yield similar accuracy in practice (<6% intrinsics error, <5° plane normal error in synthetic tests).

Derivation: Line-Distance Residual

Plane-Plane Intersection Line

The laser plane ${\hat{\mathbf{n}}}^T\mathbf{P}=d$ and the target plane (normal $\hat{\mathbf{m}}$, distance $e$ from camera, derived from pose $T_v$) intersect in a 3D line with:

$$\text{direction: }\hat{\mathbf{l}}=\hat{\mathbf{n}}\times \hat{\mathbf{m}}$$

$$\text{origin: solved from }\left[\begin{array}{c}{\hat{\mathbf{n}}}^T\\ {\hat{\mathbf{m}}}^T\end{array}\right]{\mathbf{P}}_0=\left[\begin{array}{c}d\\ e\end{array}\right]$$

Projection onto Normalized Plane

Project the 3D line to the $Z=1$ plane:

$${\mathbf{p}}_0=\text{point on line at }Z=1$$ $${\hat{\mathbf{d}}}_{\text{2D}}=\text{direction projected to }Z=1\text{ plane}$$

Perpendicular Distance

For an undistorted pixel $\mathbf{q}$ in normalized coordinates, the perpendicular distance to the 2D line is:

$$d_{\perp }=\frac{|(\mathbf{q}-{\mathbf{p}}_0)\times {\hat{\mathbf{d}}}_{\text{2D}}|}{|{\hat{\mathbf{d}}}_{\text{2D}}|}$$

Configuration

init and solver are the shared sub-structs, camera the camera model and its fixed parameters (as in Planar Intrinsics), and optimize groups the laser-specific bundle-adjustment settings:

#![allow(unused)]
fn main() {
pub struct LaserlineDeviceConfig {
    // Per-camera linear-initialization stage settings.
    pub init: IntrinsicsInitConfig,
    // { iterations: usize = 2, fix_k3: bool = true }; it fits only the
    //   distortion terms the bundle adjustment estimates

    // The camera model (any distortion kind, an identity or Scheimpflug
    // sensor) and the parameters the bundle adjustment holds fixed, named per
    // slot; a slot left out keeps the model's default (`k3` for Brown-Conrady;
    // everything but the low-order radial terms under a Scheimpflug sensor).
    pub camera: CameraConfig, // { model: CameraModelSpec, fix: CameraFix }

    // Non-linear solve stage settings. `robust_loss` is not consulted here —
    // laser-carrying stages use `optimize.calib_loss`/`optimize.laser_loss`.
    pub solver: SolverConfig, // { max_iters: usize = 50, verbosity: usize = 0, .. }

    // Bundle-adjustment options.
    pub optimize: DeviceOptimizeConfig,
}

pub struct DeviceOptimizeConfig {
    pub calib_loss: RobustLoss,        // Default: Huber { scale: 1.0 }
    pub laser_loss: RobustLoss,        // Default: Huber { scale: 1.0 }, in the residual's unit
    pub calib_weight: f64,             // Weight for calibration residuals (default: 1.0)
    pub laser_weight: f64,             // Weight for laser residuals (default: 1.0)
    pub fix_poses: Vec<usize>,         // Default: empty (each pose is set by its own corners)
    pub fix_plane: bool,
    pub laser_residual_type: LaserlineResidualType, // Default: LineDistNormalized
}
}

The line-distance residual needs a projection that keeps lines straight (the pinhole); the configuration is checked before the solve.

Weight Balancing

Since calibration residuals (in pixels) and laser residuals may have different scales, the weights allow balancing their relative influence. A common starting point is calib_weight = 1.0 and laser_weight = 1.0, adjusting if one term dominates.

Complete Example

#![allow(unused)]
fn main() {
use vision_calibration::prelude::*;
use vision_calibration::laserline_device::*;

let mut session = CalibrationSession::<LaserlineDeviceProblem>::new();
session.set_input(laserline_input)?;

run_calibration(&mut session)?;

let export = session.export()?;
println!("Plane normal: {:?}", export.params.plane.normal);
println!("Plane distance: {:.4}", export.params.plane.distance);
println!("Reprojection error: {:.4} px", export.mean_reproj_error);
println!("Laser error: {:.4}", export.stats.mean_laser_error);

// The plane's uncertainty, from corner and laser residuals together.
if let Some(cov) = export.uncertainty.covariance() {
    println!("Plane distance σ: {:.2e} m", cov.std_dev("plane.d").unwrap());
}
}

export.params.camera is the calibrated camera in its model (CameraParams). The export's uncertainty covers the camera's free parameters (intrinsics, distortion and, for a tilted sensor, the tilt) and the laser plane (plane.nx, plane.ny, plane.nz, plane.d, in the camera frame), with each view's target pose as a per-view block; see Parameter Uncertainty. The rig laserline result labels each camera's plane cam<i>.plane.nx … and lists, in front of them, the upstream calibration's camera parameters and extrinsics when the upstream carries its covariance; the joint rig hand-eye laserline result adds the planes to the rig hand-eye labels.

Scheimpflug Support

Laser profilers usually tilt their sensor. Select it in the camera model:

#![allow(unused)]
fn main() {
let mut config = LaserlineDeviceConfig::default();
config.camera = CameraConfig::new(CameraModelSpec::pinhole(
    DistortionKind::BrownConrady5,
    SensorKind::Scheimpflug,
));
}

A laser profiler's mount tilt is a hardware property, measured separately, so the device holds it by default: give it in the manual-init sensor seed (step_init_with_seed; zero when unseeded), and the camera initializes as an untilted one with the tilt attached. To estimate the tilt instead, list the sensor parameters to hold explicitly: camera.fix.sensor = Some(vec![]) refines both angles. Without a seed, step_init then estimates the tilt with the linear tilt sweep and refines the camera alone through the staged tilt multi-start before it fits the laser plane, because a cold tilt = 0 start falls into the tilt / principal-point / distortion degeneracy. A tilt seed for an untilted camera must be zero.

Scheimpflug and Laser Rigs

Four pieces extend the basic workflows to tilted-sensor cameras and laser-triangulation rigs. A tilted sensor is a value of the camera model, camera.model.sensor = Scheimpflug, in every workflow: planar intrinsics, single-camera hand-eye, the laserline device and the rigs.

See Sensor Models and Scheimpflug Tilt for the tilt model and Laserline Device Calibration for the single-camera laser problem.

Single Tilted Camera

A single tilted camera is calibrated by Planar Intrinsics with camera.model.sensor = SensorKind::Scheimpflug: same input (PlanarDataset), two extra parameters, the sensor tilt angles $({\tau }_x,{\tau }_y)$. Tilt, focal length, and distortion are strongly coupled, so seed a coarse prior (the nominal focal length and the nominal mount tilt) instead of relying on the from-scratch initialization:

#![allow(unused)]
fn main() {
use vision_calibration::core::{FxFyCxCy, ScheimpflugParams, SensorKind};
use vision_calibration::planar_intrinsics::{
    PlanarIntrinsicsConfig, PlanarIntrinsicsProblem, IntrinsicsManualInit, step_init_with_seed,
    step_optimize,
};

let mut config = PlanarIntrinsicsConfig::default();
config.camera.model.sensor = SensorKind::Scheimpflug;
let mut session = CalibrationSession::<PlanarIntrinsicsProblem>::new();
session.set_config(config)?;
session.set_input(dataset)?; // PlanarDataset

let mut seed = IntrinsicsManualInit::default();
seed.intrinsics = Some(FxFyCxCy { fx: 1050.0, fy: 1050.0, cx: 360.0, cy: 270.0 });
seed.sensor = Some(ScheimpflugParams { tilt_x: -0.087, tilt_y: 0.0 });

step_init_with_seed(&mut session, seed, None)?;
step_optimize(&mut session, None)?;
let export = session.export()?;
}

run_calibration(&mut session) runs the unseeded step_init and step_optimize in one call. The Planar Intrinsics chapter describes the seeded and cold-start solves.

Camera Models for Rigs

RigExtrinsicsProblem and RigHandeyeProblem take a camera (the model every camera uses) and camera_overrides (per-camera replacements). Any distortion kind combines with either sensor, per camera:

#![allow(unused)]
fn main() {
use vision_calibration::common::config::{CameraConfig, CameraOverride};
use vision_calibration::core::{CameraModelSpec, DistortionKind, SensorKind};

session.update_config(|c| {
    // Every camera: tilted sensor, Brown-Conrady distortion.
    c.camera = CameraConfig::new(CameraModelSpec::pinhole(
        DistortionKind::BrownConrady5,
        SensorKind::Scheimpflug,
    ));
    // Camera 1 is an untilted camera with a rational lens.
    c.camera_overrides = vec![CameraOverride::new(
        1,
        CameraModelSpec::pinhole(DistortionKind::Rational8, SensorKind::Identity),
    )];
    c.rig.refine_sensor = false; // keep the tilts fixed in the rig bundle adjustment
})?;
}

The default is a Brown-Conrady camera with an identity sensor. camera.fix and each override's fix name the parameters held fixed, per slot (intrinsics, distortion, sensor); a slot left out keeps the model's default, which for a Scheimpflug camera holds everything but k1, k2. rig.refine_intrinsics keeps these fix lists in the rig bundle adjustment, and rig.refine_sensor (hand-eye: handeye_optimize.refine_sensor) lets the tilts move there.

Tilt seeds are device data, not config: pass per_cam_sensors in the manual-init struct (RigHandeyeConfig::manual_init, or rig_extrinsics::IntrinsicsManualInit with step_intrinsics_init_all_with_seed for rig extrinsics). A seed is used verbatim and skips the linear stage's tilt sweep; without one the linear stage sweeps the tilt grid, which includes zero. A non-zero tilt seed for an untilted camera is an error.

Each tilted camera is first calibrated from its own views, as in single-camera planar intrinsics: the linear stage holds p1, p2 at zero (a tangential term and the tilt trade off, and the default fix list holds them in every later solve), and the camera's own solve frees every target pose. On clean synthetic data this solve reaches the reprojection noise floor, with the principal point and tilt at the truth within their standard deviations.

From scratch (nothing seeded, for any camera), a tilted camera's own solve can still settle in a wrong tilt / principal-point basin, which the rig bundle adjustment cannot leave while it holds the tilt. Both rig workflows then solve each tilted camera through a wider multi-start over linear tilt seeds, and re-solve a Brown-Conrady tilted camera that still fits worse than 0.5 px from a nominal seed built from the rig's well-fitted cameras. This costs extra solves (several times the per-camera stage on a hard rig), and a seeded rig skips it.

RigLaserlineDevice

Given a frozen rig calibration, RigLaserlineDeviceProblem fits one laser plane per camera and reports each plane both in the camera frame and in the rig frame.

#![allow(unused)]
fn main() {
pub struct RigLaserlineDeviceConfig {
    pub solver: SolverConfig,                       // max_iters defaults to 200
    pub laser_residual_type: LaserlineResidualType,
}

pub struct RigLaserlineDeviceInput {
    pub dataset: RigLaserlineDataset,               // target corners + laser pixels per camera/view
    pub upstream: RigUpstreamCalibration,           // cameras (CameraParams), cam_se3_rig,
                                                    // rig_se3_target, uncertainty
    pub initial_planes_cam: Option<Vec<LaserPlane>>,
}
}

RigUpstreamCalibration holds one CameraParams per camera, and each camera's laser residuals are evaluated in its own model, so the rig laser workflows accept every distortion kind and untilted and Scheimpflug cameras alike (a general homography sensor, which no camera model describes, is a typed error). The usual source is a rig hand-eye export:

#![allow(unused)]
fn main() {
use vision_calibration::rig_laserline_device::{
    RigLaserlineDeviceInput, RigLaserlineDeviceProblem, run_calibration,
};

// The laser views are the hand-eye run's views: its own board poses.
let upstream = rig_export.to_upstream_calibration(rig_export.rig_se3_target.clone())?;
let mut session = CalibrationSession::<RigLaserlineDeviceProblem>::new();
session.set_input(RigLaserlineDeviceInput { dataset, upstream, initial_planes_cam: None })?;
run_calibration(&mut session)?;
let export = session.export()?; // laser_planes_rig, laser_planes_cam, per_camera_stats, ...
}

to_upstream_calibration takes one board pose per laser view, through the hand-eye chain: the export's own rig_se3_target when the laser views are its views, or the chain at the laser views' robot poses (EyeInHand: T_R_T = T_G_R⁻¹ · T_B_G⁻¹ · T_B_T; EyeToHand: T_R_T = T_R_B · T_B_G · T_G_T).

Uncertainty. The planes are fitted with the upstream calibration held at its values, so a plane moves with the cameras, their extrinsics and the board poses it was fitted against. RigUpstreamCalibration::uncertainty is the upstream's covariance over those (cam1.fx, cam1.cam_se3_rig.rx, view3.rig_se3_target.rx, …), and to_upstream_calibration fills it from the hand-eye calibration's: the cameras and extrinsics as they are, and the hand-eye and target poses as each view's board pose, correlated across views and with the cameras. The robot poses, with any per-view corrections the hand-eye run estimated, count as exact. The export's covariance then has three parts, all in one matrix:

• each plane's own fit, from its laser pixels;
• the upstream's share, A Σ_u Aᵀ, with A how the fitted planes move with the upstream parameters: it correlates the planes of different cameras;
• the cameras' parameters and extrinsics with their upstream covariance, and their cross terms A Σ_u with the planes.

So a laser point measured with this calibration (triangulate_laser on export.calibrated_rig()) carries the whole calibration, as with the joint solve below, in the rig frame. The export does not carry the hand-eye: for points in the gripper or base frame with the hand-eye's share, measure with the joint solve's export, whose rig carries the hand-eye correlated with the planes (see Points in the Robot's Frames). With the upstream's uncertainty unavailable (an upstream built by hand, the default), the upstream counts as exact: the covariance lists the planes alone, conditional on the upstream, and a laser point's uncertainty is unavailable for want of the cameras'. The rig hand-eye run's covariance holds the extrinsics only when it refines them (handeye_optimize.refine_cam_se3_rig); otherwise they count as exact here too.

In Python, vc.rig_upstream_calibration(export, export.rig_se3_target) gives the same RigUpstreamCalibration, its uncertainty the covariance's serde payload.

The steps step_init and step_optimize are available for finer control. dataset is a RigLaserlineDataset (RigLaserlineDataset::new(views, num_cameras), also exported from this module; its view type RigLaserlineView is in vision_calibration::common) whose views hold, per camera, target correspondences and the extracted laser pixels. Laser pixel extraction itself is left to the application.

RigHandeyeLaserline

RigHandeyeLaserlineProblem solves the whole chain jointly: it runs rig hand-eye, initializes the per-camera laser planes from the frozen hand-eye geometry, then refines rig, hand-eye, and laser parameters together.

One camera. A single laser head on a robot is a rig of one camera: num_cameras: 1, one target and one laser slot per view. The camera's frame is the rig's, so the hand-eye is the camera's (gripper_se3_rig is gripper_se3_camera). Its camera, laser plane and hand-eye come out of one solve, in one camera frame.

#![allow(unused)]
fn main() {
pub struct RigHandeyeLaserlineConfig {
    pub handeye: RigHandeyeConfig,                  // warm-start stage (incl. `camera`)
    pub laserline_init: RigLaserlineDeviceConfig,   // plane-initialization stage
    pub joint_optimize: JointOptimizeConfig,        // solver, calib_loss, laser_loss,
                                                    // calib_weight, laser_weight, fix,
                                                    // fix_handeye, ...
}
}

joint_optimize.fix holds one CameraFix per camera, named in that camera's model. Left empty, every camera gets the joint stage's default: intrinsics and the low-order radial terms (k1, k2; the division model's lambda) refined, the higher-order distortion terms and the sensor tilt held at their warm-start values, because they trade off against the extrinsics and the laser planes. An entry's slot left out keeps the model's default, as in every CameraFix.

The input is a RigHandeyeLaserlineInput { views, num_cameras }. Each RigHandeyeLaserlineView (exported from this module; its parts RigLaserlineView and RobotPoseMeta are in vision_calibration::common) carries the per-camera target observations, the laser pixels, and the robot pose (base_se3_gripper). There is a single entry point:

#![allow(unused)]
fn main() {
use vision_calibration::rig_handeye_laserline::{RigHandeyeLaserlineProblem, run_calibration};

let mut session = CalibrationSession::<RigHandeyeLaserlineProblem>::new();
session.set_input(input)?;
run_calibration(&mut session)?;
let export = session.export()?; // cam_se3_rig, handeye (gripper_se3_rig and base_se3_target), laser_planes_rig, per_camera_stats, ...
}

When the rig and its hand-eye are already calibrated (a staged procedure that checks each stage before the next), or the laser planes are known from the design, run_calibration_with_seed starts the joint solve from them: its JointManualInit takes a RigHandeyeExport of the same views (rig_handeye) and one plane per camera in camera frame (planes_cam). Seeding with Prior Knowledge shows both.

As in the single-camera laser problem, calibration and laser residuals use separate robust losses (joint_optimize.calib_loss, joint_optimize.laser_loss); joint_optimize.solver.robust_loss is not consulted.

Examples

• Rust: cargo run -p vision-calibration --example laserline_device_session (single-camera laser plane with a Scheimpflug-capable sensor) and cargo run -p vision-calibration --example rig_handeye_synthetic (the rig hand-eye stage that feeds the rig laser problems).
• Rust tests showing seeded Scheimpflug calibration: crates/vision-calibration/tests/planar_intrinsics_scheimpflug.rs (one camera) and crates/vision-calibration/tests/rig_extrinsics_scheimpflug.rs (a rig).
• Python: crates/vision-calibration-py/examples/laserline_device_session.py and rig_handeye_synthetic.py; a tilted single camera runs through run_planar_intrinsics with camera.model.sensor = "scheimpflug", and the package also exposes run_rig_laserline_device and run_rig_handeye_laserline.

Seeding with Prior Knowledge

By default every workflow initializes itself with the closed-form solvers described in Why Linear Initialization Matters: Zhang's method for intrinsics, a DLT for rig extrinsics, Tsai-Lenz (or another closed-form method) for hand-eye. That is the right behaviour when the data is well conditioned. In industrial settings you often have better priors:

• datasheet focal lengths or principal points;
• factory-measured Scheimpflug tilts;
• laser-plane geometry from mechanical drawings;
• a previously calibrated rig that new data should only nudge.

Manual initialization lets you seed any subset of these and leaves the rest to the automatic path and the non-linear optimizer.

The Seed Types

For every problem type and every initialization stage there are two things:

• a step_*_with_seed function that takes a *ManualInit struct before its options;
• a typed *ManualInit struct whose fields are all Option<T>. None means "auto-initialize this group"; Some(value) means "use this value verbatim, do not auto-initialize".

#![allow(unused)]
fn main() {
// Automatic path
step_init(&mut session, None)?;
step_optimize(&mut session, None)?;

// Seeded path
let mut seed = IntrinsicsManualInit::default();
seed.intrinsics = Some(nominal_camera_k);
// distortion: None  -> auto-initialized (zeros when intrinsics are seeded)
// sensor: None      -> tilt seed, Scheimpflug cameras only (a typed error otherwise)
// poses: None       -> recovered from homographies using the seeded intrinsics
step_init_with_seed(&mut session, seed, None)?;
step_optimize(&mut session, None)?;
}

The seed structs are #[non_exhaustive]: build one with Default::default() and assign the fields you want, as above. Struct-update syntax (..Default::default()) does not compile outside the defining crate.

After step_*_with_seed returns, the session state is fully initialized, with the same postcondition as the automatic path; step_optimize does not need to know which fields came from where. The session log records the source of every stage, for example:

[intrinsics_init_all] initialized 2 cameras (manual: per_cam_intrinsics, per_cam_distortion)
[rig_init] ref_cam=0, 20 views (manual: cam_se3_rig, rig_se3_target)

Walkthrough: Replaying a Calibration on a Rig

The manual_init example seeds the rig stage of the stereo calibration from a previous run. It loads the corners committed in regression/observations/stereo_rig.json and runs the pipeline three times:

cargo run -p vision-calibration --example manual_init --release

1. Run the automatic pipeline once

#![allow(unused)]
fn main() {
use vision_calibration::prelude::*;
use vision_calibration::rig_extrinsics::{
    IntrinsicsManualInit, RigExtrinsicsProblem, RigManualInit,
    step_intrinsics_init_all_with_seed, step_intrinsics_optimize_all,
    step_rig_init_with_seed, step_rig_optimize,
};

let mut session_a = CalibrationSession::<RigExtrinsicsProblem>::new();
session_a.set_input(load_input()?)?;
step_intrinsics_init_all_with_seed(&mut session_a, IntrinsicsManualInit::default(), None)?;
let intr_opt_a = step_intrinsics_optimize_all(&mut session_a, None)?;
let rig_init_a = step_rig_init_with_seed(&mut session_a, RigManualInit::default(), None)?;
step_rig_optimize(&mut session_a, None)?;
}

IntrinsicsManualInit::default() and RigManualInit::default() have all fields None, so this is exactly the automatic path: the same as run_calibration.

2. Capture what the stages produced

The values live in the typed results the steps return:

#![allow(unused)]
fn main() {
use vision_calibration::core::{BrownConrady5, DistortionParams};

let cameras_a = intr_opt_a.cameras.clone();                     // Vec<CameraParams>
let cam_se3_rig_a = rig_init_a.initial_cam_se3_rig.clone();     // per-camera T_C_R
let rig_se3_target_a = rig_init_a.initial_rig_se3_target.clone(); // per-view T_R_T

let per_cam_k: Vec<_> = cameras_a.iter().map(|c| c.intrinsics).collect();
let per_cam_dist: Vec<BrownConrady5<f64>> = cameras_a
    .iter()
    .map(|c| c.distortion.clone())
    .collect();
}

3. Replay them on a fresh session

#![allow(unused)]
fn main() {
let mut session_b = CalibrationSession::<RigExtrinsicsProblem>::new();
session_b.set_input(load_input()?)?;

let mut intrinsics_seed = IntrinsicsManualInit::default();
intrinsics_seed.per_cam_intrinsics = Some(per_cam_k);
intrinsics_seed.per_cam_distortion = Some(per_cam_dist);
step_intrinsics_init_all_with_seed(&mut session_b, intrinsics_seed, None)?;

// Skip step_intrinsics_optimize_all: the seed is already optimized.

let mut rig_seed = RigManualInit::default();
rig_seed.cam_se3_rig = Some(cam_se3_rig_a);
rig_seed.rig_se3_target = Some(rig_se3_target_a);
step_rig_init_with_seed(&mut session_b, rig_seed, None)?;
step_rig_optimize(&mut session_b, None)?;
}

On the stereo dataset the replay reproduces the first run exactly (|Δ reproj| = 0 px); the example asserts this. A third run seeds the intrinsics with a 10 % focal-length error and a 50 px principal-point offset and still converges to the same 0.2496 px solution, which shows that a rough seed is enough for the optimizer.

Variations

Partial seeding

When intrinsics are seeded but distortion is not, distortion starts at zero in the camera's distortion model: the automatic distortion fit is coupled to the iterative intrinsics estimation and does not run when intrinsics are seeded.

A distortion seed is a DistortionParams of the camera's own model: its kind must equal camera.model.distortion (a Kannala-Brandt seed for a fisheye camera, a Rational seed for a rational one). It is used as given, never converted; a seed of another kind is a typed error naming the field, raised before any work. In JSON it is the tagged form {"type": "brown_conrady5", "k1": -0.1, ...}.

When intrinsics are seeded but poses are not, the poses are recovered from the per-view homographies using the seeded intrinsics, not automatically estimated ones. This keeps the geometric chain consistent with the seed.

Seeding from datasheet values

For a rig whose cameras share one optical design:

#![allow(unused)]
fn main() {
use vision_calibration::core::{BrownConrady5, DistortionParams, FxFyCxCy};

let datasheet_k = FxFyCxCy { fx: 1800.0, fy: 1800.0, cx: 360.0, cy: 270.0 };
let nominal_distortion = DistortionParams::BrownConrady5 {
    params: BrownConrady5 { k1: -0.1, ..Default::default() },
};

let mut seed = IntrinsicsManualInit::default();
seed.per_cam_intrinsics = Some(vec![datasheet_k; num_cameras]);
seed.per_cam_distortion = Some(vec![nominal_distortion; num_cameras]);
step_intrinsics_init_all_with_seed(&mut session, seed, None)?;
}

This unblocks scenarios where Zhang's method fails on borderline data, for example strongly tilted Scheimpflug rigs. For those, also set per_cam_sensors to the nominal mount tilts: a seeded tilt is used verbatim and skips the linear stage's tilt sweep. A non-zero tilt for a camera with an identity sensor is an error; a zero tilt is accepted. RigHandeyeConfig::manual_init takes the same per_cam_sensors. The single-camera seeds (planar_intrinsics::IntrinsicsManualInit, single_cam_handeye::IntrinsicsManualInit and laserline_device::DeviceManualInit) take one tilt in sensor, with the same rule.

The device_seed module derives these seeds from a device description (focal length in millimetres, pixel pitch, mount tilt); see Device Specifications.

Coupling rule (rig stage)

cam_se3_rig and rig_se3_target are geometrically coupled. The rig stage's RigManualInit requires both or neither, and enforces it at runtime with a typed Error::InvalidInput.

Hand-eye coupling

RigHandeye has three initialization stages (intrinsics, rig, hand-eye), each with its own *ManualInit. The hand-eye stage's mode_target_pose depends on the mounting mode:

• EyeInHand: handeye = T_G_R, mode_target_pose = T_B_T.
• EyeToHand: handeye = T_R_B, mode_target_pose = T_G_T.

If you seed handeye but not mode_target_pose, the latter is derived from the chain.

Staged rig laser calibration

RigHandeyeLaserline has no steps: run_calibration runs rig hand-eye, fits each camera's laser plane against that frozen geometry, then refines everything jointly. run_calibration_with_seed takes a JointManualInit that replaces either warm start:

• rig_handeye: a RigHandeyeExport of the same views, for example from a RigHandeyeProblem session you ran (and inspected) first. The rig hand-eye stage is skipped; its cameras, extrinsics, hand-eye transform, target poses and robot-pose corrections start the joint solve.
• planes_cam: one laser plane per camera, in camera frame, for example from the mechanical drawing. The plane stage refines them instead of fitting its starting planes linearly.

#![allow(unused)]
fn main() {
use vision_calibration::rig_handeye_laserline::{JointManualInit, run_calibration_with_seed};

// `rig_export` is the export of a rig hand-eye session on the same views,
// with `config.handeye` as its config.
let mut seed = JointManualInit::default();
seed.rig_handeye = Some(rig_export);
seed.planes_cam = Some(design_planes); // Vec<LaserPlane>, one per camera
run_calibration_with_seed(&mut joint_session, seed)?;
}

With the warm start a rig hand-eye session computes from the joint session's input and config.handeye, the result is identical to run_calibration. The export must have the input's camera count, each camera in the configured model, one target pose per view, and the configured hand-eye mode with its transforms; the seeded planes must be one per camera. Anything else is an Error::InvalidInput, raised before any solve.

In Python

The seeds are dataclasses in the problem's module, with the Rust names and fields; a field left None is estimated. The datasheet seed above:

import vision_calibration as vc
from vision_calibration import rig_extrinsics

seed = rig_extrinsics.IntrinsicsManualInit(
    per_cam_intrinsics=[vc.PinholeIntrinsics(fx=1800.0, fy=1800.0, cx=360.0, cy=270.0)] * 2,
    per_cam_distortion=[vc.BrownConradyDistortion(k1=-0.1, k2=0.0, k3=0.0, p1=0.0, p2=0.0, iters=8)] * 2,
)
init = session.step_intrinsics_init_all_with_seed(seed)
print(session.log[-1].notes)  # the init log names the manual and the estimated parts

Distortion seeds are the distortion dataclasses of the camera model (BrownConradyDistortion, KannalaBrandt4Distortion, …). A seed that does not fit the camera model (a tilt for an untilted camera, a distortion of another kind, one rig pose without the other) raises ValueError.

The staged rig laser calibration takes the rig hand-eye session's export as it is:

from vision_calibration import rig_handeye_laserline

rig_export = rig_session.export()  # a RigHandeyeSession on the same views
joint = vc.RigHandeyeLaserlineSession()
joint.set_input(dataset)
joint.run_calibration_with_seed(
    rig_handeye_laserline.JointManualInit(rig_handeye=rig_export, planes_cam=design_planes)
)

See Also

• Step Functions vs Pipeline Functions: the full list of *_with_seed variants.
• Per-Feature Residuals: inspect per-corner errors once the calibration finishes.
• The example this chapter follows: crates/vision-calibration/examples/manual_init.rs.

Per-Feature Residuals

After a calibration finishes, the only error metric most pipelines surface is the mean reprojection error in pixels. That number says something is wrong but not which corners are wrong. To debug a low-quality calibration (a board that flexed, a corner detection that drifted, a camera with a borderline field of view) you need the per-corner error broken down by view and camera.

Every *Export therefore carries per-feature residual records, so a consumer (the desktop app's Diagnose workspace, a Python notebook, a regression test) can drill in without re-running any geometry.

Structure

Every export has a diagnostics group (ExportDiagnostics) holding per_feature_residuals: PerFeatureResiduals and, when the images are known, the image_manifest. The container is always present (empty by default); empty inner vectors mean "this problem type does not produce observations of that flavour". For example, PlanarIntrinsicsExport.diagnostics.per_feature_residuals.laser is always empty.

Each export also serializes a required kind discriminator (ExportKind, snake_case: "planar_intrinsics", "rig_handeye", ...) as its first field, so a consumer that loads raw JSON can narrow on the single tag instead of probing which fields are present.

PerFeatureResiduals
├─ target: Vec<TargetFeatureResidual>     // per-corner reprojection records
├─ laser:  Vec<LaserFeatureResidual>      // per-pixel laser residuals
├─ laser_lines: Vec<LaserLineTrace>       // the projected laser line of each view
├─ target_hist_per_camera: Option<Vec<FeatureResidualHistogram>>
└─ laser_hist_per_camera:  Option<Vec<FeatureResidualHistogram>>

A target record carries an explicit (pose, camera, feature) triple, the 3D target point target_xyz_m, the observed_px and projected_px pixels and the error_px between them. Records are pose-major: the outer loop is the view, the inner loop the camera (cameras without an observation in that view are skipped), the innermost the feature index in that view's and camera's points_3d.

Option fields carry two meanings:

• error_px = None (with projected_px = None): the projection diverged for that record, for instance a point behind the camera. This separates "feature absent" from "feature present but the model is bad".
• residual_m = None on a laser record: the back-projected ray missed the laser plane.

Histogram bucket edges are fixed at [1, 2, 5, 10] px, giving the buckets <=1, <=2, <=5, <=10, >10. A histogram aggregates only records that produced a finite error_px.

Walkthrough

The examples below use the stereo chessboard rig of Stereo Rig with Real Data. The manual_init example prints the same quantities.

1. Run a calibration

No configuration is needed: the records are always populated.

#![allow(unused)]
fn main() {
use vision_calibration::prelude::*;
use vision_calibration::rig_extrinsics::{RigExtrinsicsProblem, run_calibration};

let mut session = CalibrationSession::<RigExtrinsicsProblem>::new();
session.set_input(my_dataset)?;
run_calibration(&mut session)?;
let export = session.export()?;
}

2. Summarize the histograms

#![allow(unused)]
fn main() {
let pf = &export.diagnostics.per_feature_residuals;
let total = pf.target.len();
let with_value = pf.target.iter().filter(|r| r.error_px.is_some()).count();
println!("{total} target records ({with_value} converged, {} divergent)",
         total - with_value);

if let Some(hists) = pf.target_hist_per_camera.as_ref() {
    for (i, h) in hists.iter().enumerate() {
        println!("cam {i}: count={} mean={:.4}px max={:.4}px buckets {:?}",
                 h.count, h.mean, h.max, h.counts);
    }
}
}

Output on the 20-view stereo chessboard rig:

3080 target records (3080 converged, 0 divergent)
cam 0: count=1540 mean=0.2448px max=1.0262px buckets [1539, 1, 0, 0, 0]
cam 1: count=1540 mean=0.2544px max=0.9146px buckets [1540, 0, 0, 0, 0]

3. Find the worst corners

Sort by error_px to triage outliers:

#![allow(unused)]
fn main() {
let mut worst: Vec<_> = pf
    .target
    .iter()
    .filter_map(|r| r.error_px.map(|e| (e, r)))
    .collect();
worst.sort_by(|a, b| b.0.total_cmp(&a.0));
for (err, r) in worst.iter().take(5) {
    println!(
        "pose={} cam={} feat={} target_xy_m=({:.3},{:.3}) observed=({:.1},{:.1}) err={:.3}px",
        r.pose, r.camera, r.feature,
        r.target_xyz_m[0], r.target_xyz_m[1],
        r.observed_px[0], r.observed_px[1],
        err
    );
}
}

A diagnose view typically projects this onto the source image as a heatmap; the (observed_px, projected_px) pair lets you draw an arrow from observed to projected for every record.

4. Cross-check against the aggregate

The per-camera histogram mean must agree with the mean recomputed from the records. The manual_init example asserts this:

#![allow(unused)]
fn main() {
let mean_from_records: f64 = pf
    .target
    .iter()
    .filter(|r| r.camera == cam_idx)
    .filter_map(|r| r.error_px)
    .sum::<f64>()
    / hist.count as f64;
assert!((mean_from_records - hist.mean).abs() < 1e-9);
}

Laser Residuals

LaserlineDeviceExport and RigLaserlineDeviceExport fill laser and laser_lines in addition to target. Each LaserFeatureResidual carries:

• residual_m: the point-to-plane distance in metres from the back-projected ray's intersection with the target plane to the calibrated laser plane;
• residual_px: the distance in pixels from the observed pixel to the projected laser line. This is the line-distance residual the laser solve minimizes, measured on the undistorted image plane.

Each LaserLineTrace in laser_lines is one view's projected laser line (pose, camera, points_px): a polyline over the extent of that view's laser pixels, for 2D overlays. Lens distortion bends the line, so draw all of its points rather than a segment between the first and last.

Which Camera Chain Produced Them

The records use exactly the camera chain the optimizer used, so they agree with the reported mean error:

• Hand-eye exports (SingleCamHandeyeExport, RigHandeyeExport) derive each view's camera-from-target pose from the hand-eye chain; the export already carries the resulting records.
• Scheimpflug cameras (a planar or rig camera whose CameraParams carries a tilted sensor) are projected through the full pinhole, distortion and Scheimpflug chain.

JSON and Python

Every record type derives Serialize and Deserialize, and the export round-trips through JSON:

#![allow(unused)]
fn main() {
let json = serde_json::to_string(&export)?;
let restored: RigExtrinsicsExport = serde_json::from_str(&json)?;
assert_eq!(
    restored.diagnostics.per_feature_residuals,
    export.diagnostics.per_feature_residuals,
);
}

Empty vectors and None histograms are skipped on the wire ({} for an empty container).

In Python every result has per_feature_residuals, a PerFeatureResiduals whose record lists are numpy tables: target holds one row per TargetFeatureResidual and laser one per LaserFeatureResidual, a column per field (pose, camera, feature, observed_px as (N, 2), ...). A value the record leaves unset, such as the projection of a point behind the camera, is NaN. laser_lines is a list of LaserLineTrace, the histograms lists of FeatureResidualHistogram.

import numpy as np

res = result.per_feature_residuals
worst = np.argsort(-np.nan_to_num(res.target.error_px))[:5]
for i in worst:
    print(res.target.pose[i], res.target.camera[i], res.target.feature[i],
          f"{res.target.error_px[i]:.3f}")
print(res.target_hist_per_camera[0].counts)

Where Else They Show Up

The desktop app's Diagnose workspace draws these records on the source images, with a histogram, a per-pose table and a cameras-by-poses matrix (see Inspecting Results).

CalibrationSession

CalibrationSession<P: ProblemType> is the central state container for calibration workflows. It holds all data for a calibration run — configuration, input observations, intermediate state, final output, and an audit log — with full JSON serialization for checkpointing.

Structure

A session has two public fields and otherwise exposes its data through methods:

Intermediate state is internal to the pipeline. Read intermediate results from the typed values returned by each step function (for example IntrinsicsInitResult), from log(), or from export().

The public methods of CalibrationSession<P>:

Lifecycle

1. Create

#![allow(unused)]
fn main() {
let mut session = CalibrationSession::<PlanarIntrinsicsProblem>::new();
// Or with a description:
let mut session = CalibrationSession::<PlanarIntrinsicsProblem>::with_description(
    "Lab camera calibration"
);
}

2. Set Input

#![allow(unused)]
fn main() {
session.set_input(dataset)?;
}

Input is validated via ProblemType::validate_input(). Setting input clears computed state (per the invalidation policy).

3. Configure

#![allow(unused)]
fn main() {
session.update_config(|c| {
    c.solver.max_iters = 50;
    c.solver.robust_loss = RobustLoss::Huber { scale: 2.0 };
})?;
}

Configuration is validated via ProblemType::validate_config().

4. Run Steps

#![allow(unused)]
fn main() {
step_init(&mut session, None)?;
step_optimize(&mut session, None)?;
}

Step functions are free functions operating on &mut CalibrationSession<P>. Each step reads input/state, performs computation, and updates state (or output).

5. Export

#![allow(unused)]
fn main() {
let export = session.export()?;
}

Creates an ExportRecord with the current timestamp and output. Multiple exports can be created (e.g., after re-optimization with different settings).

JSON Checkpointing

The entire session can be serialized and restored:

#![allow(unused)]
fn main() {
// Save
let json = session.to_json()?;
std::fs::write("calibration.json", &json)?;

// Restore
let json = std::fs::read_to_string("calibration.json")?;
let mut restored = CalibrationSession::<PlanarIntrinsicsProblem>::from_json(&json)?;

// Resume from where we left off
step_optimize(&mut restored, None)?;
}

This enables:

• Interruption recovery: Save progress and resume later
• Reproducibility: Share exact calibration state
• Debugging: Inspect the session at any point

Invalidation Policies

When input or configuration changes, computed state may need to be cleared:

Each problem type may override these policies.

Audit Log

Every step function appends to the session log:

#![allow(unused)]
fn main() {
pub struct LogEntry {
    pub timestamp: u64,
    pub operation: String,
    pub success: bool,
    pub notes: Option<String>,
}
}

The log records what was done and when, useful for tracking calibration history.

Reading Results

#![allow(unused)]
fn main() {
// Input (required before steps)
let input = session.require_input()?;  // Returns error if no input

// Intermediate results: the typed value returned by each step
let init = step_init(&mut session, None)?;
println!("Init fx={:.1}", init.camera.intrinsics.fx);

// Output (available after optimize)
let output = session.require_output()?;

// Session bookkeeping
println!("{} ({} log entries)", session.metadata().problem_type, session.log().len());
}

Progress and Cancellation

A long run reports its progress, and stops on request, through a RunControl (in vision_calibration::control): an optional CancelToken and observers that receive each Progress report.

use vision_calibration::control::{CancelToken, Progress, RunControl};

let token = CancelToken::new();
let control = RunControl::new()
    .with_cancel(token.clone())
    .with_observer(|progress| match progress {
        Progress::Step { name, .. } => eprintln!("{name}"),
        Progress::Iteration { iter, cost, .. } => eprintln!("  {iter}: {cost:.3e}"),
        _ => {}
    });
session.set_control(control);
// From another thread (a Ctrl-C handler, a Cancel button): token.cancel();
match run_calibration(&mut session) {
    Err(vision_calibration::Error::Cancelled) => eprintln!("stopped"),
    other => other?,
}

The token is checked before every solver iteration, between the steps of run_calibration and before every image, so a run stops within one iteration or one image of the request. A stopped call returns Error::Cancelled (CalibrateDatasetError::Cancelled from calibrate_dataset, RunError::Cancelled from a build_*_input converter), never a partial result: a stopped step stores nothing in the session. Progress is a side channel: a run with observers returns exactly what it returns without them, bit for bit, and nothing it reports enters a result.

The same control goes to dataset_runner::calibrate_dataset and the build_*_input converters as their last argument, and to optim::problem::solve as SolveOptions::control. Observers run on the calibrating thread at every report, so keep them cheap; one that must redraw a screen can drop reports that arrive too fast.

In Python

Each problem type has its session class, named after it: PlanarIntrinsicsSession, SingleCamHandeyeSession, RigExtrinsicsSession, RigHandeyeSession, LaserlineDeviceSession, RigLaserlineDeviceSession and RigHandeyeLaserlineSession. The step functions are its methods, with the Rust names; each takes the step's options and returns the step's result as a dataclass. Long steps release the GIL.

import vision_calibration as vc

session = vc.PlanarIntrinsicsSession("Lab camera calibration")
session.set_input(dataset)  # a PlanarIntrinsicsInput, or its payload dict
session.set_config(vc.PlanarIntrinsicsConfig(solver=vc.SolverConfig(max_iters=50)))

init = session.step_init(vc.IntrinsicsInitOptions(iterations=3))
print(f"Init fx={init.camera.intrinsics.fx:.1f}")
session.step_optimize()
result = session.export(notes="first pass")  # a PlanarIntrinsicsExport

for entry in session.log:  # the audit log, as LogEntry dataclasses
    print(entry.operation, entry.success, entry.notes)

# Checkpoint and resume
restored = vc.PlanarIntrinsicsSession.from_json(session.to_json())

input and output return the serde payloads (dicts), config the config dataclass, exports the ExportRecords and metadata the SessionMetadata. run_calibration() runs the problem's default steps. An input or config the problem rejects, a step out of order and a seed that does not fit raise ValueError; a numerical failure raises RuntimeError.

Ctrl-C stops a running step with KeyboardInterrupt. set_progress gives the session a callable that receives every report as a dict ({"kind": "iteration", "iter": 3, "max_iters": 50, "cost": 12.5}); raising from it stops the step, and the exception propagates. The run_* functions, calibrate_dataset and the build_*_input converters take the callable as their progress argument:

def show(report):
    if report["kind"] == "detection":
        print(f"detected {report['done']}/{report['total']}")

result = vc.calibrate_dataset("data/kuka_1/dataset.toml", progress=show)

ProblemType Trait

The ProblemType trait defines the interface for a calibration problem. It specifies the types for configuration, input, output, and export, along with validation and lifecycle hooks. The trait is intentionally minimal — behavior is implemented in external step functions, not trait methods.

ProblemType is sealed: it can only be implemented by the seven problem types that ship with the pipeline crate, so downstream crates use it as a bound (CalibrationSession<P: ProblemType>) but do not implement it. Contributors adding a problem type inside the pipeline crate should read Adding a New Pipeline Problem Type; a pipeline of your own is composed from the public pieces instead (see Compose Your Own Pipeline).

Definition

#![allow(unused)]
fn main() {
pub trait ProblemType: ProblemState + Sized + 'static {
    type Config: Clone + Default + Serialize + DeserializeOwned + Debug;
    type Input: Clone + Serialize + DeserializeOwned + Debug;
    type Output: Clone + Serialize + DeserializeOwned + Debug;
    type Export: Clone + Serialize + DeserializeOwned + Debug;

    fn name() -> &'static str;
    fn schema_version() -> u32 { 1 }

    fn validate_input(input: &Self::Input) -> Result<(), Error> { Ok(()) }
    fn validate_config(config: &Self::Config) -> Result<(), Error> { Ok(()) }
    fn validate_input_config(
        input: &Self::Input, config: &Self::Config
    ) -> Result<(), Error> { Ok(()) }

    fn on_input_change() -> InvalidationPolicy {
        InvalidationPolicy::CLEAR_COMPUTED
    }
    fn on_config_change() -> InvalidationPolicy {
        InvalidationPolicy::KEEP_ALL
    }

    fn export(
        input: &Self::Input, output: &Self::Output, config: &Self::Config
    ) -> Result<Self::Export, Error>;
}
}

ProblemState is a crate-private supertrait that carries the internal State associated type (the intermediate results between steps). It is what seals the trait; State is not part of the public API.

Associated Types

The Serialize + DeserializeOwned bounds enable JSON checkpointing (the internal state is serialized as well). Clone enables snapshot operations.

Required Method: name()

Returns a stable string identifier for the problem type:

#![allow(unused)]
fn main() {
fn name() -> &'static str { "planar_intrinsics_v2" }
}

This is stored in session metadata and used for deserialization. Changing the name breaks compatibility with existing checkpoints.

Required Method: export()

Converts the internal output to a user-facing export format:

#![allow(unused)]
fn main() {
fn export(
    input: &Self::Input, output: &Self::Output, config: &Self::Config,
) -> Result<Self::Export, Error>;
}

The export may transform, filter, or enrich the output. The input is provided so exports can attach per-feature reprojection residuals; for example, PlanarIntrinsicsProblem::export() computes per-feature residuals and reprojection error metrics from the raw optimization output.

Validation Hooks

Three optional validation hooks run at specific points:

Example: PlanarIntrinsicsProblem validates that input has at least 3 views with at least 4 points each.

Invalidation Policies

Control what happens when input or config changes:

#![allow(unused)]
fn main() {
pub struct InvalidationPolicy {
    pub clear_state: bool,
    pub clear_output: bool,
    pub clear_exports: bool,
}

impl InvalidationPolicy {
    pub const KEEP_ALL: Self = ...;       // Nothing cleared
    pub const CLEAR_COMPUTED: Self = ...; // State + output cleared
    pub const CLEAR_ALL: Self = ...;      // Everything cleared
}
}

Defaults:

• Input change → CLEAR_COMPUTED (re-running steps is needed)
• Config change → KEEP_ALL (output is still valid, but may not reflect new config)

Problem Types

Design Philosophy

The trait is minimal by design:

• No step methods: Steps are free functions, not trait methods. This allows flexible composition and avoids trait object limitations.
• No algorithm logic: The trait defines data types and validation, not computation.
• Serialization-first: All types are serializable, enabling checkpointing from day one.

Step Functions vs Pipeline Functions

calibration-rs offers two ways to run calibration workflows: step functions for granular control and pipeline functions for convenience.

Step Functions

Step functions are free functions that operate on a mutable session reference:

#![allow(unused)]
fn main() {
pub fn step_init(
    session: &mut CalibrationSession<PlanarIntrinsicsProblem>,
    opts: Option<IntrinsicsInitOptions>,
) -> Result<IntrinsicsInitResult, Error>
}

Each step:

1. Reads the input, config, and the results of earlier steps from the session
2. Performs one phase of the calibration (e.g., initialization or optimization)
3. Updates the session (and possibly its output)
4. Logs the operation
5. Returns a typed result value summarizing what it computed

Advantages

Intermediate inspection: Examine the typed result of each step.

#![allow(unused)]
fn main() {
let init = step_init(&mut session, None)?;

// Inspect initialization quality before committing to optimization
if (init.camera.intrinsics.fx - expected_fx).abs() / expected_fx > 0.5 {
    eprintln!("Warning: init fx={:.0} is far from expected {:.0}",
              init.camera.intrinsics.fx, expected_fx);
}

let opt = step_optimize(&mut session, None)?;
println!("{:.3} px after {} iterations", opt.mean_reproj_error, opt.iterations);
}

Per-step configuration: Override options for individual steps.

#![allow(unused)]
fn main() {
// Use more iterations for optimization
let mut opts = IntrinsicsOptimizeOptions::default();
opts.max_iters = Some(200);
step_optimize(&mut session, Some(opts))?;
}

Selective execution: Skip steps or re-run specific steps.

#![allow(unused)]
fn main() {
// Re-run optimization with different settings (without re-initializing)
session.update_config(|c| c.solver.robust_loss = RobustLoss::Cauchy { scale: 3.0 })?;
step_optimize(&mut session, None)?;
}

Checkpointing: Save and restore between steps.

#![allow(unused)]
fn main() {
step_init(&mut session, None)?;
let checkpoint = session.to_json()?;
std::fs::write("after_init.json", &checkpoint)?;

step_optimize(&mut session, None)?;
}

Manual Seeds

Every problem with an initialization step also has a *_with_seed variant (for example step_init_with_seed taking an IntrinsicsManualInit) that accepts a coarse prior for some parameters and auto-estimates the rest. The plain step_init is the same call with an empty seed. Seeding with Prior Knowledge walks through the seed types with a runnable example.

Names

Every problem module uses the same names, so code written against one problem reads the same against another:

• The problem is <Name>Problem, with <Name>Input, <Name>Config and <Name>Export (PlanarIntrinsicsInput, RigHandeyeConfig, …). An input that is a shared data type is a type alias (PlanarIntrinsicsInput = PlanarDataset).
• The problem's output is the final solve's estimate, *Estimate (PlanarIntrinsicsEstimate, HandeyeEstimate, …). The hand-eye estimate is shared by the single-camera and rig workflows and lives in vision_calibration::common.
• A step of stage <Stage> and action <Action> takes Option<<Stage><Action>Options> (None keeps the session config) and returns <Stage><Action>Result; its _with_seed variant takes a <Stage>ManualInit before the options. The stages are Intrinsics, Rig and Handeye; the single-stage laser problems use Device (camera and laser plane, laserline_device) and Plane (the laser planes, rig_laserline_device); the joint rig laser solve is the Joint stage (JointOptimizeConfig, JointManualInit).
• Option types shared by several problems (IntrinsicsInitOptions, RigOptimizeOptions, HandeyeInitOptions, …) are one type each, imported from vision_calibration::common. Step results and manual seeds are per problem: rig_handeye::IntrinsicsInitResult and planar_intrinsics::IntrinsicsInitResult are different types.
• run_calibration(session) has the same signature in every problem and runs with the session's config.

Pipeline Functions

Pipeline functions chain all steps into a single call:

#![allow(unused)]
fn main() {
pub fn run_calibration(
    session: &mut CalibrationSession<PlanarIntrinsicsProblem>,
) -> Result<(), Error> {
    step_init(session, None)?;
    step_optimize(session, None)?;
    Ok(())
}
}

When to Use

• Quick prototyping: Get results with minimal code
• Default settings: When the defaults work and you don't need inspection
• Scripts and automation: When human inspection is not needed

#![allow(unused)]
fn main() {
let mut session = CalibrationSession::<PlanarIntrinsicsProblem>::new();
session.set_input(dataset)?;
run_calibration(&mut session)?;
let export = session.export()?;
}

Available Steps and Pipeline Functions

Paths are relative to the facade module named after the problem (vision_calibration::<module>).

The *_with_seed variants exist for step_init / step_intrinsics_init / step_intrinsics_init_all / step_rig_init / step_handeye_init of the problems above. RigHandeyeLaserlineProblem has no steps; run_calibration_with_seed takes a JointManualInit that replaces its rig hand-eye warm start, its starting laser planes, or both.

step_filter removes high-residual points from the session input, which invalidates the computed state; run step_init and step_optimize again afterwards. run_calibration_with_filtering does exactly that: solve, filter, solve.

In Python

The steps are methods of the problem's session class, with the same names. Options are dataclasses whose None fields keep the session config; results are dataclasses. The shared option types are importable from the package root; each problem's seeds and results live in the problem's module, which mirrors the Rust module of the same name:

import vision_calibration as vc
from vision_calibration import rig_extrinsics

session = vc.RigExtrinsicsSession()
session.set_input(dataset)
init = session.step_intrinsics_init_all()           # rig_extrinsics.IntrinsicsInitResult
session.step_intrinsics_optimize_all(vc.IntrinsicsOptimizeOptions(max_iters=200))
rig = session.step_rig_init()
session.step_rig_optimize()

# Re-run the rig stage from its own estimate
session.step_rig_init_with_seed(
    rig_extrinsics.RigManualInit(
        cam_se3_rig=rig.initial_cam_se3_rig,
        rig_se3_target=rig.initial_rig_se3_target,
    )
)

The one-shot run_<problem> functions (run_planar_intrinsics, run_rig_extrinsics, ...) are the pipeline functions: a session, run_calibration and export in one call.

Recommendation

Use step functions for production calibration where you need to:

• Verify initialization quality
• Adjust parameters between steps
• Handle failures gracefully
• Log and audit the process

Use pipeline functions for:

• Examples and tutorials
• Batch processing with known-good settings
• Quick experiments

Calibration Targets and Detection

The calibration workflows consume 2D-3D correspondences: a pixel in the image and the position of the same point on the target, in metres, in the target's own frame. Turning images into those correspondences is the job of the vision-calibration-detect crate. It ships four target detectors behind one interface and a content-addressed cache, so repeated runs over the same images never repeat the detection.

You rarely call a detector yourself. The dataset runner picks the detector from the manifest's [target] table, runs it over every image through the cache and assembles the problem input. This chapter is for choosing a target, understanding what each detector returns, and using the detectors or the cache directly.

What a Detector Returns

Every detector turns one image into a list of Feature values:

#![allow(unused)]
fn main() {
pub struct Feature {
    pub image_xy: [f64; 2],   // pixel (column, row) in the source image
    pub world_xyz: [f64; 3],  // metres, in the target frame; the target plane is z = 0
}
}

A frame that simply contains no target is not an error: every detector returns an empty list. Errors (DetectError) are reserved for a malformed configuration (Config, InvalidConfig) and genuine backend failures (Backend), so a misconfigured board cannot masquerade as an unlucky image.

Every detector also passes its result through reject_ambiguous_detection. The targets are planar, so the image-to-target map is a homography and therefore injective: one pixel is one board point. A detection in which two correspondences share an image point has mislabelled its grid. The whole detection is rejected (an empty result), because a duplicated point is evidence that the grid walk lost track of the labels, and the labels that happen not to collide come from the same broken walk. The caller sees a skipped view, which every workflow handles.

The Four Targets

The target is chosen by the manifest's [target] table, kind being one of the four names below. All lengths are in metres.

Chessboard

rows and cols count interior corners: a 9x6 setting is a 10x7 grid of squares. cols corners run across the image and rows down; a board seen turned by 90 degrees is also accepted whole.

The underlying detector labels each view on its own: grid index (0, 0) is the detection's top-left corner in that image, +u runs along the board axis closest to the image's +x and +v along the one closest to +y, and the corner positions are lifted to (i * square_size_m, j * square_size_m, 0). A board seen whole and the same way up gets the same labels in every view. A partial board is labelled from its own top-left corner, so its target frame is shifted by the rows and columns out of view; a board seen turned gets labels turned with it.

That is harmless for single-camera intrinsics and the single-camera laserline device, which estimate one target pose per view: a shifted frame only changes that view's pose, and partial views near the image border are valuable for the distortion model. It is not harmless where views share one target frame: hand-eye (the target is fixed relative to the robot base or gripper) and every rig topology (the cameras of one view see one target pose). A shifted frame there contradicts the other views. On a three-camera hand-eye rig, keeping partial boards raises the worst camera's mean reprojection error from 0.29 px to 4.7 px.

So in the shared-frame topologies the dataset runner keeps a chessboard detection only when it spans the whole board: corners on all four edges, each edge holding at least half of its corners (and two). Interior corners the detector missed do not matter, since the edges still fix the labels; an edge held by a stray corner or two does not count, because the detector can pad a partial board that way. A partial board is dropped, and the result's rejected list records it with its reason (PartialChessboard). There is no switch to turn this off, because a view that fails the check would give a wrong result.

A whole board is the board's own grid up to a turn in its plane: two possible labellings for a rectangular board (rows != cols), four for a square one. What the runner does with the turn depends on what the labels must agree with.

• The cameras of one view (every rig topology). Each camera's labels are turned to agree with the view's first camera, so a hand-held board may be turned any way in any view. The runner picks the labelling from the board's direction in each image: two cameras of a rigid rig see the board turned by the same relative angle in every view, up to the board's symmetry (half a turn for a rectangular board, a quarter for a square one), and how the board moves through both images across the views tells that angle whole, so a camera mounted turned or upside down relative to the others is matched too. When the views are too few (fewer than three shared views) or the board moves by less than a square, the cameras are taken to be mounted the same way up. A detection whose direction does not single out one labelling, within 45 degrees for a rectangular board and 22.5 for a square one, is dropped as AmbiguousChessboardTurn.
• One target frame through every view (hand-eye and the laser rigs). The frame is the one camera 0 labels. The board must be seen as declared, cols across and rows down, by camera 0, or, in a view camera 0 misses, within 45 degrees of how camera 0 would see it; a view where it is turned by 90 degrees is dropped as TurnedChessboard. A board seen upside down by camera 0, or a square board seen turned by 90 degrees, cannot be told from upright: capture the board the same way up in every view, and prefer a non-square board.

A camera left with no usable chessboard view is an error (NoWholeChessboardViews) before any solve. To use partial views in a shared frame, use a ChArUco or PuzzleBoard target. Their corners are identified by codes printed on the board, so a partial view is labelled correctly.

ChArUco

A ChArUco board interleaves a chessboard with ArUco markers, and each corner is identified by the markers around it. A partly visible board therefore still yields correctly labelled corners, and the result is sparse: occluded or unidentified cells simply contribute no feature, so the per-view feature count varies.

rows and cols count squares (not corners). marker_size_m must be positive and at most square_size_m. dictionary is an ArUco dictionary name such as "DICT_4X4_1000"; the layout is checked before any image is read, because a larger board needs more marker codes than a small dictionary holds (a 12x12 board needs 72, which DICT_4X4_50 cannot supply). validate_dictionary and validate_charuco_layout expose these checks. The marker layout follows the OpenCV ChArUco convention, so coordinates are indexed the way an OpenCV-printed board expects.

PuzzleBoard

A PuzzleBoard is a self-identifying chessboard: each saddle decodes to a unique grid index from a local code window, so a single partial view yields globally consistent correspondences. The manifest names the layout as "puzzle_<rows>x<cols>" (the prefix is optional, "130x130" also parses); the dimensions are all the detector needs. Coordinates are in the board's own frame with its origin at the top-left, like the chessboard and ChArUco detectors, and decoded saddles outside the printed board are discarded.

Two fixed parts of the recipe are not configurable: the image is upscaled 2x before decoding (saddle decoding degrades on small or blurred tiles; pixel coordinates are divided back down), and the search is restricted to the declared board.

Ring Grid

A coded ring grid is a hexagonal lattice of ring markers, each carrying a 16-sector code. A decoded marker contributes one correspondence at its centre when its code maps to a known board position. rows is the number of marker rows and long_row_cols the number of columns of the longest (even-indexed) row; the shorter rows follow from the lattice. The three radii describe one marker (outer radius, inner radius, ring-band width) and are validated against pitch_m when the board layout is built. Detection runs in adaptive-scale mode, since the marker pixel size is unknown for an arbitrary image.

The detector fits a flat shape: a hexagonal lattice of coded rings. Rectangular lattices and plain uncoded annuli are not exposed.

Corner Detector Overrides

The chessboard and ChArUco detectors both start from a ChESS-family corner detector. Four optional overrides live in the manifest's [detector.chess_corners] table (and in each detector's chess_corners field). They are hashed into the detection-cache key, so changing any of them re-runs detection.

Set min_corner_strength = 0.0 on small or soft image tiles. There the floor can remove a third of the board, and the resulting sparse, unevenly covered view conditions worse than the weak corners did; the loss shows up as a diverged camera, not as a slightly worse residual.

Corner covariance from the detector

The chessboard and ChArUco detectors can report, for every corner, the covariance of its position: the image noise propagated through the corner localizer, plus the localizer's systematic-error model. Ask for it in the manifest:

[detector.chess_corners.covariance]
image_noise = { known = 2.0 }

image_noise is "estimate" (the default when the section is present) or { known = σ }, the standard deviation of the image's pixel noise in gray levels of the 8-bit image (finite and at least 0), for example the standard deviation of the difference of two frames of a static scene divided by √2.

With the section present, every observation carries the covariances in cov_px (px², [xx, xy, yy] in Feature::cov_px), the residuals are whitened by them, and the report gives the unit-weight scale factor σ̂₀ instead of an RMS in pixels; see Per-Point Covariance. Positions are identical with and without it.

• JPEG and demosaiced images. The per-image noise estimate is unreliable there, and the detector may then report no covariance. The run stops with an error naming the image and suggesting image_noise = { known = σ }. No corner is dropped to get past this.
• Supported targets. chessboard and charuco. The PuzzleBoard detector runs on an upscaled image, for which no covariance is reported, and ring grids have none; a manifest that sets covariance for another target is an error, raised before any detection.
• Not recommended for calibration. On real chessboard and ChArUco images, weighting a calibration by the detector's covariance did not improve its accuracy on held-out views, and on some cameras it made it slightly worse; its stated uncertainty was no closer to the real scatter either. It stays off unless you ask for it. The setting is part of the detection-cache key.

In Python, Features.cov_px is the (N, 3) array of covariances (None when not requested), and Features.to_observation() carries it into the observation.

detector.min_features_per_view (a manifest field, not a detector option) sets the minimum number of features for a view to be kept. It defaults to 4, the homography minimum. Raise it for detectors whose sparse detections are unreliable, such as ring-grid decodes with fewer than 8 markers. A plain chessboard in a shared-frame topology needs no floor: the whole-board rule above already applies.

Calling a Detector Directly

A DetectorConfig names the target and carries its typed configuration, one variant per detector; the set is closed and owned by the crate. It detects in an 8-bit grayscale GrayImageView (row-major pixels; decode and convert the image with any library):

#![allow(unused)]
fn main() {
use vision_calibration::detect::{ChessboardConfig, DetectorConfig, GrayImageView};

let config = DetectorConfig::Chessboard(ChessboardConfig {
    rows: 17,
    cols: 28,
    square_size_m: 0.02,
    chess_corners: None,
});
let gray = image::open("data/kuka_1/01.png")?.to_luma8();
let features = config.detect(GrayImageView::new(gray.width(), gray.height(), gray.as_raw())?)?;
println!("{} corners", features.len());
}

ChessboardConfig, CharucoConfig, PuzzleboardConfig and RinggridConfig reject unknown fields when deserialized, so a typo in a hand-written config fails loudly. Each detector sits behind its own cargo feature of vision-calibration-detect (chessboard, charuco, puzzleboard, ringgrid; all on by default), so a build that needs one target does not compile the others. The facade crate's vision_calibration::detect module re-exports the configs, the cache, Feature and the error types.

To get features into a calibration without a manifest, build the problem input from Feature values yourself: world_xyz and image_xy are exactly the points_3d and points_2d of a CorrespondenceView.

The facade's dataset_runner::detect_target runs a detector from a manifest's target (and optional detector) section instead of a detector config, lowered exactly as a manifest run lowers it:

#![allow(unused)]
fn main() {
use vision_calibration::dataset::TargetSpec;
use vision_calibration::dataset_runner::detect_target;
use vision_calibration::detect::GrayImageView;

let target = TargetSpec::Chessboard { rows: 17, cols: 28, square_size_m: 0.02 };
let gray = image::open("data/kuka_1/01.png")?.to_luma8();
let features = detect_target(GrayImageView::new(gray.width(), gray.height(), gray.as_raw())?, &target, None)?;
}

In Python

detect_target takes a (H, W) uint8 grayscale numpy image and the target as its manifest payload, and returns a Features table: image_xy (N, 2) and world_xyz (N, 3) arrays. An impossible target (an unknown ChArUco dictionary) raises RunError with kind == "invalid_target_config"; an image without a usable board gives an empty table.

import numpy as np
from PIL import Image
import vision_calibration as vc

target = {"kind": "chessboard", "rows": 17, "cols": 28, "square_size_m": 0.02}
views = []
for path in ["data/kuka_1/01.png", "data/kuka_1/02.png", "data/kuka_1/03.png"]:
    gray = np.asarray(Image.open(path).convert("L"))
    features = vc.detect_target(gray, target)
    print(path, len(features), "corners")
    views.append(vc.PlanarView(observation=features.to_observation()))
result = vc.run_planar_intrinsics(vc.PlanarIntrinsicsInput(views=views))

vc.reject_ambiguous_detection(features) returns the detection, or an empty one when two features share an image point.

The Detection Cache

Detection dominates the cost of iterating on a calibration: the solvers take milliseconds to seconds, the detectors seconds per image. The DetectionCache trait stores detections so a rerun with the same images and the same detector parameters reads files instead of detecting again.

A cache entry is keyed by CacheKey:

• the image content: a 64-bit FNV-1a digest of the raw file bytes. File modification time is deliberately not used, so moving or copying an image does not invalidate its entry;
• the detector name: "chessboard", "charuco", ... (laser extraction uses a laser:<name> namespace);
• the configuration: a hash of the detector configuration serialized to JSON with sorted keys, together with the crop the image was detected in, the crate's release and the detector's output revision (CacheKey::for_detection). Semantically equal configurations collide on purpose; changing any detector parameter, upgrading the detector stack or bumping a detector's output revision gives a new key and so recomputes, while the entries of the other detectors stay valid.

#![allow(unused)]
fn main() {
use vision_calibration::detect::{CacheKey, CachedFeatures, DetectionCache, FsDetectionCache, GrayImageView};

let cache = FsDetectionCache::new("cache/kuka_1");   // created on first write
let bytes = std::fs::read("data/kuka_1/01.png")?;
let key = CacheKey::for_detection(&bytes, &config, None); // no crop

let features = match cache.get(&key)? {
    Some(hit) => hit.features,
    None => {
        let gray = image::load_from_memory(&bytes)?.to_luma8();
        let features = config.detect(GrayImageView::new(gray.width(), gray.height(), gray.as_raw())?)?;
        cache.put(&key, &CachedFeatures { features: features.clone() })?;
        features
    }
};
}

FsDetectionCache writes one JSON file per entry under its root, named <image>-<detector>-<config>.json with a portable file-name encoding that is valid on Linux, macOS and Windows. Entries are written atomically, so the cache is safe under concurrent readers and writers. The dataset runner takes a Detection mode: Detection::Cached(&cache) reuses and stores entries, Detection::Refresh(&cache) detects every image again and overwrites its entry, and Detection::Uncached uses no cache. The cache is any &dyn DetectionCache, so an application can supply its own store.

In Python, vc.FsDetectionCache(root) is the filesystem cache, passed as cache= to calibrate_dataset and the build_*_input functions (without one, every image is detected). Its get(key) and put(key, features) take a CacheKey; CacheKey.from_inputs(image_bytes, detector, config) keys entries you manage yourself.

Choosing a Target

The detectors differ in what a partial view gives you:

• A chessboard is labelled from its own top-left corner in each image, so only complete views of the board are safe in a rig or hand-eye run (see above).
• ChArUco, PuzzleBoard and ring-grid targets identify every corner or marker, so any visible part of the board is usable and the target frame is the same in every view. This matters when a robot occludes the board or the board is larger than the field of view of some cameras.

Whatever the target, the accuracy of the calibration is bounded by the accuracy of the detected points. The reprojection floor on real data is usually set by the detector and the print quality, not by the solver. The per-feature residuals show whether the error is spread evenly or concentrated on a few views or corners.

The Dataset Manifest

A calibration dataset is a folder of images, usually a robot-pose file, and some knowledge only you have: what the target is, how big it is, which frame convention the robot controller uses. The dataset manifest (DatasetSpec, in the vision-calibration-dataset crate) is the one on-disk format that describes all of it. The manifest is descriptive, never prescriptive: your data stays where it is, and the manifest points at it.

The same manifest drives three front ends: the Rust dataset runner, the desktop app's Run workspace (which edits it in a schema-driven form) and the folder sniffer that writes a first draft.

An Example

data/kuka_1/dataset.toml in the repository describes a single-camera hand-eye dataset: 30 chessboard images and a file with one robot pose per line.

version = 1
topology = "single_cam_handeye"
description = "kuka_1"

[[cameras]]
id = "cam0"
images = { kind = "glob", pattern = "*.png" }

[target]
kind = "chessboard"
rows = 17
cols = 28
square_size_m = 0.02

[robot_poses]
path = "RobotPosesVec.txt"
format = "rowmajor4x4"

[pose_pairing]
kind = "by_index"

[pose_convention]
transform = "t_base_tcp"
rotation_format = "matrix4x4_row_major"
translation_units = "m"

Paths are relative to the directory that holds the manifest unless they are absolute. TOML is the usual authoring format; the same structure is valid JSON, and a field reference generated from the JSON schema is in Manifest Field Reference.

Topologies

topology selects the problem type the runner dispatches to. It describes the data (how many cameras, whether there are robot poses and laser frames), not the camera model: a tilted-sensor camera is a planar_intrinsics dataset whose calibration config names the sensor.

A field that the topology does not use is an error, not something silently ignored: laser, upstream_calibration and per-camera laser_images are rejected on the non-laser topologies, and upstream_calibration is also rejected on laserline_device and rig_handeye_laserline, which calibrate their own upstream geometry. rig_laserline_device is a second stage that needs upstream_calibration, the path to a frozen rig hand-eye export; see Laser Datasets.

Cameras and Images

Each [[cameras]] entry has an id (unique; used in pairings and exports) and an images source:

• { kind = "glob", pattern = "cam0/*.png" }: a shell glob, expanded relative to the manifest directory and sorted in natural order (2.png before 10.png);
• { kind = "list", paths = ["a.png", "b.png"] }: an explicit ordered list.

roi_xywh = [x, y, w, h] crops the image to a rectangle in source pixels before detection; the width and height must be non-zero. It is how one image holding several camera tiles becomes several cameras. Laser topologies add a laser_images source per camera with the same two forms.

Target

[target] is tagged by kind: chessboard, charuco, puzzleboard or ringgrid, with the fields documented in Calibration Targets and Detection. [detector] holds optional detection overrides shared by the target families (min_features_per_view, and the [detector.chess_corners] table).

Robot Poses

[robot_poses] points at the pose file; [pose_convention] says how to read it. There are no defaults for the convention: it is required whenever robot_poses is set, because a wrong guess (gripper in base or base in gripper, millimetres or metres) produces a plausible but wrong calibration.

The [pose_convention] table has three required fields:

• transform: which transform a row encodes. t_base_tcp is the gripper pose in the robot base frame (KUKA and UR controllers), t_tcp_base its inverse (some ABB exports), t_world_tcp and t_tcp_world the same two for a fixed world frame.
• rotation_format: quat_xyzw, quat_wxyz, euler_xyz_deg, euler_xyz_rad, euler_zyx_deg, euler_zyx_rad, axis_angle_rad or matrix4x4_row_major.
• translation_units: m or mm.

For tabular formats, [robot_poses.columns] maps your column names to tx, ty, tz, a rotation list whose length matches the rotation format, and an optional pose_id.

[pose_pairing] aligns images with each other and with pose rows:

• { kind = "by_index" }: the i-th image of every camera pairs with pose row i; counts must match.
• { kind = "shared_filename_token", regex = "...", group = "..." }: a regex with a named capture group extracts a view token from every filename (and the pose_id column of the pose file); the cameras and the poses must produce the same set of tokens.

The Folder Sniffer

Writing a manifest by hand for a folder of 400 images is tedious, so sniff_folder writes a first draft:

#![allow(unused)]
fn main() {
use std::path::Path;
use vision_calibration::dataset::sniff_folder;

let spec = sniff_folder(Path::new("data/my_dataset"))?;
println!("{:?}", spec.unresolved);
}

The same function is vcal init on the command line (The Command Line) and the Sniff folder button of the app's Run workspace:

vcal init data/my_dataset        # writes data/my_dataset/dataset.toml

It looks only at file names and folder structure, never at image content, and infers only what is structurally unambiguous:

• cameras: images are grouped by parent directory; a single group is cam0, several groups take their directory names; each gets a *.ext glob for its dominant extension (png, jpg, jpeg, tif, tiff, bmp; dot-files are skipped);
• a robot-pose file: a non-image file whose name contains pose, classified by extension and content (csv, json, jsonl, a 16-float line for rowmajor4x4, a count-then-matrices file for counted4x4);
• by_index pairing, when every camera and the pose file have the same count;
• the topology from the camera count and the presence of poses (one camera, no poses: planar_intrinsics; one camera with poses: single_cam_handeye; several cameras with poses: rig_handeye; several without: rig_extrinsics).

The Unresolved-Fields Rule

Everything that needs domain knowledge is never guessed. The sniffer leaves a placeholder in the manifest (an empty chessboard, an incomplete convention) and records the dotted path of the field in the manifest's _unresolved list:

_unresolved = ["pose_convention.transform", "pose_convention.translation_units", "target"]

The placeholders exist only so that the manifest deserializes and a form has something to edit; they are never trusted numerically. The runner refuses a manifest whose _unresolved list is not empty. You resolve each field (fill in the target, confirm the convention) and delete the list.

The sniffer flags, as applicable:

The same philosophy applies at run time. When the runner meets an ambiguity it cannot resolve from the manifest (for example a hand-eye manifest with no pose_pairing, or token pairing without a pose_id column) it returns RunError::AskUser { field, prompt, suggestions } instead of choosing. The app turns that into a dialog that asks you to pick one of the suggestions; library code decides what to do with it.

Validation

vision_calibration_dataset::validate(&spec) checks the structural rules without touching the filesystem (it does not expand globs), and every runner converter calls it first. It returns a typed ValidationError: unresolved fields; a topology that needs robot poses and has none, or a camera count outside the topology's range; a missing pose_convention; duplicate camera ids; empty image sources; a rotation-column count that does not match the rotation format; tabular pose formats without columns; matrix formats with a columns mapping; degenerate ROIs; laser fields on a non-laser topology and missing laser sources on a laser one; out-of-range laser or detector parameters.

Running a Manifest

The converters in vision_calibration::dataset_runner turn a DatasetSpec into the input of a problem type, running detection through a detection cache:

#![allow(unused)]
fn main() {
use std::path::Path;
use vision_calibration::dataset::DatasetSpec;
use vision_calibration::control::RunControl;
use vision_calibration::dataset_runner::{Detection, build_single_cam_handeye_input};
use vision_calibration::detect::FsDetectionCache;
use vision_calibration::prelude::*;
use vision_calibration::single_cam_handeye::{SingleCamHandeyeProblem, run_calibration};

let base_dir = Path::new("data/kuka_1");
let spec = DatasetSpec::from_path(base_dir.join("dataset.toml"))?; // TOML or JSON
let cache = FsDetectionCache::new("cache/kuka_1");

let run = build_single_cam_handeye_input(&spec, base_dir, Detection::Cached(&cache), &RunControl::default())?;
println!("{}/{} views usable", run.usable_views, run.total_views);

let mut session = CalibrationSession::<SingleCamHandeyeProblem>::new();
session.set_input(run.input)?;
run_calibration(&mut session)?;
}

Each converter checks the topology and target, validates the manifest and resolves the pairing before it reads a single image, so a wrong manifest fails in milliseconds with a message that names the field. A view whose detection yields fewer features than detector.min_features_per_view (default 4) is dropped. With a plain chessboard, the topologies whose views share one target frame (single_cam_handeye and every rig topology) also drop a view unless its corners span the whole board: a partial chessboard is labelled from its own top-left corner and would move that frame. A rig turns each camera's labels to agree with the other cameras of the view, so a board seen turned is kept; where the target is fixed through the views (hand-eye), a board camera 0 sees turned by 90° is dropped. Calibration Targets and Detection explains the rules, and recommends ChArUco or PuzzleBoard targets when you need partial views. planar_intrinsics and laserline_device estimate one target pose per view and keep partial boards.

Each result lists the detections it dropped in rejected, each with its camera, image and RejectReason (TooFewFeatures, PartialChessboard, TurnedChessboard or AmbiguousChessboardTurn). A camera left with no usable view is an error rather than a poor calibration: NoWholeChessboardViews when the whole-board rule dropped any of its views, InsufficientUsableViews otherwise. Each result also carries the source image path of every accepted view, which is how the app shows the right image under each residual.

In Python

calibrate_dataset runs a manifest end to end: it reads the file (TOML or JSON), picks the converter and the session by the manifest's topology, calibrates with the given config and returns the result. The manifest's relative paths resolve against its directory.

import vision_calibration as vc

cache = vc.FsDetectionCache("cache/kuka_1")
result = vc.calibrate_dataset("data/kuka_1/dataset.toml", vc.SingleCamHandeyeConfig(), cache=cache)
print(result.handeye.mode, f"{result.mean_reproj_error:.3f} px")

The converters are vision_calibration.dataset_runner.build_*_input, with the Rust names, for a run you drive step by step; each returns the run result (input as a payload, view_paths, usable_views, total_views). A manifest can also be a DatasetSpec payload, with base_dir= naming the directory its paths resolve against. vc.sniff_folder(path) returns the drafted manifest as a payload.

The runner's errors keep their types. RunError carries the Rust variant as kind ("validation", "empty_image_match", "pose_count_mismatch", ...); the ask-user ambiguity is its subclass AskUserError, with the field, prompt and suggestions:

from vision_calibration.dataset_runner import build_single_cam_handeye_input

try:
    run = build_single_cam_handeye_input(spec, base_dir="data/kuka_1")
except vc.AskUserError as err:
    print(f"set {err.field}: {err.prompt} (e.g. {err.suggestions})")
session = vc.SingleCamHandeyeSession()
session.set_input(run.input)

Device Specifications

A second, optional file describes the hardware rather than the data: spec.json next to the manifest (DEVICE_SPEC_FILENAME), parsed as DeviceSpec. It holds datasheet values: per camera the focal length in millimetres, the pixel pitch in micrometres, the image resolution, an optional principal point and an optional Scheimpflug mount tilt in degrees; for rigs the nominal camera mounts, the hand-eye mounting and laser-plane nominals.

The vision_calibration::device_seed functions derive manual-init seeds from it (scheimpflug_seed, rig_intrinsics_seed, nominal_cam_se3_rig, handeye_seed, rig_layout_seed). The intrinsics derivation is f_px = focal_mm * 1000 / pixel_pitch_um for both focal lengths with the principal point defaulting to the image centre; all unit conversion (millimetres, micrometres and degrees to pixels, metres and radians) happens there and nowhere else. The functions work per camera id, never by position, and a mismatch is a typed DeviceSeedError. The seeds feed the rig family; the single-camera problems take a seed directly (see Seeding with Prior Knowledge).

JSON Schema

The manifest types derive a JSON Schema, and cargo xtask emit-schemas writes it to app/src/schemas/dataset_spec.json. The app generates its manifest form from that file, so every field the Rust type accepts is editable without hand-writing TOML; external tools can use it for validation and completion in an editor. Manifest Field Reference renders the schema as tables.

Laser Datasets

A laser-triangulation device captures two frames per view: a target frame (board visible, projector off) and a laser frame (projector on). The manifest describes both. The runner detects the target features, extracts the laser-line pixels (cached like detections) and assembles the input of the laser problem types: LaserlineDevice, RigLaserlineDevice and RigHandeyeLaserline. For the rig laser topology that includes the frozen upstream rig hand-eye calibration.

This chapter extends The Dataset Manifest with the laser-specific fields. The calibration workflows themselves are in Laserline Device Calibration and Scheimpflug and Laser Rigs.

How the Manifest Describes Laser Data

• laser_images per camera mirrors images: a glob or an explicit list. It pairs with the target views through the same pose_pairing: with by_index the i-th laser image pairs with the i-th target image, with shared_filename_token the same regex runs on both filename sets. A target view without a laser frame is a gap; a laser frame without a target view is an error.
• [laser] tunes the extraction: scan_axis (cols walks image columns, one point per column, for lines running mostly left to right; rows walks rows, for vertical lines), sigma (Gaussian derivative scale in pixels), pos_thresh and neg_thresh (edge-response thresholds) and min_points (the smallest usable extraction, at least 2). Omit the table for the defaults: extraction tuning has safe defaults and is not an ambiguity the runner refuses to guess.
• RigLaserlineDevice is a second stage: upstream_calibration points at a frozen RigHandeyeExport JSON. The per-view board poses are recomputed from the laser dataset's own robot poses through the frozen hand-eye chain, so the laser views need not match the upstream run's views. The export's covariance comes with it, in the terms of the input (the cameras and extrinsics as they are, the hand-eye and target poses as each laser view's board pose), so the planes' covariance carries the upstream calibration's share.
• RigHandeyeLaserline is the joint solve: it computes its own hand-eye warm start and takes no upstream_calibration.

Walkthrough: A Scheimpflug Rig, Eye-to-Hand

version = 1
topology = "rig_laserline_device"
upstream_calibration = "rig_handeye_export.json"

[target]
kind = "charuco"
rows = 12
cols = 9
square_size_m = 0.01
marker_size_m = 0.0075
dictionary = "DICT_4X4_1000"

[laser]
min_points = 120          # views below the bar become gap slots

[robot_poses]
path = "poses.json"
format = "json"
matrix_field = "tcp2base" # a nested 4x4 per row (vendor JSON)

[pose_convention]
transform = "t_base_tcp"
rotation_format = "matrix4x4_row_major"
translation_units = "mm"

[pose_pairing]
kind = "by_index"

[[cameras]]
id = "cam0"
roi_xywh = [0, 0, 720, 540]   # one tile of a multi-camera strip image
[cameras.images]
kind = "glob"
pattern = "target_*.png"
[cameras.laser_images]
kind = "glob"
pattern = "laser_*.png"
# ... one [[cameras]] entry per camera

Run the hand-eye stage first (topology = "rig_handeye", the same cameras and poses, no laser fields) and save its export as rig_handeye_export.json. In the desktop app each stage is one Run click; such a setup needs non-default configuration (Scheimpflug sensors, eye-to-hand, PointToPlane laser residuals), set in the config form or in a preset.

A single-camera LaserlineDevice is the same minus robot_poses and upstream_calibration: it calibrates the intrinsics and the laser plane jointly from the target and laser frames alone.

Library Consumers: Inject an Extractor

The published crates do not ship a laser-line extractor, and extraction is the one step of the runner that you provide. Implement the open LaserPixelExtractor trait and hand it to the converter:

#![allow(unused)]
fn main() {
use vision_calibration::dataset::LaserExtractionSpec;
use vision_calibration::control::RunControl;
use vision_calibration::dataset_runner::{Detection, LaserPixelExtractor, build_rig_laserline_device_input};
use vision_calibration::detect::GrayImageView;

struct MyExtractor;

impl LaserPixelExtractor for MyExtractor {
    fn name(&self) -> &str { "my-extractor-v1" } // cache-key namespace
    fn extract(
        &self,
        image: GrayImageView<'_>, // 8-bit grayscale, row-major
        spec: &LaserExtractionSpec,
    ) -> Result<Vec<[f64; 2]>, Box<dyn std::error::Error + Send + Sync>> {
        // subpixel laser-line points in the (ROI-cropped) image
        todo!()
    }
}

let run = build_rig_laserline_device_input(
    &spec,
    base_dir,
    Detection::Cached(&cache),
    &MyExtractor,
    &RunControl::default(),
)?;
}

The runner owns the caching (keyed laser:<name> plus a hash of the extraction spec), the ROI cropping and the coordinate lifting; the extractor only extracts. Unlike the target detectors, a closed set, this trait is open. name() is part of the cache key, so switching implementations re-extracts instead of serving another extractor's pixels.

In Python

Any object with a name and an extract(image, spec) method is an extractor (the LaserPixelExtractor protocol of vision_calibration.dataset_runner). extract receives the cropped laser image as a (H, W) uint8 grayscale array and the manifest's [laser] section as a dict, and returns the line's points as an (N, 2) array of (x, y) pixels:

import numpy as np
import vision_calibration as vc


class ColumnPeak:
    name = "column-peak-v1"

    def extract(self, image, spec):
        """Per column, the row centroid of the pixels above half the peak."""
        values = image.astype(float)
        points = []
        for col in range(image.shape[1]):
            column = values[:, col]
            if column.max() < 50:
                continue
            weights = np.where(column > column.max() / 2, column, 0.0)
            points.append((col, weights @ np.arange(len(column)) / weights.sum()))
        return np.array(points).reshape(-1, 2)


result = vc.calibrate_dataset("data/my_laser_rig/dataset.toml", laser_extractor=ColumnPeak())

An exception the extractor raises fails the run with a RunError of kind == "laser_extraction".

Variations

• Sparse laser coverage (rig): cameras may miss the laser in some views (None slots), but every camera needs at least one usable laser view; otherwise its plane is unconstrained and the runner fails fast.
• Vertical laser lines: scan_axis = "rows".
• Re-extraction: edit [laser] and the cache key changes, so nothing stale is served. The same contract holds for detector configurations.

Manifest Field Reference

Every field of the dataset manifest, generated from its JSON Schema. The prose chapter The Dataset Manifest explains how the pieces fit together; this page is the exhaustive list. The schema itself is app/src/schemas/dataset_spec.json in the repository, and the desktop app builds its manifest form from it.

Paths in a manifest are relative to the directory that holds it unless they are absolute. A field marked required has no default and must be present; an optional field may be omitted.

DatasetSpec

Top-level dataset manifest. Authoritative description of where the user's calibration data lives and how to interpret it.

Unknown fields are rejected.

CameraSource

Per-camera image source.

Unknown fields are rejected.

ImagePattern

How image paths for a camera are listed.

Tagged on kind; the fields depend on the variant.

kind = "glob": Single shell-glob pattern, evaluated relative to the manifest directory. Example: "cam0/*.png".

kind = "list": Explicit ordered list of paths.

TargetSpec

Calibration-target metadata. Tagged on kind.

Tagged on kind; the fields depend on the variant.

kind = "chessboard": Standard checkerboard. Counts are interior corners (i.e. 9x6 ⇒ a 10×7 black-and-white square grid).

A plain chessboard has no markers: each view is labelled from the detection's own top-left corner. Where views share one target frame (single-camera hand-eye and every rig topology) the runner therefore keeps a view only when its corners reach every edge of the board. In a rig, each camera's labels are turned to agree with the other cameras of the view, so a board seen turned is kept. Where the target is fixed through the views (hand-eye), the board must appear the same way up in every image of camera 0 (cols across and rows down); a board seen turned by 90° is dropped, and one seen upside down, or a square board seen turned by 90°, cannot be told apart. Use a ChArUco or PuzzleBoard target to calibrate a shared frame from partial views.

kind = "charuco": ChArUco board. Marker IDs and dictionary live here so detection is fully deterministic from the manifest.

kind = "puzzleboard": Puzzleboard target: a self-identifying chessboard. The layout name carries the board's dimensions, so the manifest fully determines detection.

kind = "ringgrid": Coded ring-grid target (the ringgrid crate). The board is a hex-lattice of self-identifying ring markers; its geometry mirrors ringgrid::BoardLayout so the manifest fully determines detection.

Topology

Calibration topology — selects the problem-type dispatch.

A topology describes the data (how many cameras, robot poses, laser images), not the camera model: a tilted (Scheimpflug) single camera is a planar_intrinsics dataset whose calibration config names the sensor.

DetectorSpec

Target-detector overrides. These options affect feature detection only; they are hashed into the detection-cache key by the runner.

Unknown fields are rejected.

ChessCornersDetectorSpec

ChESS corner extractor overrides.

Unknown fields are rejected.

CornerCovarianceSpec

Per-corner covariance options; see [ChessCornersDetectorSpec::covariance].

Unknown fields are rejected.

ImageNoiseSpec

The pixel-noise level behind a corner covariance.

In JSON / TOML: "estimate" or { known = 2.0 }.

Tagged on kind; the fields depend on the variant.

kind = "": Estimate the noise from each image. Unreliable on JPEG or demosaiced images, where a corner then reports no covariance and the run fails.

kind = "": The standard deviation of the image's pixel noise, in gray levels of the 8-bit image (finite and >= 0), e.g. the standard deviation of the difference of two frames of a static scene divided by √2.

Unknown fields are rejected.

CornerStrategySpec

The corner detector a chessboard-like detector runs.

LaserExtractionSpec

Laser-line extraction parameters. Consumed by the runner's injected LaserPixelExtractor; hashed (canonical JSON) into the detection-cache key so edits re-extract.

Unknown fields are rejected.

LaserScanAxis

Laser scan direction.

PoseConvention

Frame conventions for the robot pose stream.

All three components are required (no defaults). Each is a closed enum so the runtime supports a finite, validated set of conventions.

Unknown fields are rejected.

RotationFormat

Rotation parameterisation in the file.

TransformConvention

Which transform a pose row encodes.

TranslationUnits

Translation units in the file.

PosePairing

How per-camera image lists align across views and (optionally) align with robot poses.

Tagged on kind; the fields depend on the variant.

kind = "by_index": All cameras have the same number of images, paired by index (image[i] from cam0 ↔ image[i] from cam1 ↔ pose[i]).

kind = "shared_filename_token": A regex applied to filenames extracts a shared "view token" (e.g. timestamp or pose id). All cameras and the pose file must produce the same set of tokens.

RobotPoseSource

Where and how robot poses are stored on disk.

Unknown fields are rejected.

RobotPoseFormat

Robot-pose file format.

PoseColumnMap

Mapping from user-defined column / field names to the canonical pose fields. All seven of tx, ty, tz, qx, qy, qz, qw are required when rotation_format is a quaternion; the runner rejects partial mappings as a fail-fast event.

Unknown fields are rejected.

Parameter Uncertainty

A calibration result says not only what the parameters are but how well the data determines them. Every result carries an uncertainty: the covariance of the estimated parameters at the solution, the noise level the residuals imply, and, when no covariance can be given, the reason. The laserline device result keeps it with its estimate, as estimate.uncertainty (in Python also result.uncertainty).

Reprojection error alone does not answer this. A camera calibrated from a handful of similar views can reproject its own corners to 0.1 px and still have a focal length that is off by a percent, with the principal point traded off against it. The covariance shows that directly: a large σ of fx, and a strong fx–cx correlation. On the real images of data/kuka_1, the planar_real example reports fx = 2057.8 ± 0.8 px.

What a σ means

The covariance is first order at the solution: it describes how the estimate would scatter if the calibration were repeated with new noise of the same kind, for the same views. The noise level is taken from the residuals themselves, so nothing has to be known in advance; when the observations state a covariance per point, the residuals are weighed by them and the level becomes a factor on what they state (Per-Point Pixel Covariance). Two consequences follow:

• σ measures the noise-driven error, and includes errors that differ from view to view. The residual noise model takes every corner's error as independent, but on real images much of the error is shared by a view's corners: the lens model, the target or the detector does not fit exactly, differently in each view. The covariance measures this by leaving out one view at a time: view_scatter says how many times the shared parameters vary more across views than the noise model predicts, and the covariance is scaled by it. It is set only when the excess is larger than its own sampling noise (from six views), so data that fit the noise model keep the plain covariance. On the public real-image entries it is 2.8 to 4.9, and 25 for the joint three-camera ds8 rig. An error common to every view, such as a wrong board cell size, still leaves σ small and the estimate biased. Watch the per-feature residuals for structure.
• σ shrinks with more data roughly as 1/√views and grows quickly when the views are similar. The most effective way to reduce σ of fx and cx is board tilt: views at varied angles separate focal length from distance.

With a robust loss (Huber, Cauchy, Arctan), the covariance is that of the robust estimate actually computed, including the bounded pull of the residuals the loss down-weights. It does not ignore the outliers, and it does not treat them as Gaussian noise either. Without a loss it is the classical σ̂² (JᵀJ)⁻¹.

These σ were checked against Monte-Carlo scatter on synthetic scenes (single cameras, a stereo rig, an eye-in-hand camera, a laserline device with corner and laser residuals together), a few hundred noise realizations each. The predicted σ matched the empirical scatter within 10 % on clean data, and within 20 % with 5 % of the corners displaced by 10–30 px under Huber or Cauchy. The projection σ (Projection Uncertainty) was checked the same way.

In This Part

• Reading the Covariance: the labels, the views, the fixed parameters and the noise estimate, from Rust, Python and JSON.
• Per-Point Pixel Covariance: stating each point's noise, and what it changes in the solve and the noise estimate.
• Projection Uncertainty: what the parameter covariance implies for where a ray lands in the image.
• Limits and Unavailable Results: when no covariance can be given, what it is conditional on, and what it does not cover.
• Judging a Result: a pass, warn or fail verdict on a result, built from its covariance, coverage and residuals.

Reading the Covariance

The covariance is part of every result. This chapter shows how to read it and what each part of it contains.

Rust

#![allow(unused)]
fn main() {
use vision_calibration::core::{NoiseLevel, Uncertainty};

let export = session.export()?;
match &export.uncertainty {
    Uncertainty::Available(cov) => {
        let fx = export.params.camera.intrinsics.fx;
        println!("fx = {fx:.1} ± {:.2} px", cov.std_dev("fx").unwrap());
        println!("corr(fx, cx) = {:.2}", cov.correlation("fx", "cx").unwrap());
        for noise in &cov.noise {
            match noise.level {
                NoiseLevel::Sigma(s) => println!("{:?} noise σ ≈ {s:.3}", noise.kind),
                NoiseLevel::Sigma0(s) => println!("{:?} noise σ̂₀ ≈ {s:.3}", noise.kind),
            }
        }
    }
    Uncertainty::Unavailable { reason } => println!("no covariance: {reason:?}"),
}
}

Python

result = vc.run_planar_intrinsics(dataset, vc.PlanarIntrinsicsConfig())
u = result.uncertainty
if u.available:
    print(f"fx = {result.camera.intrinsics.fx:.1f} ± {u.std_dev('fx'):.2f} px")
    print(f"corr(fx, cx) = {u.correlation('fx', 'cx'):.2f}")
else:
    print("no covariance:", u.reason, u.reason_labels)

JSON

In JSON the field is {"status": "available", "labels": [...], "covariance": [[...]], ...} or {"status": "unavailable", "reason": {"kind": "...", ...}}.

What Is Reported

• labels and covariance: the shared parameters and their covariance matrix, one row per label:

  • intrinsics fx, fy, cx, cy;
  • distortion by coefficient name (k1, k2, k3, p1, p2 for Brown-Conrady, lambda for the division model, and so on);
  • the Scheimpflug tilt tilt_x, tilt_y, in radians;
  • the poses a result reports once, under their export names, with six labels each (<pose>.rx … <pose>.tz, see views below): cam_se3_rig per camera, the hand-eye pose (gripper_se3_rig or rig_se3_base, gripper_se3_camera or camera_se3_base for a single camera) and the target pose (base_se3_target or gripper_se3_target);
  • a laser plane n · p + d = 0 in its camera's frame: plane.nx, plane.ny, plane.nz and plane.d (metres). The normal is a unit vector, so its three components have a covariance of rank 2.

  In a result with several cameras, each camera's labels carry its index: cam1.fx, cam1.k1, cam1.cam_se3_rig.tx. std_dev(label) is the square root of a diagonal entry, and correlation(a, b) the normalized off-diagonal one.

• views: the parameters that belong to one view, each view's block separately: the board pose camera_se3_target for intrinsics results, rig_se3_target for rig results, and the robot-pose correction robot_delta for hand-eye results that refine the robot poses. A pose's six labels are rx, ry, rz (radians) and tx, ty, tz (metres) of the perturbation T · Exp(δ) on the right. Both are expressed in the pose's source frame, which is the target frame for camera_se3_target. The correlation of a view's parameters with the shared ones is not reported.

• fixed: the parameters the solve held fixed, such as k3 with the default configuration, or cam0.cam_se3_rig, the reference camera that defines the rig frame. A fixed pose is named once, without its six coordinates. The covariance is conditional on them: it does not include their uncertainty, so fixing a parameter that the data cannot pin down makes the others look better determined than they are. Holding the intrinsics in a rig's final solve (rig.refine_intrinsics: false), for example, leaves their uncertainty out of the extrinsics' σ; by default the rig workflows refine them there.

• noise: for each kind of residual, the Gaussian-equivalent standard deviation of one scalar residual. For reprojection that is the corner noise per pixel axis; for the laser, the line distance in pixels or the point-to-plane distance in metres. It is estimated from the median absolute residual, so a few outliers do not inflate it. In JSON the level is sigma, in the residual's unit, or, when the observations stated a covariance per point, sigma0: the dimensionless σ̂₀ of the whitened residuals, about 1 when the stated covariances hold (Per-Point Pixel Covariance). Rust types the two as NoiseLevel::Sigma and NoiseLevel::Sigma0; Python's ResidualNoise.whitened says which sigma holds. The rig laserline solves each camera's plane on its own, with the rig calibration held fixed, so it reports each camera's noise levels apart (camera names the camera). When the upstream calibration carries its covariance, the planes' covariance carries its share and the planes of different cameras are correlated through it; without one, it is conditional on the upstream and the planes are uncorrelated.

• num_residuals, dof: the scalar residuals and the redundancy they leave after the estimated parameters.

• view_scatter: how many times the shared parameters vary more across views than the residual noise predicts, when significantly more; the covariance is already scaled by it (Parameter Uncertainty).

• second_order: how far the error of the shared parameters is predicted to depart from the covariance once its second-order term is counted. The covariance is the error's first-order term, linear in the noise. Weak views pin some combinations of parameters down only through larger errors of others that cancel in them (centred, mildly tilted boards leave a camera's rotation in a rig and its principal point to trade off this way); there the second-order term, quadratic in the noise, can exceed the first, while every parameter's own σ still holds. max_scatter_ratio is the predicted scatter of the worst combination over the σ the covariance states for it, nees_ratio the predicted mean of eᵀ Σ⁻¹ e per parameter (both 1 when the first order describes the error), and combination names the combination as Σ wᵢ pᵢ / σᵢ, its largest weights first. The prediction comes from the residuals' curvature along the covariance's own errors, at the result's noise level, so it grows with the noise variance. Judging a Result warns when the ratio is above 1.2. It costs s (s + 1) residual evaluations for s shared parameters, so it is left out (null) above 64 of them, and for noise-free data.

Per-Point Pixel Covariance

By default every observed image point is trusted alike, and the noise level is estimated from the residuals in pixels. When you know more about each point's noise, say a detector that reports how well it located each corner, state it as a covariance per point. The solve then weighs each point by what it states: a point with a large covariance counts less, and an elongated one is trusted more across its long axis than along it.

Stating covariances

A covariance is the symmetric 2 × 2 matrix [[xx, xy], [xy, yy]] in px², one per point of a view. It must be finite and positive definite (xx > 0, yy > 0, xy² < xx·yy); anything else is rejected when the view is built or read.

#![allow(unused)]
fn main() {
use vision_calibration::core::{CorrespondenceView, PointCovariance};

// σ = 0.3 px along u, 0.1 px along v.
let cov = PointCovariance::new(0.09, 0.0, 0.01)?;
let view = CorrespondenceView::new_with_covariances(points_3d, points_2d, vec![cov; n])?;
}

cov = vc.PointCovariance(0.09, 0.0, 0.01)           # or vc.PointCovariance.isotropic(0.2)
obs = vc.Observation(points_3d=p3, points_2d=p2, cov_px=[cov] * len(p2))

In JSON a view carries them as cov_px, one object per point:

{"points_3d": [[0.0, 0.0, 0.0]], "points_2d": [[412.3, 228.9]],
 "cov_px": [{"xx": 0.09, "xy": 0.0, "yy": 0.01}]}

All or none. Within one calibration input either every view states covariances or none does, across every camera of a rig. Residuals in pixels and whitened residuals are not one noise population, so an input that mixes them is rejected (MixedCovariance) before any work. There is no setting to switch the weighting: the solve uses what the observations carry.

A uniform weight w on every point is the covariance I / w; covariances that are all the same multiple of the identity give the same estimate and covariance as stating none.

What changes in the solve