Multi-Camera Rig Extrinsics

A multi-camera rig is a set of cameras rigidly attached to a common frame. Rig extrinsics calibration estimates the relative pose of each camera within the rig, in addition to per-camera intrinsics and distortion.

Problem Formulation

Parameters

Per-camera intrinsics: ${K_{k}}$ — one per camera
Per-camera distortion: ${d_{k}}$ — one per camera
Per-camera extrinsics: ${T_{R, C_{k}}}$ — camera-to-rig transforms (one per camera, reference camera = identity)
Per-view rig poses: ${T_{R, T}^{(v)}}$ — rig-to-target transforms (one per view)

Transform Chain

For camera $k$ in view $v$ :

$T_{C_{k}, T}^{(v)} = T_{C_{k}, R} \cdot T_{R, T}^{(v)}$

where $T_{C_{k}, R} = (T_{R, C_{k}})^{- 1}$ .

Objective

$min k = 1 \sum C v = 1 \sum M j = 1 \sum N_{k v} π (K_{k}, d_{k}, T_{C_{k}, R} \cdot T_{R, T}^{(v)}, P_{j}) - p_{k v j}^{2}$

This jointly optimizes all cameras' parameters and all rig geometry. Each observation depends on two SE(3) transforms composed together (a ReprojPoint factor with the TwoSe3 chain).

4-Step Pipeline

Step 1: Per-Camera Intrinsics Initialization (`step_intrinsics_init_all`)

For each camera independently:

Compute homographies from the views where that camera has observations
Run iterative intrinsics estimation (Zhang + distortion fit)
Recover per-camera, per-view poses

Step 2: Per-Camera Intrinsics Optimization (`step_intrinsics_optimize_all`)

For each camera independently:

Run planar intrinsics bundle adjustment
Refine $K_{k}$ , $d_{k}$ , and per-view poses

Step 3: Rig Extrinsics Initialization (`step_rig_init`)

Uses the linear initialization from Multi-Camera Rig Initialization:

Define the rig frame by the reference camera ( $T_{R, C_{0}} = I$ )
For each non-reference camera, compute $T_{R, C_{k}}$ by averaging across views
Compute rig-to-target poses from the reference camera's poses

Step 4: Rig Bundle Adjustment (`step_rig_optimize`)

Joint optimization of all parameters:

Parameter blocks:

"cam/k" (4D): per-camera intrinsics
"dist/k" (5D for Brown-Conrady): per-camera distortion
"extrinsics/k" (7D, SE3): per-camera camera-to-rig transforms
"rig_pose/v" (7D, SE3): per-view rig-to-target poses

Residual blocks: one ReprojPoint { model: <camera's CameraModelSpec>, chain: TwoSe3, .. } per observation. A tilted camera adds a "sensor/k" block (2D); "dist/k" has the size of the camera's distortion model.

Fixed parameters: The reference camera's extrinsics are fixed at identity (gauge freedom).

Configuration

Settings are grouped by stage; the shared sub-structs (IntrinsicsInitConfig, RigConfig, SolverConfig) are reused across every rig-family problem type:

#![allow(unused)]
fn main() {
pub struct RigExtrinsicsConfig {
    // Per-camera linear-initialization stage settings.
    pub intrinsics: IntrinsicsInitConfig, // { iterations, fix_k3 }
    // (fits only the distortion terms each camera's solves estimate)

    // Camera model shared by every camera (projection, distortion, sensor)
    // and its named fix lists; default Brown-Conrady, identity sensor.
    pub camera: CameraConfig, // { model: CameraModelSpec, fix: CameraFix }

    // Per-camera replacements of `camera`, for rigs that mix lenses or sensors.
    pub camera_overrides: Vec<CameraOverride>, // { camera, model, fix }

    // Rig frame options.
    pub rig: RigConfig, // { reference_camera_idx, refine_intrinsics, refine_sensor }

    // Non-linear solve stage settings.
    pub solver: SolverConfig, // { max_iters, verbosity, robust_loss }
}
}

Every camera uses camera unless camera_overrides names it. An override replaces the model and the fix lists of one camera, so a rig can mix distortion kinds and sensors. A camera index out of range or listed twice is a validation error. A slot left out of fix keeps the model's default. With rig.refine_intrinsics set, each camera's fix lists are held during the rig bundle adjustment.

#![allow(unused)]
fn main() {
use vision_calibration::common::config::CameraOverride;
use vision_calibration::core::{CameraModelSpec, DistortionKind, SensorKind};

// Camera 1 has a tilted sensor and a rational lens; the others are straight Brown-Conrady.
session.update_config(|c| {
    c.camera_overrides = vec![CameraOverride::new(
        1,
        CameraModelSpec::pinhole(DistortionKind::Rational8, SensorKind::Scheimpflug),
    )];
})?;
}

A rig of tilted cameras sets config.camera = CameraConfig::new(CameraModelSpec::pinhole(DistortionKind::BrownConrady5, SensorKind::Scheimpflug)) instead. Tilt seeds go through IntrinsicsManualInit::per_cam_sensors and step_intrinsics_init_all_with_seed; zero tilts need no seed (see Scheimpflug and Laser Rigs). The export's cameras is one CameraParams per camera, stating the calibrated distortion and sensor.

The reference camera (rig.reference_camera_idx) is fixed at identity, which removes the rig's 6-DOF gauge freedom; no separate pose-gauge setting is needed.

Input Format

#![allow(unused)]
fn main() {
pub struct RigDataset<Meta> {
    pub num_cameras: usize,
    pub views: Vec<RigView<Meta>>,
}

pub struct RigView<Meta> {
    pub obs: RigViewObs,  // Per-camera observations (Option per camera)
    pub meta: Meta,
}
}

Each view has an Option<CorrespondenceView> per camera — cameras that don't see the target in a given view have None.

Per-Camera Reprojection Error

After optimization, per-camera reprojection errors are computed independently:

#![allow(unused)]
fn main() {
let export = session.export()?;
for (cam_idx, error) in export.per_cam_reproj_errors.iter().enumerate() {
    println!("Camera {}: {:.4} px", cam_idx, error);
}
println!("Overall: {:.4} px", export.mean_reproj_error);
}

Stereo Baseline

For a 2-camera rig, the baseline (distance between camera centers) is:

$baseline = ∥ T_{R, C_{1}} . translation ∥$

since $T_{R, C_{0}} = I$ (camera 0 is the reference).

Complete Example

#![allow(unused)]
fn main() {
use vision_calibration::prelude::*;
use vision_calibration::rig_extrinsics::*;

let mut session = CalibrationSession::<RigExtrinsicsProblem>::new();
session.set_input(rig_dataset)?;

step_intrinsics_init_all(&mut session, None)?;
step_intrinsics_optimize_all(&mut session, None)?;
step_rig_init(&mut session, None)?;
step_rig_optimize(&mut session, None)?;

let export = session.export()?;
let baseline = export.cam_se3_rig[1].translation.vector.norm();
println!("Stereo baseline: {:.1} mm", baseline * 1000.0);
}

OpenCV equivalence: cv::stereoCalibrate for 2-camera rigs. calibration-rs generalizes to $N$ cameras.

vision-calibration Book