Single-Camera Hand-Eye Calibration

Hand-eye calibration estimates the rigid transform between a camera and a robot's gripper (or base). This workflow combines intrinsics calibration with hand-eye estimation in a 4-step pipeline.

Problem Formulation

Transformation Chain (Eye-in-Hand)

For a camera mounted on the robot gripper:

$T_{C, T} = T_{C, G} \cdot T_{G, B} \cdot T_{B, T}$

where:

$T_{C, G} = T_{handeye}^{- 1}$ : camera-to-gripper (the calibrated hand-eye transform, inverted)
$T_{G, B} = T_{robot}^{- 1}$ : gripper-to-base (known from robot kinematics, inverted)
$T_{B, T}$ : base-to-target (the target's pose in the robot base frame, also calibrated)

Equivalently:

$T_{C, T} = (T_{robot} \cdot T_{handeye})^{- 1} \cdot T_{B, T}$

Eye-to-Hand Variant

For a fixed camera observing a target on the gripper:

$T_{C, T} = T_{C, B} \cdot T_{B, G} \cdot T_{G, T}$

where $T_{C, B}$ is the camera-to-base transform and $T_{G, T}$ is the target-to-gripper transform.

Optimization Objective

Minimize reprojection error across all views, jointly over all parameters:

$K, d, X, T_{B, T} min v = 1 \sum M j = 1 \sum N_{v} π (K, d, T_{C, T}^{(v)}, P_{j}) - p_{v j}^{2}$

where $T_{C, T}^{(v)}$ depends on the known robot pose for view $v$ , the hand-eye transform $X$ , and the target pose $T_{B, T}$ .

4-Step Pipeline

Step 1: Intrinsics Initialization (`step_intrinsics_init`)

Runs the Iterative Intrinsics solver to estimate $K$ and distortion from the calibration board observations, ignoring robot poses entirely.

Step 2: Intrinsics Optimization (`step_intrinsics_optimize`)

Runs planar intrinsics bundle adjustment to refine $K$ , distortion, and per-view camera-to-target poses. This gives accurate per-view poses $T_{C, T}^{(v)}$ needed for hand-eye initialization.

Step 3: Hand-Eye Initialization (`step_handeye_init`)

Solves the hand-eye and the target pose in closed form with the method of handeye_init.method: Tsai-Lenz by default, or Park-Martin, Daniilidis or Shah (see Hand-Eye Linear):

Compute relative motions from robot poses and camera poses
Filter pairs by rotation magnitude (reject small rotations)
Estimate the hand-eye rotation and translation
Estimate $T_{B, T}$ : through the first view, or over all views with Shah's method, which estimates it together with the hand-eye

Step 4: Hand-Eye Optimization (`step_handeye_optimize`)

Joint non-linear optimization of all parameters:

Parameter blocks:

"cam" (4D, Euclidean): intrinsics
"dist" (Euclidean): the distortion coefficients of the configured model (absent without distortion)
"sensor" (2D, Euclidean): the sensor tilt, for a Scheimpflug camera
"handeye" (7D, SE3): gripper-to-camera transform
"target" (7D, SE3): base-to-target transform

Residual blocks: one ReprojPoint { model, chain: Handeye { .. }, .. } per observation, model being the camera's CameraModelSpec.

Without robot-pose refinement the robot poses are not optimized: they are known constants from the robot kinematics (passed as per-residual data).

Robot poses carry errors of their own, usually larger than the corner noise once projected through the lever arm to the board: 0.05° at 1 m moves the camera's view by almost a millimetre. Taken as exact, these errors end up in the camera parameters. robot_poses.refine (on by default) gives every view an SE(3) correction of its measured pose:

Additional parameter blocks: "robot_delta/v" (6D, Euclidean se(3) tangent [ω, v]) per view, applied in the gripper frame as base_se3_gripper · exp(δ): ω is the tool's orientation error and v its position error

Additional residuals: a zero-mean Se3TangentPrior per view, with the robot's rotation and translation standard deviations σ_r (rot_sigma) and σ_t (trans_sigma), weighed against the corner residuals at their pixel noise σ_px:

$r_{prior} = σ_{p x} [ω / σ_{r} v / σ_{t}]$

The objective is then the maximum-likelihood one, in units of σ_px²: a correction costs as much as the corner evidence it explains away. σ_px is not known in advance, so the solve runs once at 1 px, takes σ_px from the corner residuals and runs again from that solution. Every view's correction is free; the priors are what tell the corrections apart from a change of the hand-eye or target pose that would move every view alike.

The priors' standard deviations decide how far the result trusts the robot. Too loose, and the hand-eye and target standard deviations come out larger than the data warrants; too tight, and the robot's errors leak into the camera parameters while the standard deviations come out too small. So by default (robot_poses.estimate_sigma) the solve estimates them: rot_sigma and trans_sigma (0.5°, 1 mm) are only its starting values. Each prior's variance factor is the sum of its squared normalized corrections over its redundancy, ŝ² = Σ (δₖ/σ)² / Σ (1 − Var(δₖ)/σ²), the share of each correction the corner data do not already determine. σ is scaled by ŝ and the solve repeats until both factors are 1. The export reports the estimate as robot_pose_sigma, the robot's orientation and position error in this calibration. On synthetic scenes it recovers the simulated error from a prior ten times too tight or too loose, and the normalized errors of the hand-eye and target come out with an RMS of 1.02–1.13. With estimate_sigma off, the configured values are taken as the robot's true accuracy.

A correction common to every view looks exactly like a change of the transform carried by the gripper: the hand-eye in eye-in-hand, the target's pose in the gripper in eye-to-hand. That transform is therefore known only as well as the robot's mean error over the views: its rotation's standard deviation is at least about σ_r / √views, its translation's σ_t / √views.

Configuration

Settings are grouped by stage; the shared sub-structs (IntrinsicsInitConfig, CameraConfig, HandeyeInitConfig, SolverConfig, RobotPoseConfig) are reused across every hand-eye problem type:

#![allow(unused)]
fn main() {
pub struct SingleCamHandeyeConfig {
    // Per-camera linear-initialization stage settings.
    pub intrinsics: IntrinsicsInitConfig,
    // { iterations: usize = 2, fix_k3: bool = true }; it fits only the
    //   distortion terms the camera's solves estimate

    // The camera model (any distortion kind, an identity or Scheimpflug
    // sensor) and the parameters its solves hold fixed, named per slot.
    pub camera: CameraConfig, // { model: CameraModelSpec, fix: CameraFix }

    // Hand-eye linear-initialization stage settings (Tsai-Lenz DLT).
    pub handeye_init: HandeyeInitConfig,
    // { handeye_mode: HandeyeMode, method: HandeyeMethod = TsaiLenz,
    //   min_motion_angle_deg: f64 = 5.0 }

    // Non-linear solve stage settings.
    pub solver: SolverConfig, // { max_iters, verbosity, robust_loss }

    // Robot-pose refinement settings for the hand-eye bundle adjustment.
    pub robot_poses: RobotPoseConfig,
    // { refine: bool = true, rot_sigma: f64, trans_sigma: f64 }
}
}

Input Format

Each view is a View<RobotPoseMeta> (aliased as SingleCamHandeyeView); RobotPoseMeta is in vision_calibration::common, shared with the rig hand-eye workflows:

#![allow(unused)]
fn main() {
pub struct RobotPoseMeta {
    pub base_se3_gripper: Iso3,  // Robot pose (from kinematics)
}

// Type alias
pub type SingleCamHandeyeView = View<RobotPoseMeta>;
// Where View<Meta> has fields: obs: CorrespondenceView, meta: Meta
}

Mounting Modes

Poses follow the frame_se3_frame naming: base_se3_gripper reads "gripper pose expressed in the base frame" ( $T_{B, G}$ ). The unknowns are named the same way. The mounting mode is HandeyeInitConfig::handeye_mode:

HandeyeMode::EyeInHand (the default): the camera rides on the gripper and observes a fixed target. The calibration solves for gripper_se3_camera ( $T_{G, C}$ ) and base_se3_target ( $T_{B, T}$ ).
HandeyeMode::EyeToHand: the camera is fixed in the scene and observes a target mounted on the gripper. The calibration solves for camera_se3_base ( $T_{C, B}$ ) and gripper_se3_target ( $T_{G, T}$ ).

The export's four pose fields are mutually exclusive by mode: the active mode fills its pair and leaves the other pair None.

To switch to eye-to-hand:

#![allow(unused)]
fn main() {
use vision_calibration::optim::HandeyeMode;
use vision_calibration::single_cam_handeye::SingleCamHandeyeConfig;

let mut config = SingleCamHandeyeConfig::default();
config.handeye_init.handeye_mode = HandeyeMode::EyeToHand;
session.set_config(config)?;
}

In eye-to-hand mode the pixel-to-3D chain for a fixed camera observing a gripper-mounted target composes as $T_{C, T} = T_{C, B} \cdot T_{B, G} \cdot T_{G, T}$ , the same identity step_handeye_init_with_seed uses to derive camera_se3_base when only gripper_se3_target is seeded (see Seeding with Prior Knowledge).

Complete Example

#![allow(unused)]
fn main() {
use vision_calibration::prelude::*;
use vision_calibration::single_cam_handeye::*;

let views: Vec<SingleCamHandeyeView> = /* load data */;
let input = SingleCamHandeyeInput::new(views)?;

let mut session = CalibrationSession::<SingleCamHandeyeProblem>::new();
session.set_input(input)?;

step_intrinsics_init(&mut session, None)?;
step_intrinsics_optimize(&mut session, None)?;
step_handeye_init(&mut session, None)?;
step_handeye_optimize(&mut session, None)?;

let export = session.export()?;
println!("Camera: {:?}", export.camera); // CameraParams, in the configured model
println!("Hand-eye and target: {:?}", export.handeye);
println!("Reprojection error: {:.4} px", export.mean_reproj_error);
}

Reading the Export

#![allow(unused)]
fn main() {
use vision_calibration::single_cam_handeye::SingleCamHandeyeSolution;

match export.handeye {
    SingleCamHandeyeSolution::EyeInHand { gripper_se3_camera, base_se3_target } => {
        // gripper_se3_camera: T_G_C, base_se3_target: T_B_T
    }
    SingleCamHandeyeSolution::EyeToHand { camera_se3_base, gripper_se3_target } => {
        // camera_se3_base: T_C_B, gripper_se3_target: T_G_T
    }
}
}

export.handeye holds exactly the two poses of the mode, so an export cannot carry the wrong pose for its mode. In JSON it is the handeye object, tagged by mode:

"handeye": {
  "mode": "EyeInHand",
  "gripper_se3_camera": { "rotation": [0, 0, 0, 1], "translation": [0.05, 0, 0.1] },
  "base_se3_target":    { "rotation": [0, 0, 0, 1], "translation": [0, 0, 1] }
}

In Python, result.handeye is a CameraEyeInHand or a CameraEyeToHand, and result.handeye.mode names the mode.

export.camera is the calibrated camera (CameraParams) in the configured model, export.robot_deltas the per-view robot-pose corrections when refinement is on, and export.diagnostics.per_feature_residuals the per-corner reprojection records (see Per-Feature Residuals).

The SingleCamHandeyeInput::new constructor rejects an empty set of views and any view with fewer than 4 correspondences (the minimum for a homography). Each stage has a *_with_seed sibling and a typed step result (IntrinsicsInitResult, HandeyeOptimizeResult, ...); see Seeding with Prior Knowledge.

From a Dataset Manifest

Instead of assembling the views by hand, a manifest with topology = "single_cam_handeye" describes where the images, the target and the robot-pose file live. vision_calibration::dataset_runner::build_single_cam_handeye_input runs the (cached) corner detection and returns the SingleCamHandeyeInput; see The Dataset Manifest.

Camera Models

The camera model is the config's camera.model: any distortion kind (Brown-Conrady by default, rational, thin-prism, division, or none) and an identity or Scheimpflug sensor. The intrinsics stage dispatches on it like Planar Intrinsics: Zhang's method embedded into the model's distortion, and for a tilted sensor the linear tilt sweep followed by the staged tilt multi-start. A known tilt goes in the intrinsics manual-init sensor seed (step_intrinsics_init_with_seed); a tilt seed for an untilted camera must be zero. camera.fix names the parameters both solves hold.

Practical Considerations

Rotation diversity is critical for Tsai-Lenz initialization. Include rotations around at least 2 axes.
5-10 views minimum, with 10-20 recommended for robustness.
Robot accuracy: If the robot's reported poses have errors > 1mm or > 0.1°, enable robot pose refinement.
Initialization failure: If Tsai-Lenz produces a poor estimate (large reprojection error after optimization), the most likely cause is insufficient rotation diversity.

OpenCV equivalence: cv::calibrateHandEye provides the linear initialization step. calibration-rs adds joint non-linear refinement and optional robot pose correction.

vision-calibration Book