Homography Cheirality
Why the homography cheirality check is "all $\lambda_i$ have the same sign".
1. Setup
Consider two pinhole cameras observing a single world plane $\pi$. Working in camera 1's coordinate frame, let $\boldsymbol{n}\in\mathbb{R}^3$ be the plane's unit normal and $d>0$ its signed distance from the camera centre, so the plane equation is
\[\boldsymbol{n}^{\mathsf{T}} X \;=\; d, \qquad X \in \pi \tag{1}\]
with $d>0$ meaning the plane is in front of camera 1.
Let $(R,\boldsymbol{t})$ be the rigid transformation from camera 1 to camera 2, and $K_1, K_2$ be the intrinsics. A 3-D point $X_{\text{cam}_{1}}$ in camera 1's frame maps to
\[X_{\text{cam}_{2}} \;=\; R\,X_{\text{cam}_{1}} + \boldsymbol{t}. \tag{2}\]
Its homogeneous pixel projections are $x = K_1 X_{\text{cam}_{1}}$ and $x' = K_2 X_{\text{cam}_{2}}$.
2. Plane-Induced Homography
For any $X\in\pi$ we have $\boldsymbol{n}^{\mathsf{T}} X = d$, i.e. $\boldsymbol{n}^{\mathsf{T}} X / d = 1$. Substitute into the inter-camera transform:
\[\begin{aligned} X_{\text{cam}_{2}} &= R\,X_{\text{cam}_{1}} + \boldsymbol{t} \\ &= R\,X_{\text{cam}_{1}} + \boldsymbol{t} \cdot \frac{\boldsymbol{n}^{\mathsf{T}} X_{\text{cam}_{1}}}{d} \\ &= \left(R + \tfrac{1}{d}\,\boldsymbol{t}\,\boldsymbol{n}^{\mathsf{T}}\right) X_{\text{cam}_{1}}. \end{aligned} \tag{3}\]
Adding the intrinsics, the pixel-space homography induced by the plane is
\[\boxed{\;H \;=\; K_2\!\left(R + \tfrac{1}{d}\,\boldsymbol{t}\,\boldsymbol{n}^{\mathsf{T}}\right) K_1^{-1}.\;} \tag{4}\]
3. Projective Scale Factor λᵢ
Write the correspondence in homogeneous image coordinates:
\[x'_i \;=\; \lambda_i \, H \, x_i. \tag{5}\]
We show that
\[\boxed{\;\lambda_i \;=\; \dfrac{Z_{1,i}}{Z_{2,i}}\;} \tag{6}\]
where $Z_{1,i}$ and $Z_{2,i}$ are the depths (i.e. $z$-components) of the 3-D point $X_i$ in camera 1's and camera 2's frames respectively.
Derivation. Let $u_i = K_1^{-1} x_i$ and $u'_i = K_2^{-1} x'_i$ be the normalized image points. From the projection equations,
\[u_i = \frac{X_{\text{cam}_{1},i}}{Z_{1,i}}, \qquad u'_i = \frac{X_{\text{cam}_{2},i}}{Z_{2,i}}. \tag{7}\]
Multiplying the normalized-frame plane homography by $X_{\text{cam}_{1},i}$,
\[\left(R + \tfrac{1}{d}\boldsymbol{t}\boldsymbol{n}^{\mathsf{T}}\right) X_{\text{cam}_{1},i} \;=\; X_{\text{cam}_{2},i}, \tag{8}\]
and dividing both sides by $Z_{1,i}$,
\[\left(R + \tfrac{1}{d}\boldsymbol{t}\boldsymbol{n}^{\mathsf{T}}\right) u_i \;=\; \frac{X_{\text{cam}_{2},i}}{Z_{1,i}} \;=\; \frac{Z_{2,i}}{Z_{1,i}} \, u'_i. \tag{9}\]
Equivalently,
\[u'_i \;=\; \frac{Z_{1,i}}{Z_{2,i}} \left(R + \tfrac{1}{d}\boldsymbol{t}\boldsymbol{n}^{\mathsf{T}}\right) u_i. \tag{10}\]
Left-multiplying by $K_2$ and using $K_1^{-1} x_i = u_i$ on the right gives (5) with $\lambda_i = Z_{1,i}/Z_{2,i}$, as claimed.
4. Cheirality
For a physically meaningful correspondence, the 3-D point $X_i$ must lie in front of both cameras:
\[Z_{1,i} > 0 \quad\text{and}\quad Z_{2,i} > 0. \tag{11}\]
Therefore
\[\lambda_i \;=\; \frac{Z_{1,i}}{Z_{2,i}} \;>\; 0 \qquad \forall i. \tag{12}\]
Invariance under $H\mapsto -H$. The homography is a projective object, defined up to a non-zero global scale. Replacing $H\mapsto -H$ sends $\lambda_i\mapsto -\lambda_i$simultaneously for every $i$, so the cheirality condition is really
\[\text{“}\lambda_i \text{ all have the same sign”.} \tag{13}\]
In practice one may normalize $H$ so that $\det H > 0$; after this normalization the condition becomes the one-sided test
\[\lambda_i > 0 \quad \forall i. \tag{14}\]
VisualGeometryToolkitCore's test_cheirality tests the sign-uniform form (13) directly, which needs no sign normalization of $H$.
5. When Signs Disagree
Mixed-sign $\lambda_i$'s appear precisely when the 4 correspondences cannot be explained by a single plane in front of both cameras:
- Outlier correspondence. The pair $(x_i,x'_i)$ does not come from a common 3-D point, so the depth-ratio identity (6) does not apply and $\lambda_i$ can take any sign.
- Bow-tie / non-planar sample. The 4 points cannot lie on a single plane in the shared-visibility half-space; some subset gets placed behind a camera, flipping the sign of its $\lambda$.
- Near-degenerate sample (e.g. three collinear points). Numerical error in the 4-point solver pushes some $\lambda_i$ slightly negative.
All three cases are rejected by the same all-positive-$\lambda$ test.