Fundamental Matrix

The fundamental matrix is a matrix that captures the epipolar geometry between two images of the same scene.

It is one of the main tools used in stereo vision and 3D reconstruction from two views.

Core Constraint §

For a point in the first image and its matching point in the second image,

This is called the epipolar constraint.

It means:

• a point in image 1 determines an epipolar line in image 2
• the matching point in image 2 must lie on that line

Geometric Meaning §

The fundamental matrix maps image points to epipolar lines:

where is the epipolar line in the second image.

Similarly,

gives the epipolar line in the first image.

So instead of searching for a match anywhere in the second image, we only need to search along one line.

Why It Matters for 3D Reconstruction §

In two-view reconstruction, the hard part is finding correct correspondences between the two images.

The fundamental matrix helps by enforcing a geometric consistency rule:

• matched points must satisfy
• bad matches can be rejected because they do not satisfy the epipolar constraint
• once reliable matches are found, the corresponding camera rays can be triangulated to recover 3D points

So the fundamental matrix does not directly give 3D points, but it gives the geometry needed to make triangulation possible.

Reconstruction Pipeline §

A typical two-view pipeline is:

1. detect feature points in both images
2. match features across the two views
3. estimate the fundamental matrix from the matches
4. remove outliers using the epipolar error
5. recover camera geometry
6. triangulate matched points to estimate the 3D scene

This is why the fundamental matrix is central in structure-from-motion and stereo reconstruction.

Key Facts §

• is a matrix
• it is defined up to scale
• it has rank 2
• it depends only on the relative geometry of the two cameras
• it works with pixel coordinates, so camera calibration is not required

Epipoles §

The epipoles are the image points where the baseline between the two camera centers intersects each image plane.

They satisfy

and

where and are the epipoles in the two images.

All epipolar lines in one image pass through that image’s epipole.

Relation to the Essential Matrix §

The essential matrix is the calibrated version of the fundamental matrix.

If and are the camera intrinsic matrices, then

and

So:

• use the fundamental matrix when working with pixel coordinates from uncalibrated cameras
• use the essential matrix when camera intrinsics are known

Limitation §

The fundamental matrix alone does not determine the true Euclidean 3D structure of the scene.

Without calibration, reconstruction is only determined up to a projective transformation.

To recover metric 3D structure, camera calibration is needed, which leads to the essential matrix.