The Toulouse Vanishing Point Dataset (TVPD)

Vincent Angladon1,2, Simone Gasparini1, Vincent Charvillat1

1 Université de Toulouse; INPT – IRIT www.irit.fr
2 Telequid www.telequid.com

Image understanding and vanishing points
  • Image segmentation
  • Find rectilinear structures
  • Spatial layout recovery
Joint 3D layout and object reasoning from single images [Schwing2013]
Camera and Inertial Motion Unit (IMU)
video/stabilized images ← → inertial data
A CMOS camera and an IMU sensor1
1: Bosch BMX055 IMU sensor, present in Project Tango
Fusion IMU -- vision
IMU Vision
Pros:
  • Data processing is instantaneous
  • Motion does not affect the quality of the measures
 Pros:
  • High accuracy
Cons:
  • Various noises
  • Important drift
 Cons:
  • Requires good luminosity
  • Sensitive to motion (rolling shutter and motion blur)
  • Image processing is often slow
A new dataset
A new dataset of still images with vanishing points and IMU data:
the Toulouse Vanishing Point Dataset
Outline
  1. Introduction
  2. Background
    1. Background on vanishing points
    2. Background on IMU
  3. The dataset
  4. Dataset tools
Vanishing Point definition
A classical example of vanishing point.
Manhattan World definition
Vanishing point and Manhattan world
The three Manhattan directions behind the three vanishing points form an orthogonal basis and can be used to define the world axis.

In this case: VP + camera parameters → camera orientation

Background on VPs detection

Compute features (gradients, edges, lines, line segments, ...)

Cluster the features (Hough, RANSAC, JLinkage, ...)

Compute the lines intersection for each cluster

The extracted line segments
The segments clustered
The estimated vanishing point in green on the first cluster of line segments
VP detection in a Manhattan world
Multiple strategies:
  • [Tardif2009] seeks the three vps that best satisfy the orthogonality constraint two by two
  • [Bazin2012] seeks the orthogonal frame which maximizes the number of clustered segments
  • [Antunes2013] orthogonality is enforced in the 2nd layer of his Hierarchical Facility Location problem formulation
IMU sensors
IMU -- Vanishing point relationship
The gravity vector provided by the IMU is aligned on the Z axis of the Manhattan scene.
Outline
  1. Introduction
  2. Background
  3. The dataset
    1. The other datasets
    2. Ground truth creation in the TVPD
  4. Dataset tools
The York Urban Database [Denis2008]
  • 102 images of outdoor and indoor scenes 640x480
  • Ground truth line segments
  • Ground truth VPs computed using [Collins1990]
  • Orthogonalization of the Manhattan directions
The problem with the York Database
Vanishing point and horizon line before (green square) and after (green diamond and dashed line) the orthogonalization of the Manhattan directions
Other vanishing point datasets
The PKU dataset [Li2012] and the Eurasian cities dataset [Barinova2010] with their ground truth line segments.
The Toulouse Vanishing Point Dataset
  • IMU data embedded in each photo
  • High resolution photos (1920x1080)
  • 114 photos (40 indoor and 74 outdoor)
  • Uncertainty regions for the vanishing points
When the lines are not perfect, which intersection is the best?
Uncertainty modelling
Line segment uncertainty modelling with circular regions on the endpoints. The area in grey is called double wedge [Shufelt1999], it is where the VP v should lie
Double wedges intersection
Intersection of the double wedges.
Properties of double wedges
Long line segments narrow down the uncertainty region, short one have no influence.
The computed uncertainty regions
The computed uncertainty regions and the horizon line computed from the inertial data.
Comparison on the York Urban Database
Comparison on a photo from the YUD. Our uncertainty regions are in black.
Outline
  1. Introduction
  2. Background
  3. The dataset
  4. Dataset tools
    1. Photo capture
    2. Line segment creation
    3. Computation of the uncertainty regions
Photo capture
The iOS capture application

→ Photo with embedded synchronized inertial data
stored in the EXIF UserComment field.

Requires the synchronization of the inertial data
with the camera frames.

Line segments creation
The web application used to annotate the images with line segments
Computation of the uncertainty regions
The uncertainty regions computed
Conclusion
  • A dataset under CC license
  • ... with inertial data
  • ... and uncertainty regions
  • Open source software
  • IMUs can ease the detection of VP
  • IMUs are game changer
Dataset webpage: http://ubee.enseeiht.fr/tvpd
References