The Visual Localization Benchmark

Visual localization is the problem of estimating the 6 Degree-of-Freedom (DoF) camera pose from which a given image was taken relative to a reference scene representation. Visual localization is a key technology for applications such as Augmented, Mixed, and Virtual Reality, as well as for robotics, e.g., for self-driving cars. In order to evaluate visual localization over longer periods of time, we provide benchmark datasets aimed at evaluating 6 DoF pose estimation accuracy over large appearance variations caused by changes in seasonal (summer, winter, spring, etc.) and illumination (dawn, day, sunset, night) conditions. Each dataset consists of a set of reference images, together with their corresponding ground truth poses, and a set of query images. A triangulated 3D model is provided for each dataset and can be used by structure-based localization approaches. To ensure fairness and comparability of results, the reference poses for the query images is withheld and we provide an evaluation service to measure pose accuracy.

Benchmark highlights:

New challengesQuery images taken under different seasonal and illumination conditions
Varied datasets Datasets taken with hand-held cameras and from vehicle-mounted cameras, covering various geographical locations
Unified evaluation protocol Both the position and orientation accuracy are taken into account.

Updates

2020-06-18: Enabled submission to the RobotCar-Seasons v2 dataset. This consists of the same database and query images as the original RobotCar-Seasons dataset, but uses a different test-train split.

2020-06-18: The Aachen Day-Night v 1.1 dataset has been added. This is an extension of the Aachen Day-Night dataset that contains additional night-time queries. Please refer to the corresponding arXiv paper for more information. Additionally, the reference poses for the night-time query images in the original Aachen Day-Night dataset have been updated to more accurate poses. The results in the Aachen Day-Night table may therefore have changed. Since we consider the new night-time poses to be more accurate, we have also changed the error thresholds to be the same as the ones for daytime.

2020-01-21: Added the Symphony Seasons dataset

2019-04-15: Submissions for the InLoc dataset and the SILDa Weather and Time of Day datasets are now available.