Master Thesis presentation by Alexander Bodin and Isak Meding

Examiner: Lennart Svensson

Abstract:
In recent years, machine learning models that can provide uncertainty estimates that match their observed accuracy have seen an increased interest in academia. Such models are called calibrated, a quality essential for the safe application of neural networks in high-stakes situations. However, good calibration is not enough – high predictive performance is also essential. Autonomous driving (AD) is a setting where this combination of model qualities is much-needed, with the additional requirements of real-time processing of sensor inputs such as camera video sequences. Deep ensembles (DEs) are state-of-the-art for non-Bayesian uncertainty quantification with high predictive performance. However, their deployment in AD has been limited due to their high computational load.

We propose deep ensembles spread over time (DESOTs), a simple modification to DEs that seeks to limit their computational load on image sequence data by letting a single ensemble member perform inference on each frame of the sequence. We apply this proposed system to the problem of traffic sign recognition (TSR), a subfield of AD with a distinctly long-tailed class distribution. DESOTs display predictive performance competitive with DEs for traffic sign classification, using only a fraction of the computational power. For in-distribution uncertainty performance, DESOTs outperform MC-dropout and perform on par with DEs. We conduct two out-of-distribution (OOD) experiments. First, we show that DESOTs increase calibration robustness to common augmentations compared to single models while matching DEs. Second, we test performance on a completely unseen class, for which all models increase their uncertainty in terms of output distribution entropy. Post-hoc calibration using temperature scaling is also evaluated and is shown to improve the uncertainty quantification performance of DESOTs, both in and out of distribution.

Keywords: Machine learning, artificial intelligence, computer vision, deep ensemble,
deep neural network, uncertainty quantification, calibration, traffic sign recognition.

Welcome!
Alexander, Isak and Lennart