Preprint | 2024

Pretraining with random noise for uncertainty calibration

Jeonghwan Cheon, Sang Wan Lee, Se-Bum Paik

Abstract

Uncertainty calibration, the process of aligning confidence with accuracy, is a hallmark of human intelligence. However, most machine learning models struggle to achieve this alignment, particularly when the training dataset is small relative to the network's capacity. Here, we demonstrate that uncertainty calibration can be effectively achieved through a pretraining method inspired by developmental neuroscience. Specifically, training with random noise before data training allows neural networks to calibrate their uncertainty, ensuring that confidence levels are aligned with actual accuracy. We show that randomly initialized, untrained networks tend to exhibit erroneously high confidence, but pretraining with random noise effectively calibrates these networks, bringing their confidence down to chance levels across input spaces. As a result, networks pretrained with random noise exhibit optimal calibration, with confidence closely aligned with accuracy throughout subsequent data training. These pre-calibrated networks also perform better at identifying "unknown data" by exhibiting lower confidence for out-of-distribution samples. Our findings provide a fundamental solution for uncertainty calibration in both in-distribution and out-of-distribution contexts.

Highlights

• Deep neural networks often struggle to properly calibrate both accuracy and confidence.
• Pretraining with random noise aligns confidence levels with actual accuracy.
• Random noise pretraining facilitates meta-learning by aligning confidence with the chance level.
• Pre-calibrated networks assess both known and unknown datasets through confidence estimation.

Preprint and Proceedings

• • Pretraining with random noise for uncertainty calibration (2024)
  • Jeonghwan Cheon, Se-Bum Paik
  • arXiv 2412.17411
  Preprint Bibtex

Conferences

• • Computational and Systems Neuroscience (COSYNE), Mar 2025, Montreal, QC, CN.
  • Jeonghwan Cheon, Se-Bum Paik.
  • ”Uncertainty Calibration through Pretraining with Random Noise”