(presentation page)
or go to:
home.heeere.com
@///////// TITLE
@: .title-slide /no-status
# // comment
- {{ nd.vars.author }} − {{ nd.vars['author-from'] }}
- {{ nd.vars.venue }}
- {{ nd.vars.date }}
- sources, updates and code available on github
(press to play)
@///////// END
@: .title-slide .centered /no-status
@for-copy
@: #endslide
# *// comment* {.centered}
- {{ nd.vars.author }} − {{ nd.vars['author-from'] }}
- {{ nd.vars.venue }}
- {{ nd.vars.date }}
- {.no-bullet}
- {.no-bullet}
@////////////////////////////////////////
@////////////////////////////////////////
@////////////////////////////////////////
@/////////////////////// overview/plan //
@////////////////////////////////////////
@sticky-add: @:.libyli
@eval-header: part = 0
@for-copy
@: #global .no-libyli .overview
## (Overview)
1. Why we need to quantify uncertainty?
1. Some sources of uncertainty
1. Statistical machine learning approaches \
for general uncertainty modeling
1. Deep Learning practices for uncertainty modeling
1. Bayesian Neural Networks
1. Bayesian view of machine learning
1. Variational inference
1. Variational Dropout
1. Applications and Openings
@////////////////////////////////////////
@////////// Motivation
@eval-header: return highlightLi(part+=1)
# @copy: global
## Why we need uncertainty quantification?
- Good decision making is based on some assessment of uncertainties
- Medical diagnosis
- Asymmetric costs situations
- Benefit/risk evaluation
- Multi-factorial decisions
- Self driving cars
- ...
@////////////////////////////////////////
@////////// Sources of uncertainty
@eval-header: return highlightLi(part+=1)
# @copy: global
## What is uncertainty?
> Uncertainty refers to epistemic situations involving **imperfect or unknown information**.
> \
> \
> It applies to predictions of **future events**, to **physical measurements** that are already made, or to the **unknown**. \
> Uncertainty arises in partially observable and/or stochastic environments, as well as due to **ignorance, indolence, or both**. \
> \
> It arises in any number of fields, including insurance, philosophy, physics, **statistics**, economics, finance, psychology, sociology, **engineering**, metrology, meteorology, ecology and information science.
> wikipedia
// machine learning is engineering of statistical software
## Sources of Uncertainty in Models
- Traditional ideal (deterministic) models, like rules in physics
- e.g., $x_{t+1} = f(x_{t})$ (e.g., dynamics, $f$ includes gravity etc)
- e.g., $Y = f(X)$ (e.g., ideal gas law, 
@:.helped-svg
## Uncertainty in Machine Learning: classification (2D "input", binary)
@:.helped-svg
## Toy 2D-Dataset: *Aleatoric*{.alea} Uncertainty
@:.helped-svg
## Toy 2D-Dataset: *Epistemic*{.epi} Uncertainty
## A Quick Visual Summary on Uncertainty
## How Neural Nets do Classification? *(reminder) (example with 3 classes)*{.dense}
- @anim: NOPE | #softmax | #output | #logits
- $o = softmax(l) \Leftrightarrow \forall i, o_i = \frac{\exp(l_i)}{\sum_j{\exp(l_j)}}$ {style="margin-top: -1em;"}
## A Probability for each class?
- A probability vector is better than just predicting a class
- parallel with a regression setting
- instead of predicting a single output value
- predict a distribution (e.g., a mean and variance)
- But... ambiguity: *aleatoric*{.alea} vs *epistemic*{.epi}
- what is *actually uncertain*{.alea} *(in the current representation space)*{.dense}
- what the model *doesn't know*{.epi}
- A probability vector cannot convey all information {.challenge}
(but a good probability vector can be enough for some decision making)
{.denser .centered style="margin-top: -1em"}
## 50% dog, 50% plane?
- *aleatoric*{.alea} vs *epistemic*{.epi}
- Example setup
- a model trained to distinguish 2 classes, dog vs plane
- on a new image, the network predicts 50%/50%
- two possible situations
Case 1 !
{.halfwidth .centered .will-alea .step}
Case 2 !
{.halfwidth .centered .will-epi .step style="vertical-align: top"}
@anim: %+alea: .will-alea
@anim: %+epi: .will-epi +
## ReLU Networks are Overconfident (Hein et al., CVPR2019)
{style="position: absolute; top: 10px; right: 0;"}
- Over-confident predictions
- A deep model doesn't know *what it doesn't know*{.epi} {.challenge}
@anim: .moons
- NB: it is also over-confident in *regions of inherent uncertainty*{.alea} {.dense}
( Image from the companion-webpage https://github.com/max-andr/relu_networks_overconfident of:
@anim: .last
## Support Vector Machines (SVM) Bagging Example
@anim: .last
## SVM Bagging Example with polynomial features (kernel) // deg3
@anim: .last
## Multi-Layer Perceptron (MLP) Bagging Example // [100, 50, 20]
@anim: .last
## Very-Deep MLP Bagging Example // [10]*10
@anim: .last
## Summary on bagging
- Bagging nicely handles quantifying *aleatoric uncertainty*{.alea} {.challenge}
- It works even with overconfident base models
- The sampling creates the necessary noise in *boundary regions*{.alea}
- The model family controls Bagging's *Out of Distribution behavior*{.epi} {.challenge}
- Simple models have low OOD variety ⇒ 100% over-confidence
- Complex/varied models/features better assess *epistemic uncertainty*{.epi}
- However, still a major OOD over-confidence *(like most discriminative models)*{.dense}
{.dense}
@: .slide_75663798
## Zooming out to see Out Of Distribution Over-confidence
Re-seeding for stochastic methods
- works very well in practice
- resource intensive (mem., process)
- some variations/optimizations
- snapshot + cyclic learning rate
- 
Different families of models
- different hyper-parameters
- different architectures
@: .slide_66893590 .dense
## Stochastic Learning as Model Ensembling
Re-seeding, Dropout, ... 
- "Bayesianism"
- Everything as random variables
- Use (conditional) probabilities ... a lot // A generalization of traditional logic
- .{.empty}
- Two probability rules
- .{.empty}
- Product rule:
$P(A, B) = P(A|B) ~ P(B)$ = $P(B|A) ~ P(A)$
{.dense}
- .{.empty}
- Marginalization, Sum rule:
$P(B) = \sum_A P(A,B) \triangleq \sum_a P(A=a,B)$
{.dense}
- @anim: .floatright +
- And in the
- Use probabilities to
- represent non-deterministic laws
- represent uncertainty (*aleatoric*{.alea} and *epistemic*{.epi})
- reason about uncertainty (do learning, inference)
{.dense}
- Considering
- some parameters (e.g., weights of the network, $W$)
- some dataset (e.g., both training inputs and labels, $X$)
- We have *﹏*{.pen}
- $P(W|X) = \frac{P(X|W) ~ P(W)}{P(X)} \propto P(X|W) ~ P(W)$
- More verbosely
- $P_{posterior}(weights | trainset) = \frac{P_{likelihood}(trainset|weights) ~ P_{prior}(weights)}{P_{constant}(trainset)}$
{.denser .no-bullet .challenge}
- Posterior probability
- probability distribution of the parameters given the training set
- i.e. what we know about the parameters after seeing the training set
{.dense}
## Principle of Bayesian Neural Networks
@anim: .weight | .bump | #small-bumps
- Typical BNN: have a 1D Normal distribution on each weight
- 1 mean and 1 variance per weight
- 1 billion weights ⇒ 2 billions parameters
{style="margin-top: -.2em;" }
@////////// BNN: VI
@eval-header: return highlightLi(part, subpart+=1)
# @copy: global
## Training a Bayesian Neural Network
- Bayesian "learning" {.libyli}
- Goal: finding the 
- (Biraja Ghoshal, Allan Tucker)
- dropweight on a BCNN
- improve classification
- good quantification
- human/machine combination
@: .no-libyli .paper-with-image .two-lines
## Towards safe deep learning: accurately quantifying biomarker uncertainty in neural network predictions
- (Zach Eaton-Rosen, Felix Bragman, Sotirios Bisdas, Sebastien Ourselin, M. Jorge Cardoso)
{style="max-width:300px"}
@: .no-libyli .paper-with-image
## ... (cont)
@: .no-libyli .paper-with-image .two-lines
## Propagating uncertainty across cascaded medical imaging tasks for improved deep learning inference
- Raghav Mehta, Thomas Christinck, Tanya Nair, Paul Lemaitre, Douglas L. Arnold, Tal Arbel
{style="max-width:300px"}
## Gaussian processes (GP)
(Avoiding pathologies in very deep networks, Duvenaud et al.)
## Going beyond probabilities?
- Probabilities are a way of represent belief
- It might be necessary to also represent confidence
- Some possible directions
- 
(e.g., Deep Evidential Regression (above))