On Network Design Spaces for Visual Recognition

Read Paper

Which metric is a better and more robust way to compare model families (e.g. ResNets vs VGG family) than the traditional way of point estimates or a curve estimate of a handful of picked models (see Fig 1 (a) and (b))

Comparing empirical distribution functions (EDFs) is a better way. Specifically one can compare the **error EDF: $F(e) = \frac{1}{n} \sum^n_{i=1} \mathbb{1}[e_i < e]$ ** $F(e)$ gives the fraction of models with error less than e.

What is an issue with the error EDF $F(e) = \frac{1}{n} \sum^n_{i=1} \mathbb{1}[e_i < e]$ when comparing distributions of different network families, and how can it be solved?

It does not control for confounding factors like network complexity. Model families which in general produce models of a larger complexity tend to have a better error.

This can be solved by working with a normalized error EDF: $F(e) = \frac{1}{n} \sum^n_{i=1} w_i\mathbb{1}[e_i < e]$

In practice, we set the weights for a model set such that its complexity distribution is uniform. Specifically, we bin the complexity range into $k$ bins, and assign each of the $m_j$ models that fall into a bin $j$ a weight $w_j = \frac{1}{km_j}$ .