An Ensemble of Epoch-wise Empirical Bayes for Few-shot Learning

Paper Code Slides Video

Abstract

Few-shot learning aims to train efficient predictive models with a few examples. The lack of training data leads to poor models that perform high-variance or low-confidence predictions. In this paper, we propose to meta-learn the ensemble of epoch-wise empirical Bayes models (E³BM) to achieve robust predictions. "Epoch-wise" means that each training epoch has a Bayes model whose parameters are specifically learned and deployed. "Empirical" means that the hyperparameters, e.g., used for learning and ensembling the epoch-wise models, are generated by hyperprior learners conditional on task-specific data. We introduce four kinds of hyperprior learners by considering inductive vs. transductive, and epoch-dependent vs. epoch-independent, in the paradigm of meta-learning. We conduct extensive experiments for five-class few-shot tasks on three challenging benchmarks: miniImageNet, tieredImageNet, and FC100, and achieve top performance using the epoch-dependent transductive hyperprior learner, which captures the richest information. Our ablation study shows that both "epoch-wise ensemble" and "empirical" encourage high efficiency and robustness in the model performance.

Contributions

A novel approach E³BM that learns to learn and combine an ensemble of epoch-wise Bayes models for more robust few-shot learning;
Novel hyperprior learners in E³BM to generate the task-specific hyperparameters for learning and combining epochwise Bayes models;
Extensive experiments on three challenging few-shot benchmarks, miniImageNet, tieredImageNet and Fewshot-CIFAR100 (FC100).

Presentation

Our Method: E³BM

Fig 1. Conceptual illustrations of the model adaptation on the blue, red and yellow tasks. (a) MAML is the classical inductive method that meta-learns a network initialization θ that is used to learn a single base-learner on each task. (b) SIB is a transductive method that formulates a variational posterior as a function of both labeled training data T^(tr) and unlabeled test data x^(te). It also uses a single base-learner and optimizes the learner by running several synthetic gradient steps on x^(te). (c) Our E³BM is a generic method that learns to combine the epoch-wise base-learners, and to generate task-specific learningcrates α and combination weights v that encourage robust adaptation.

Fig 2. The computing flow of the proposed E³BM approach in one meta-training episode. For the meta-test task, the computation will be ended with predictions. Hyperlearner predicts task-specific hyperparameters, i.e., learning rates and multi-model combination weights. When its input contains x^(te), it is transductive, otherwise inductive. Its detailed architecture is given in Fig. 3.

Fig 3. Two options of hyperprior learner at the m-th base update epoch. In terms of the mapping function, we deploy either FC layers to build epoch-independent hyperprior learners, or LSTM to build an epoch-dependent learner. Values in dashed box were learned from previous tasks.

Performance

Table 1. The 5-class few-shot classification accuracies (%) on miniImageNet, tieredImageNet, and FC100. “(+time, +param)” denote the additional computational time (%) and parameter size (%), respectively, when plugging-in E³BM to baselines (MAML, MTL and SIB).

Citation

Please cite our paper if it is helpful to your work:

@inproceedings{Liu2020E3BM,
  author    = {Liu, Yaoyao and
               Schiele, Bernt and
               Sun, Qianru},
  title     = {An Ensemble of Epoch-wise Empirical Bayes for Few-shot Learning},
  booktitle = {European Conference on Computer Vision (ECCV)},
  year      = {2020}
}

References

[1] Finn, Chelsea, Pieter Abbeel, and Sergey Levine. "Model-agnostic meta-learning for fast adaptation of deep networks." ICML 2017.
[2] Hu, Shell Xu, et al. "Empirical Bayes Transductive Meta-Learning with Synthetic Gradients." ICLR 2020.

Contact

If you have any questions, please feel free to contact us via email: yyliu@cs.jhu.edu.