Adaptive Hessian-free Optimization

• Adaptive Hessian-free optimization for training neural networks
  • NeurIPS Workshop on OPT2020: 12th Annual Workshop on Optimization for Machine Learning
• paper: https://opt-ml.org/papers/2020/paper_43.pdf
• poster: https://opt-ml.org/posters/2020/poster_43.png

Required

• Docker version >= 19.03.2
• docker-compose version >= 1.24.1

  Set up

  Build container

  $ cd build-env
  $ docker-compose build
  $ docker-compose run adaptive-hf /bin/bash

Training model using Adaptive-HF

Example

$ (adaptive-hf) cd /code/adaptive-hessian-free-optimization
$ (adaptive-hf) python exp/training.py

Argments

• —arch: model architecture, default is ‘LeNet’.
• —gpu-num: GPU Device number, default is 1.
• —seed: random seed for training model, default is 1.
• —batch-size: input batch size for training model, default is 128.
• —epochs: number of epochs for training model, default is 10.
• —lr: learning rate, default is 0.001.
• —damping: damping rate for constract a positive-definite Hessian matrix, default is 10.
• —beta2: hyperparameter of Adam, default is 1e-8.

The following three parametes are the unique hyperparameters of Adaptive-HF.

• —cg-epsilon: default is 1e-3.
• —cg-sigma: the upper bound of the conjugate gradients, default is 50.
• —cg-L-smoothness: the lower bound of L-smoothness, default is 100.

Comparison between Original-HF and Adaptive-HF