COLA

(Unofficial)-Re-Impementation of the model archiecture of CONTRASTIVE LEARNING OF GENERAL-PURPOSE AUDIO REPRESENTATIONS in PyTorch.

Paper: CONTRASTIVE LEARNING OF GENERAL-PURPOSE AUDIO REPRESENTATIONS

Paper’s Official Code (In tensorflow): tf-code

Install

git clone https://github.com/CVxTz/COLA_pytorch
cd COLA_pytorch
python -m pip install .

Run

# Data download: download fma data and metadata
wget -c https://os.unil.cloud.switch.ch/fma/fma_metadata.zip
wget -c https://os.unil.cloud.switch.ch/fma/fma_small.zip
wget -c https://os.unil.cloud.switch.ch/fma/fma_large.zip
# Data preparation : prepare json with fma_small labels and pre-compute mel-spectrograms and save them as .npy
python supervised_examples/prepare_data.py --metadata_path "/media/ml/data_ml/fma_metadata/"
python audio_encoder/audio_processing.py --mp3_path "/media/ml/data_ml/fma_large/"
python audio_encoder/audio_processing.py --mp3_path "/media/ml/data_ml/fma_small/"
# Training
# Train with COLA
python audio_encoder/train_encoder.py --mp3_path "/media/ml/data_ml/fma_large/"
# Train Supervised
python supervised_examples/cnn_genre_classification.py --metadata_path "/media/ml/data_ml/fma_metadata/" \
  --mp3_path "/media/ml/data_ml/fma_small/"
python supervised_examples/cnn_genre_classification.py --metadata_path "/media/ml/data_ml/fma_metadata/" \
  --mp3_path "/media/ml/data_ml/fma_small/" \
  --encoder_path "models/encoder.ckpt"

Data

FMA Data

Description

This post is a short summary and steps to implement the following paper:

Learning of General-Purpose Audio
Representations

The objective of this paper is to learn self-supervised general-purpose audio
representations using Discriminative Pre-Training. The authors train a 2D CNN
EfficientNet-B0 to transform Mel-spectrograms into 1D-512 vectors. Those
representations are then transferred to other tasks like Speaker Identification
or Bird Song detection.

The basic idea behind DPT is to define an anchor element, a positive element,
and one or more distractors. A model is then trained to match the anchor with
the positive example.

DPT — Image By Author

One such way of using DPT is to use the triplet loss along with the Cosine
similarity measure to train the model such as Cosine(F(P), F(A)) is much higher
than Cosine(F(D), F(A)). This will make it so the representation in the latent
space of the Anchor is much closer to the Positive example than it is to the
Distractor. The authors of the paper linked above used this approach as a
baseline to show that their approach COLA works much better.

COLA

This approach is applied to the audio domain. For each audio clip, the authors
pick a segment to be the anchor and another to be the positive example, for each
of those samples (Anchor, Positive) they pick the other samples in the training
batch as the distractors. This is a great idea for two reasons:

There are multiple distractors, this makes the training task harder, forcing the
model to learn more meaningful representations to solve it.
The distractors are reused from other samples in the batch, which reduces the
IO, computing, and memory cost of generating them.

COLA also uses a Bi-linear similarity, which is learned directly from the data.
The authors show that Bi-Linear similarity works much better than Cosine, giving
an extra 7% average accuracy on downstream tasks in comparison.

After computing the similarity between the anchor and the other examples, the
similarity values are used in a cross-entropy loss that measures the model’s
ability to identify the positive example among the distractors (Eq 2 in the
paper).

COLA Evaluation

Linear Model Evaluation

COLA is used to train an EfficientNet-B0 on AudioSet, a dataset of around 1M
audio clips taken from YouTube. The feature vectors generated by the model are
then used to train a linear classifier on a wide range of downstream tasks. The
better the representations learned by the model are, the better its performance
will be when used as input to a linear model that performs a supervised task.
The authors found that COLA outperforms other methods like triplet loss by an
extra 20% average accuracy on downstream tasks ( Table 2 of the paper)

Fine-tuning Evaluation

Another way to test this approach is to fine-tune the model on downstream tasks.
This allowed the authors to compare a model pre-trained using COLA to one that
is trained from scratch. Their results show that the pre-trained model
outperforms the model trained from scratch by around 1.2% on average.

PyTorch Implementation

I don’t have the compute resources to reproduce the experiments of the paper, so
I tried to do something similar on a much smaller scale. I pre-trained a model
using COLA on FMA Large ( without the labels)
for a few epochs and then fine-tune on music genre detection applied to FMA
Small.

The results on FMA small are as follows:

Random guess: 12.5%
Trained from scratch: 51.1%
Pre-trained using COLA: 54.3%

Conclusion

The paper Learning of General-Purpose Audio
Representations introduces the COLA
pre-training approach, which implements some great ideas that make
self-supervised training more effective, like using batch samples as distractors
and the bi-linear similarity measure. This approach can be used to improve the
performance of downstream supervised audio tasks.

Code:
https://github.com/CVxTz/COLA_pytorch