Generative Pretraining From Pixels

It is found that a GPT-2 scale model learns strong image representations as measured by linear probing, fine-tuning, and low-data classification, despite training on low-resolution ImageNet without labels.