Compete to Compute

This paper applies the concept of local competition among neighboring neurons to gradient-based, backprop-trained artificial multilayer NNs, finding that NNs with competing linear units tend to outperform those with non-competing nonlinear units, and avoid catastrophic forgetting when training sets change over time.