Bridging Nonlinearities and Stochastic Regularizers with Gaussian Error Linear Units
Bridging Nonlinearities and Stochastic Regularizers with Gaussian Error Linear Units
Dan Hendrycks,Kevin Gimpel
2016 · DBLP: journals/corr/HendrycksG16
arXiv.org · 779 Citações
TLDR
An empirical evaluation of the GELU nonlinearity against the ReLU and ELU activations and finding performance improvements across all tasks suggests a new probabilistic understanding of nonlinearities.