Fast Attention Over Long Sequences With Dynamic Sparse Flash Attention

This work extends FlashAttention to accommodate a large class of attention sparsity patterns that, in particular, encompass key/query dropping and hashing-based attention, leading to implementations with no computational complexity overhead and a multi-fold runtime speedup on top of FlashAttention.