Dask: Parallel Computation with Blocked algorithms and Task Scheduling
Dask: Parallel Computation with Blocked algorithms and Task Scheduling
M. Rocklin
2015 · DOI: 10.25080/MAJORA-7B98E3ED-013
700 Citations
TLDR
This work couple blocked algorithms with dynamic and memory aware task scheduling to achieve a parallel and out-of-core NumPy clone and shows how this extends the effective scale of modern hardware to larger datasets.
Abstract
Dask enables parallel and out-of-core computation. We couple blocked algorithms with dynamic and memory aware task scheduling to achieve a parallel and out-of-core NumPy clone. We show how this extends the effective scale of modern hardware to larger datasets and discuss how these ideas can be more broadly applied to other parallel collections.