Dask Array implements a subset of the NumPy ndarray interface using blocked algorithms, cutting up the large array into many small arrays. This lets us compute on arrays larger than memory using all of our cores. We coordinate these blocked algorithms using dask graphs.
Dask arrays coordinate many NumPy arrays arranged into a grid. These NumPy arrays may live on disk or on other machines.
Dask array used in fields like atmospheric and oceanographic science, large scale imaging, genomics, numerical algorithms for optimization or statistics , and more.
Dask arrays supports most of the Numpy interface like the following:
- Arithmetic and scalar mathematics,
+, *, exp, log, ...
- Reductions along axes,
sum(), mean(), std(), sum(axis=0), ...
- Tensor contractions / dot products / matrix multiply,
- Axis reordering / transpose,
- Fancy indexing along single axes with lists or numpy arrays,
x[:, [10, 1, 5]]
- Array protocols like
- Some linear algebra
svd, qr, solve, solve_triangular, lstsq
However Dask array does not implement the entire numpy interface. Users expecting this will be disappointed. Notably, Dask array lacks the following features:
- Much of
np.linalghas not been implemented. This has been done by a number of excellent BLAS/LAPACK implementations, and is the focus of numerous ongoing academic research projects.
- Arrays with unknown shapes do not support all operations
- Operations like
sortwhich are notoriously difficult to do in parallel, and are of somewhat diminished value on very large data (you rarely actually need a full sort). Often we include parallel-friendly alternatives like
- Dask array doesn’t implement operations like
tolistthat would be very inefficient for larger datasets. Likewise it is very inefficient to iterate over a Dask array with for loops.
- Dask development is driven by immediate need, and so many lesser used functions have not been implemented. Community contributions are encouraged.
See the dask.array API for a more extensive list of functionality.