Dask

Dask is a flexible parallel computing library for analytics. See documentation for more information. Dask allows distributed computation in Python.Dask provides advanced parallelism for analytics, enabling performance at scale for the tools you love.

Chart Details

This chart will deploy the following:

• 1 x Dask scheduler with port 8786 (scheduler) and 80 (Web UI) exposed on an external LoadBalancer
• 3 x Dask workers that connect to the scheduler
• 1 x Jupyter notebook (optional) with port 80 exposed on an external LoadBalancer
• All using Kubernetes Deployments

BUILT WITH THE BROADER COMMUNITY

Dask is open source and freely available. It is developed in coordination with other community projects like Numpy, Pandas, and Scikit-Learn

• Numpy: Dask arrays scale Numpy workflows, enabling multi-dimensional data analysis in earth science, satellite imagery, genomics, biomedical applications, and machine learning algorithms.
• Pandas: Dask dataframes scale Pandas workflows, enabling applications in time series, business intelligence, and general data munging on big data.
• Scikit-Learn: Dask-ML scales machine learning APIs like Scikit-Learn and XGBoost to enable scalable training and prediction on large models and large datasets.

EASY TO GET STARTED

Dask uses existing Python APIs and data structures to make it easy to switch between Numpy, Pandas, Scikit-learn to their Dask-powered equivalents.

You don't have to completely rewrite your code or retrain to scale up.

Scale up to clusters
OR JUST USE IT ON YOUR LAPTOP

Dask's schedulers scale to thousand-node clusters and its algorithms have been tested on some of the largest supercomputers in the world.

But you don't need a massive cluster to get started. Dask ships with schedulers designed for use on personal machines. Many people use Dask today to scale computations on their laptop, using multiple cores for computation and their disk for excess storage.

Customizable
ENABLING YOU TO PARALLELIZE INTERNAL SYSTEMS

Not all computations fit into a big data frame.

Dask exposes lower-level APIs letting you build custom systems for in-house applications. This helps open source leaders parallelize their own packages and helps business leaders scale custom business logic.