This work is supported by Continuum Analytics the XDATA Program and the Data Driven Discovery Initiative from the Moore Foundation

Summary

Dask Dataframe extends the popular Pandas library to operate on big data-sets on a distributed cluster. We show its capabilities by running through common dataframe operations on a common dataset. We break up these computations into the following sections:

1. Introduction: Pandas is intuitive and fast, but needs Dask to scale
2. Read CSV and Basic operations
   1. Read CSV
   2. Basic Aggregations and Groupbys
   3. Joins and Correlations
3. Shuffles and Time Series
4. Parquet I/O
5. Final thoughts
6. What we could have done better

Accompanying Plots

Throughout this post we accompany computational examples with profiles of exactly what task ran where on our cluster and when. These profiles are interactive Bokeh plots that include every task that every worker in our cluster runs over time. For example the following computation read_csv computation produces the following profile:

>>>df=dd.read_csv('s3://dask-data/nyc-taxi/2015/*.csv')

If you are reading this through a syndicated website like planet.python.org or through an RSS reader then these plots will not show up. You may want to visit http://matthewrocklin.com/blog/work/2017/01/12/dask-dataframes directly.

Dask.dataframe breaks up reading this data into many small tasks of different types. For example reading bytes and parsing those bytes into pandas dataframes. Each rectangle corresponds to one task. The y-axis enumerates each of the worker processes. We have 64 processes spread over 8 machines so there are 64 rows. You can hover over any rectangle to get more information about that task. You can also use the tools in the upper right to zoom around and focus on different regions in the computation. In this computation we can see that workers interleave reading bytes from S3 (light green) and parsing bytes to dataframes (dark green). The entire computation took about a minute and most of the workers were busy the entire time (little white space). Inter-worker communication is always depicted in red (which is absent in this relatively straightforward computation.)

Introduction

Pandas provides an intuitive, powerful, and fast data analysis experience on tabular data. However, because Pandas uses only one thread of execution and requires all data to be in memory at once, it doesn’t scale well to datasets much beyond the gigabyte scale. That component is missing. Generally people move to Spark DataFrames on HDFS or a proper relational database to resolve this scaling issue. Dask is a Python library for parallel and distributed computing that aims to fill this need for parallelism among the PyData projects (NumPy, Pandas, Scikit-Learn, etc.). Dask dataframes combine Dask and Pandas to deliver a faithful “big data” version of Pandas operating in parallel over a cluster.

I’ve written about this topic before. This blogpost is newer and will focus on performance and newer features like fast shuffles and the Parquet format.

CSV Data and Basic Operations

I have an eight node cluster on EC2 of m4.2xlarges (eight cores, 30GB RAM each). Dask is running on each node with one process per core.

We have the 2015 Yellow Cab NYC Taxi data as 12 CSV files on S3. We look at that data briefly with s3fs

>>>imports3fs>>>s3=S3FileSystem()>>>s3.ls('dask-data/nyc-taxi/2015/')['dask-data/nyc-taxi/2015/yellow_tripdata_2015-01.csv','dask-data/nyc-taxi/2015/yellow_tripdata_2015-02.csv','dask-data/nyc-taxi/2015/yellow_tripdata_2015-03.csv','dask-data/nyc-taxi/2015/yellow_tripdata_2015-04.csv','dask-data/nyc-taxi/2015/yellow_tripdata_2015-05.csv','dask-data/nyc-taxi/2015/yellow_tripdata_2015-06.csv','dask-data/nyc-taxi/2015/yellow_tripdata_2015-07.csv','dask-data/nyc-taxi/2015/yellow_tripdata_2015-08.csv','dask-data/nyc-taxi/2015/yellow_tripdata_2015-09.csv','dask-data/nyc-taxi/2015/yellow_tripdata_2015-10.csv','dask-data/nyc-taxi/2015/yellow_tripdata_2015-11.csv','dask-data/nyc-taxi/2015/yellow_tripdata_2015-12.csv']

This data is too large to fit into Pandas on a single computer. However, it can fit in memory if we break it up into many small pieces and load these pieces onto different computers across a cluster.

We connect a client to our Dask cluster, composed of one centralized dask-scheduler process and several dask-worker processes running on each of the machines in our cluster.

fromdask.distributedimportClientclient=Client('scheduler-address:8786')

And we load our CSV data using dask.dataframe which looks and feels just like Pandas, even though it’s actually coordinating hundreds of small Pandas dataframes. This takes about a minute to load and parse.

importdask.dataframeasdddf=dd.read_csv('s3://dask-data/nyc-taxi/2015/*.csv',parse_dates=['tpep_pickup_datetime','tpep_dropoff_datetime'],storage_options={'anon':True})df=client.persist(df)

This cuts up our 12 CSV files on S3 into a few hundred blocks of bytes, each 64MB large. On each of these 64MB blocks we then call pandas.read_csv to create a few hundred Pandas dataframes across our cluster, one for each block of bytes. Our single Dask Dataframe object, df, coordinates all of those Pandas dataframes. Because we’re just using Pandas calls it’s very easy for Dask dataframes to use all of the tricks from Pandas. For example we can use most of the keyword arguments from pd.read_csv in dd.read_csv without having to relearn anything.

This data is about 20GB on disk or 60GB in RAM. It’s not huge, but is also larger than we’d like to manage on a laptop, especially if we value interactivity. The interactive image above is a trace over time of what each of our 64 cores was doing at any given moment. By hovering your mouse over the rectangles you can see that cores switched between downloading byte ranges from S3 and parsing those bytes with pandas.read_csv.

Our dataset includes every cab ride in the city of New York in the year of 2015, including when and where it started and stopped, a breakdown of the fare, etc.

>>>df.head()

Basic Aggregations and Groupbys

As a quick exercise, we compute the length of the dataframe. When we call len(df) Dask.dataframe translates this into many len calls on each of the constituent Pandas dataframes, followed by communication of the intermediate results to one node, followed by a sum of all of the intermediate lengths.

>>>len(df)146112989

This takes around 400-500ms. You can see that a few hundred length computations happened quickly on the left, followed by some delay, then a bit of data transfer (the red bar in the plot), and a final summation call.

More complex operations like simple groupbys look similar, although sometimes with more communications. Throughout this post we’re going to do more and more complex computations and our profiles will similarly become more and more rich with information. Here we compute the average trip distance, grouped by number of passengers. We find that single and double person rides go far longer distances on average. We acheive this one big-data-groupby by performing many small Pandas groupbys and then cleverly combining their results.

>>>df.groupby(df.passenger_count).trip_distance.mean().compute()passenger_count02.279183115.541413211.81587131.62005247.48106653.06601962.97715895.45976373.30305483.866298Name:trip_distance,dtype:float64

As a more complex operation we see how well New Yorkers tip by hour of day and by day of week.

df2=df[(df.tip_amount>0)&(df.fare_amount>0)]# filter out bad rowsdf2['tip_fraction']=df2.tip_amount/df2.fare_amount# make new columndayofweek=(df2.groupby(df2.tpep_pickup_datetime.dt.dayofweek).tip_fraction.mean())hour=(df2.groupby(df2.tpep_pickup_datetime.dt.hour).tip_fraction.mean())

We see that New Yorkers are generally pretty generous, tipping around 20%-25% on average. We also notice that they become very generous at 4am, tipping an average of 38%.

This more complex operation uses more of the Dask dataframe API (which mimics the Pandas API). Pandas users should find the code above fairly familiar. We remove rows with zero fare or zero tip (not every tip gets recorded), make a new column which is the ratio of the tip amount to the fare amount, and then groupby the day of week and hour of day, computing the average tip fraction for each hour/day.

Dask evaluates this computation with thousands of small Pandas calls across the cluster (try clicking the wheel zoom icon in the upper right of the image above and zooming in). The answer comes back in about 3 seconds.

Joins and Correlations

To show off more basic functionality we’ll join this Dask dataframe against a smaller Pandas dataframe that includes names of some of the more cryptic columns. Then we’ll correlate two derived columns to determine if there is a relationship between paying Cash and the recorded tip.

>>>payments=pd.Series({1:'Credit Card',2:'Cash',3:'No Charge',4:'Dispute',5:'Unknown',6:'Voided trip'})>>>df2=df.merge(payments,left_on='payment_type',right_index=True)>>>df2.groupby(df2.payment_name).tip_amount.mean().compute()payment_nameCash0.000217CreditCard2.757708Dispute-0.011553Nocharge0.003902Unknown0.428571Name:tip_amount,dtype:float64

We see that while the average tip for a credit card transaction is $2.75, the average tip for a cash transaction is very close to zero. At first glance it seems like cash tips aren’t being reported. To investigate this a bit further lets compute the Pearson correlation between paying cash and having zero tip. Again, this code should look very familiar to Pandas users.

zero_tip=df2.tip_amount==0cash=df2.payment_name=='Cash'dd.concat([zero_tip,cash],axis=1).corr().compute()

So we see that standard operations like row filtering, column selection, groupby-aggregations, joining with a Pandas dataframe, correlations, etc. all look and feel like the Pandas interface. Additionally, we’ve seen through profile plots that most of the time is spent just running Pandas functions on our workers, so Dask.dataframe is, in most cases, adding relatively little overhead. These little functions represented by the rectangles in these plots are just pandas functions. For example the plot above has many rectangles labeled merge if you hover over them. This is just the standard pandas.merge function that we love and know to be very fast in memory.

Shuffles and Time Series

Distributed dataframe experts will know that none of the operations above require a shuffle. That is we can do most of our work with relatively little inter-node communication. However not all operations can avoid communication like this and sometimes we need to exchange most of the data between different workers.

For example if our dataset is sorted by customer ID but we want to sort it by time then we need to collect all the rows for January over to one Pandas dataframe, all the rows for February over to another, etc.. This operation is called a shuffle and is the base of computations like groupby-apply, distributed joins on columns that are not the index, etc..

You can do a lot with dask.dataframe without performing shuffles, but sometimes it’s necessary. In the following example we sort our data by pickup datetime. This will allow fast lookups, fast joins, and fast time series operations, all common cases. We do one shuffle ahead of time to make all future computations fast.

We set the index as the pickup datetime column. This takes anywhere from 25-40s and is largely network bound (60GB, some text, eight machines with eight cores each on AWS non-enhanced network). This also requires running something like 16000 tiny tasks on the cluster. It’s worth zooming in on the plot below.

>>>df=c.persist(df.set_index('tpep_pickup_datetime'))

This operation is expensive, far more expensive than it was with Pandas when all of the data was in the same memory space on the same computer. This is a good time to point out that you should only use distributed tools like Dask.datframe and Spark after tools like Pandas break down. We should only move to distributed systems when absolutely necessary. However, when it does become necessary, it’s nice knowing that Dask.dataframe can faithfully execute Pandas operations, even if some of them take a bit longer.

As a result of this shuffle our data is now nicely sorted by time, which will keep future operations close to optimal. We can see how the dataset is sorted by pickup time by quickly looking at the first entries, last entries, and entries for a particular day.

>>>df.head()# has the first entries of 2015

>>>df.tail()# has the last entries of 2015

>>>df.loc['2015-05-05'].head()# has the entries for just May 5th

Because we know exactly which Pandas dataframe holds which data we can execute row-local queries like this very quickly. The total round trip from pressing enter in the interpreter or notebook is about 40ms. For reference, 40ms is the delay between two frames in a movie running at 25 Hz. This means that it’s fast enough that human users perceive this query to be entirely fluid.

Time Series

Additionally, once we have a nice datetime index all of Pandas’ time series functionality becomes available to us.

For example we can resample by day:

>>>(df.passenger_count.resample('1d').mean().compute().plot())

We observe a strong periodic signal here. The number of passengers is reliably higher on the weekends.

We can perform a rolling aggregation in about a second:

>>>s=client.persist(df.passenger_count.rolling(10).mean())

Because Dask.dataframe inherits the Pandas index all of these operations become very fast and intuitive.

Parquet

Pandas’ standard “fast” recommended storage solution has generally been the HDF5 data format. Unfortunately the HDF5 file format is not ideal for distributed computing, so most Dask dataframe users have had to switch down to CSV historically. This is unfortunate because CSV is slow, doesn’t support partial queries (you can’t read in just one column), and also isn’t supported well by the other standard distributed Dataframe solution, Spark. This makes it hard to move data back and forth.

Fortunately there are now two decent Python readers for Parquet, a fast columnar binary store that shards nicely on distributed data stores like the Hadoop File System (HDFS, not to be confused with HDF5) and Amazon’s S3. The already fast Parquet-cpp project has been growing Python and Pandas support through Arrow, and the Fastparquet project, which is an offshoot from the pure-python parquet library has been growing speed through use of NumPy and Numba.

Using Fastparquet under the hood, Dask.dataframe users can now happily read and write to Parquet files. This increases speed, decreases storage costs, and provides a shared format that both Dask dataframes and Spark dataframes can understand, improving the ability to use both computational systems in the same workflow.

Writing our Dask dataframe to S3 can be as simple as the following:

df.to_parquet('s3://dask-data/nyc-taxi/tmp/parquet')

However there are also a variety of options we can use to store our data more compactly through compression, encodings, etc.. Expert users will probably recognize some of the terms below.

df=df.astype({'VendorID':'uint8','passenger_count':'uint8','RateCodeID':'uint8','payment_type':'uint8'})df.to_parquet('s3://dask-data/nyc-taxi/tmp/parquet',compression='snappy',has_nulls=False,object_encoding='utf8',fixed_text={'store_and_fwd_flag':1})

We can then read our nicely indexed dataframe back with the dd.read_parquet function:

>>>df2=dd.read_parquet('s3://dask-data/nyc-taxi/tmp/parquet')

The main benefit here is that we can quickly compute on single columns. The following computation runs in around 6 seconds, even though we don’t have any data in memory to start (recall that we started this blogpost with a minute-long call to read_csv.and Client.persist)

>>>df2.passenger_count.value_counts().compute()110299104522090137257939001361351076512395142981071040853723981819169Name:passenger_count,dtype:int64

Final Thoughts

With the recent addition of faster shuffles and Parquet support, Dask dataframes become significantly more attractive. This blogpost gave a few categories of common computations, along with precise profiles of their execution on a small cluster. Hopefully people find this combination of Pandas syntax and scalable computing useful.

Now would also be a good time to remind people that Dask dataframe is only one module among many within the Dask project. Dataframes are nice, certainly, but Dask’s main strength is its flexibility to move beyond just plain dataframe computations to handle even more complex problems.

Learn More

If you’d like to learn more about Dask dataframe, the Dask distributed system, or other components you should look at the following documentation:

1. http://dask.pydata.org/en/latest/
2. http://distributed.readthedocs.io/en/latest/

The workflows presented here are captured in the following notebooks (among other examples):

1. NYC Taxi example, shuffling, others
2. Parquet

What we could have done better

As always with computational posts we include a section on what went wrong, or what could have gone better.

1. The 400ms computation of len(df) is a regression from previous versions where this was closer to 100ms. We’re getting bogged down somewhere in many small inter-worker communications.
2. It would be nice to repeat this computation at a larger scale. Dask deployments in the wild are often closer to 1000 cores rather than the 64 core cluster we have here and datasets are often in the terrabyte scale rather than our 60 GB NYC Taxi dataset. Unfortunately representative large open datasets are hard to find.
3. The Parquet timings are nice, but there is still room for improvement. We seem to be making many small expensive queries of S3 when reading Thrift headers.
4. It would be nice to support both Python Parquet readers, both the Numba solution fastparquet and the C++ solution parquet-cpp

This work is supported by Continuum Analytics the XDATA Program and the Data Driven Discovery Initiative from the Moore Foundation

This page includes embedded large profiles. It may look better on the actual site TODO: link to live site (rather than through syndicated pages like planet.python) and it may take a while to load on non-broadband connections (total size is around 20MB)

Summary

We analyze a stack of images in parallel with NumPy arrays distributed across a cluster of machines on Amazon’s EC2 with Dask array. This is a model application shared among many image analysis groups ranging from satellite imagery to bio-medical applications. We go through a series of common operations:

1. Inspect a sample of images locally with Scikit Image
2. Construct a distributed Dask.array around all of our images
3. Process and re-center images with Numba
4. Transpose data to get a time-series for every pixel, compute FFTs

This last step is quite fun. Even if you skim through the rest of this article I recommend checking out the last section.

Inspect Dataset

I asked a colleague at the US National Institutes for Health (NIH) for a biggish imaging dataset. He came back with the following message:

*Electron microscopy may be generating the biggest ndarray datasets in the field - terabytes regularly. Neuroscience needs EM to see connections between neurons, because the critical features of neural synapses (connections) are below the diffraction limit of light microscopes. This type of research has been called “connectomics”. Many groups are looking at machine vision approaches to follow small neuron parts from one slice to the next. *

This data is from drosophila: http://emdata.janelia.org/. Here is an example 2d slice of the data http://emdata.janelia.org/api/node/bf1/grayscale/raw/xy/2000_2000/1800_2300_5000.

importskimage.ioimportmatplotlib.pyplotaspltsample=skimage.io.imread('http://emdata.janelia.org/api/node/bf1/grayscale/raw/xy/2000_2000/1800_2300_5000'skimage.io.imshow(sample)

The last number in the URL is an index into a large stack of about 10000 images. We can change that number to get different slices through our 3D dataset.

samples=[skimage.io.imread('http://emdata.janelia.org/api/node/bf1/grayscale/raw/xy/2000_2000/1800_2300_%d'%i)foriin[1000,2000,3000,4000,5000,6000,7000,8000,9000]]fig,axarr=plt.subplots(1,9,sharex=True,sharey=True,figsize=(24,2.5))fori,sampleinenumerate(samples):axarr[i].imshow(sample,cmap='gray')

We see that our field of interest wanders across the frame over time and drops off in the beginning and at the end.

Create a Distributed Array

Even though our data is spread across many files, we still want to think of it as a single logical 3D array. We know how to get any particular 2D slice of that array using Scikit-image. Now we’re going to use Dask.array to stitch all of those Scikit-image calls into a single distributed array.

importdask.arrayasdafromdaskimportdelayedimread=delayed(skimage.io.imread,pure=True)# Lazy version of imreadurls=['http://emdata.janelia.org/api/node/bf1/grayscale/raw/xy/2000_2000/1800_2300_%d'%iforiinrange(10000)]# A list of our URLslazy_values=[imread(url)forurlinurls]# Lazily evaluate imread on each urlarrays=[da.from_delayed(lazy_value,# Construct a small Dask arraydtype=sample.dtype,# for every lazy valueshape=sample.shape)forlazy_valueinlazy_values]stack=da.stack(arrays,axis=0)# Stack all small Dask arrays into one

>>>stackdask.array<shape=(10000,2000,2000),dtype=uint8,chunksize=(1,2000,2000)>

>>>stack=stack.rechunk((20,2000,2000))# combine chunks to reduce overhead>>>stackdask.array<shape=(10000,2000,2000),dtype=uint8,chunksize=(20,2000,2000)>

So here we’ve constructed a lazy Dask.array from 10 000 delayed calls to skimage.io.imread. We haven’t done any actual work yet, we’ve just constructed a parallel array that knows how to get any particular slice of data by downloading the right image if necessary. This gives us a full NumPy-like abstraction on top of all of these remote images. For example we can now download a particular image just by slicing our Dask array.

>>>stack[5000,:,:].compute()array([[0,0,0,...,0,0,0],[0,0,0,...,0,0,0],[0,0,0,...,0,0,0],...,[0,0,0,...,0,0,0],[0,0,0,...,0,0,0],[0,0,0,...,0,0,0]],dtype=uint8)>>>stack[5000,:,:].mean().compute()11.49902425

However we probably don’t want to operate too much further without connecting to a cluster. That way we can just download all of the images once into distributed RAM and start doing some real computations. I happen to have ten m4.2xlarges on Amazon’s EC2 (8 cores, 30GB RAM each) running Dask workers. So we’ll connect to those.

fromdask.distributedimportClient,progressclient=Client('schdeduler-address:8786')>>>client<Client:scheduler="scheduler-address:8786"processes=10cores=80>

I’ve replaced the actual address of my scheduler (something like 54.183.180.153 with `scheduler-address. Let’s go ahead and bring in all of our images, persisting the array into concrete data in memory.

stack=client.persist(stack)

This starts downloads of our 10 000 images across our 10 workers. When this completes we have 10 000 NumPy arrays spread around on our cluster, coordinated by our single logical Dask array. This takes a while, about five minutes. We’re mostly network bound here (Janelia’s servers are not co-located with our compute nodes). Here is a parallel profile of the computation as an interactive Bokeh plot.

There will be a few of these profile plots throughout the blogpost, so you might want to familiarize yoursel with them now. Every horizontal rectangle in this plot corresponds to a single Python function running somewhere in our cluster over time. Because we called skimage.io.imread 10 000 times there are 10 000 purple rectangles. Their position along the y-axis denotes which of the 80 cores in our cluster that they ran on and their position along the x-axis denotes their start and stop times. You can hover over each rectangle (function) for more information on what kind of task it was, how long it took, etc.. In the image below, purple rectangles are skimage.io.imread calls and red rectangles are data transfer between workers in our cluster. Click the magnifying glass icons in the upper right of the image to enable zooming tools.

Now that we have persisted our Dask array in memory our data is based on hundreds of concrete in-memory NumPy arrays across the cluster, rather than based on hundreds of lazy scikit-image calls. Now we can do all sorts of fun distributed array computations more quickly.

For example we can easily see our field of interest move across the frame by averaging across time:

skimage.io.imshow(stack.mean(axis=0).compute())

Or we can see when the field of interest is actually present within the frame by averaging across x and y

plt.plot(stack.mean(axis=[1,2]).compute())

By looking at the profile plots for each case we can see that averaging over time involves much more inter-node communication, which can be quite expensive in this case.

Recenter Images with Numba

In order to remove the spatial offset across time we’re going to compute a centroid for each slice and then crop the image around that center. I looked up centroids in the Scikit-Image docs and came across a function that did way more than what I was looking for, so I just quickly coded up a solution in Pure Python and then JIT-ed it with Numba (which makes this run at C-speeds).

fromnumbaimportjit@jit(nogil=True)defcentroid(im):n,m=im.shapetotal_x=0total_y=0total=0foriinrange(n):forjinrange(m):total+=im[i,j]total_x+=i*im[i,j]total_y+=j*im[i,j]iftotal>0:total_x/=totaltotal_y/=totalreturntotal_x,total_y>>>centroid(sample)# this takes around 9ms(748.7325324581344,802.4893005160851)

defrecenter(im):x,y=centroid(im.squeeze())x,y=int(x),int(y)ifx<500:x=500ify<500:y=500ifx>1500:x=1500ify>1500:y=1500returnim[...,x-500:x+500,y-500:y+500]plt.figure(figsize=(8,8))skimage.io.imshow(recenter(sample))

Now we map this function across our distributed array.

importnumpyasnpdefrecenter_block(block):""" Recenter a short stack of images """returnnp.stack([recenter(block[i])foriinrange(block.shape[0])])recentered=stack.map_blocks(recenter,chunks=(20,1000,1000),# chunk size changesdtype=a.dtype)recentered=client.persist(recentered)

This profile provides a good opportunity to talk about a scheduling failure; things went a bit wrong here. Towards the beginning we quickly recenter several images (Numba is fast), taking around 300-400ms for each block of twenty images. However as some workers finish all of their allotted tasks, the scheduler erroneously starts to load balance, moving images from busy workers to idle workers. Unfortunately the network at this time appeared to be much slower than expected and so the move + compute elsewhere strategy ended up being much slower than just letting the busy workers finish their work. The scheduler keeps track of expected compute times and transfer times precisely to avoid mistakes like this one. These sorts of issues are rare, but do occur on occasion.

We check our work by averaging our re-centered images across time and displaying that to the screen. We see that our images are better centered with each other as expected.

skimage.io.imshow(recentered.mean(axis=0))

This shows how easy it is to create fast in-memory code with Numba and then scale it out with Dask.array. The two projects complement each other nicely, giving us near-optimal performance with intuitive code across a cluster.

Rechunk to Time Series by Pixel

We’re now going to rearrange our data from being partitioned by time slice, to being partitioned by pixel. This will allow us to run computations like Fast Fourier Transforms (FFTs) on each time series efficiently. Switching the chunk pattern back and forth like this is generally a very difficult operation for distributed arrays because every slice of the array contributes to every time-series. We have N-squared communication.

This analysis may not be appropriate for this data (we won’t learn any useful science from doing this), but it represents a very frequently asked question, so I wanted to include it.

Currently our Dask array has chunkshape (20, 1000, 1000), meaning that our data is collected into 500 NumPy arrays across the cluster, each of size (20, 1000, 1000).

>>>recentereddask.array<shape=(10000,1000,1000),dtype=uint8,chunksize=(20,1000,1000)>

But we want to change this shape so that the chunks cover the entire first axis. We want all data for any particular pixel to be in the same NumPy array, not spread across hundreds of different NumPy arrays. We could solve this by rechunking so that each pixel is its own block like the following:

>>>rechunked=recentered.rechunk((10000,1,1))

However this would result in one million chunks (there are one million pixels) which will result in a bit of scheduling overhead. Instead we’ll collect our time-series into 10 x 10 groups of one hundred pixels. This will help us to reduce overhead.

>>># rechunked = recentered.rechunk((10000, 1, 1))  # Too many chunks>>>rechunked=recentered.rechunk((10000,10,10))# Use larger chunks

Now we compute the FFT of each pixel, take the absolute value and square to get the power spectrum. Finally to conserve space we’ll down-grade the dtype to float32 (our original data is only 8-bit anyway).

x=da.fft.fft(rechunked,axis=0)power=abs(x**2).astype('float32')power=client.persist(power,optimize_graph=False)

This is a fun profile to inspect; it includes both the rechunking and the subsequent FFTs. We’ve included a real-time trace during execution, the full profile, as well as some diagnostics plots from a single worker. These plots total up to around 20MB. I sincerely apologize to those without broadband access.

Here is a real time plot of the computation finishing over time:

And here is a single interactive plot of the entire computation after it completes. Zoom with the tools in the upper right. Hover over rectangles to get more information. Remember that red is communication.

Screenshots of the diagnostic dashboard of a single worker during this computation.

This computation starts with a lot of communication while we rechunk and realign our data (recent optimizations here by Antoine Pitrou in dask #417). Then we transition into doing thousands of small FFTs and other arithmetic operations. All of the plots above show a nice transition from heavy communication to heavy processing with some overlap each way (once some complex blocks are available we get to start overlapping communication and computation). Inter-worker communication was around 100-300 MB/s (typical for Amazon’s EC2) and CPU load remained high. We’re using our hardware.

Finally we can inspect the results. We see that the power spectrum is very boring in the corner, and has typical activity towards the center of the image.

plt.semilogy(1+power[:,0,0].compute())

plt.semilogy(1+power[:,500,500].compute())

Final Thoughts

This blogpost showed a non-trivial image processing workflow, emphasizing the following points:

1. Construct a Dask array from lazy SKImage calls.
2. Use NumPy syntax with Dask.array to aggregate distributed data across a cluster.
3. Build a centroid function with Numba. Use Numba and Dask together to clean up an image stack.
4. Rechunk to facilitate time-series operations. Perform FFTs.

Hopefully this example has components that look similar to what you want to do with your data on your hardware. We would love to see more applications like this out there in the wild.

What we could have done better

As always with all computationally focused blogposts we’ll include a section on what went wrong and what we could have done better with more time.

1. Communication is too expensive: Interworker communications that should be taking 200ms are taking up to 10 or 20 seconds. We need to take a closer look at our communications pipeline (which normally performs just fine on other computations) to see if something is acting up. Disucssion here dask/distributed #776 and early work here dask/distributed #810.
2. Faulty Load balancing: We discovered a case where our load-balancing heuristics misbehaved, incorrectly moving data between workers when it would have been better to let everything alone. This is likely due to the oddly low bandwidth issues observed above.
3. Loading from disk blocks network I/O: While doing this we discovered an issue where loading large amounts of data from disk can block workers from responding to network requests (dask/distributed #774)
4. Larger datasets: It would be fun to try this on a much larger dataset to see how the solutions here scale.

This work is supported by Continuum Analytics the XDATA Program and the Data Driven Discovery Initiative from the Moore Foundation

To increase transparency I’m blogging weekly about the work done on Dask and related projects during the previous week. This log covers work done between 2017-01-01 and 2016-01-17. Nothing here is ready for production. This blogpost is written in haste, so refined polish should not be expected.

Themes of the last couple of weeks:

1. Stability enhancements for the distributed scheduler and micro-release
2. NASA Grant writing
3. Dask-EC2 script
4. Dataframe categorical flexibility (work in progress)
5. Communication refactor (work in progress)

Stability enhancements and micro-release

We’ve released dask.distributed version 1.15.1, which includes important bugfixes after the recent 1.15.0 release. There were a number of small issues that coordinated to remove tasks erroneously. This was generally OK because the Dask scheduler was able to heal the missing pieces (using the same machinery that makes Dask resilience) and so we didn’t notice the flaw until the system was deployed in some of the more serious Dask deployments in the wild. PR dask/distributed #804 contains a full writeup in case anyone is interested. The writeup ends with the following line:

This was a nice exercise in how coupling mostly-working components can easily yield a faulty system.

This also adds other fixes, like a compatibility issue with the new Bokeh 0.12.4 release and others.

NASA Grant Writing

I’ve been writing a proposal to NASA to help fund distributed Dask+XArray work for atmospheric and oceanographic science at the 100TB scale. Many thanks to our scientific collaborators who are offering support here.

Dask-EC2 startup

The Dask-EC2 project deploys Anaconda, a Dask cluster, and Jupyter notebooks on Amazon’s Elastic Compute Cloud (EC2) with a small command line interface:

pip install dask-ec2 --upgrade
dask-ec2 up --keyname KEYNAME \
            --keypair /path/to/ssh-key \
            --type m4.2xlarge
            --count 8

This project can be either very useful for people just getting started and for Dask developers when we run benchmarks, or it can be horribly broken if AWS or Dask interfaces change and we don’t keep this project maintained. Thanks to a great effort from Ben Zaitlen `dask-ec2 is again in the very useful state, where I’m hoping it will stay for some time.

If you’ve always wanted to try Dask on a real cluster and if you already have AWS credentials then this is probably the easiest way.

This already seems to be paying dividends. There have been a few unrelated pull requests from new developers this week.

Dataframe Categorical Flexibility

Categoricals can significantly improve performance on text-based data. Currently Dask’s dataframes support categoricals, but they expect to know all of the categories up-front. This is easy if this set is small, like the ["Healthy", "Sick"] categories that might arise in medical research, but requires a full dataset read if the categories are not known ahead of time, like the names of all of the patients.

Jim Crist is changing this so that Dask can operates on categorical columns with unknown categories at dask/dask #1877. The constituent pandas dataframes all have possibly different categories that are merged as necessary. This distinction may seem small, but it limits performance in a surprising number of real-world use cases.

Communication Refactor

Since the recent worker refactor and optimizations it has become clear that inter-worker communication has become a dominant bottleneck in some intensive applications. Antoine Pitrou is currently refactoring Dask’s network communication layer, making room for more communication options in the future. This is an ambitious project. I for one am very happy to have someone like Antoine looking into this.

This work is supported by Continuum Analytics the XDATA Program and the Data Driven Discovery Initiative from the Moore Foundation

Summary

This post describes Dask as a computational task scheduler that fits somewhere on a spectrum between big data computing frameworks like Hadoop/Spark and task schedulers like Airflow/Celery/Luigi. We see how, by combining elements from both of these types of systems Dask is able to handle complex data science problems particularly well.

This post is in contrast to two recent posts on structured parallel collections:

1. Distributed DataFrames
2. Distributed Arrays

Big Data Collections

Most distributed computing systems like Hadoop or Spark or SQL databases implement a small but powerful set of parallel operations like map, reduce, groupby, and join. As long as you write your programs using only those operations then the platforms understand your program and serve you well. Most of the time this is great because most big data problems are pretty simple.

However, as we explore new complex algorithms or messier data science problems, these large parallel operations start to become insufficiently flexible. For example, consider the following data loading and cleaning problem:

1. Load data from 100 different files (this is a simple map operation)
2. Also load a reference dataset from a SQL database (not parallel at all, but could run alongside the map above)
3. Normalize each of the 100 datasets against the reference dataset (sort of like a map, but with another input)
4. Consider a sliding window of every three normalized datasets (Might be able to hack this with a very clever join? Not sure.)
5. Of all of the 98 outputs of the last stage, consider all pairs. (Join or cartesian product) However, because we don’t want to compute all ~10000 possibilities, let’s just evaluate a random sample of these pairs
6. Find the best of all of these possibilities (reduction)

In sequential for-loopy code this might look like the following:

filenames=['mydata-%d.dat'%iforiinrange(100)]data=[load(fn)forfninfilenames]reference=load_from_sql('sql://mytable')processed=[process(d,reference)fordindata]rolled=[]foriinrange(len(processed)-2):a=processed[i]b=processed[i+1]c=processed[i+2]r=roll(a,b,c)rolled.append(r)compared=[]foriinrange(200):a=random.choice(rolled)b=random.choice(rolled)c=compare(a,b)compared.append(c)best=reduction(compared)

This code is clearly parallelizeable, but it’s not clear how to write it down as a MapReduce program, a Spark computation, or a SQL query. These tools generally fail when asked to express complex or messy problems. We can still use Hadoop/Spark to solve this problem, but we are often forced to change and simplify our objectives a bit. (This problem is not particularly complex, and I suspect that there are clever ways to do it, but it’s not trivial and often inefficient.)

Task Schedulers

So instead people use task schedulers like Celery, Luigi, or Airflow. These systems track hundreds of tasks, each of which is just a normal Python function that runs on some normal Python data. The task scheduler tracks dependencies between tasks and so runs as many as it can at once if they don’t depend on each other.

This is a far more granular approach than the Big-Bulk-Collection approach of MapReduce and Spark. However systems like Celery, Luigi, and Airflow are also generally less efficient. This is both because they know less about their computations (map is much easier to schedule than an arbitrary graph) and because they just don’t have machinery for inter-worker communication, efficient serialization of custom datatypes, etc..

Dask Mixes Task Scheduling with Efficient Computation

Dask is both a big data system like Hadoop/Spark that is aware of resilience, inter-worker communication, live state, etc. and also a general task scheduler like Celery, Luigi, or Airflow, capable of arbitrary task execution.

Many Dask users use something like Dask dataframe, which generates these graphs automatically, and so never really observe the task scheduler aspect of Dask This is, however, the core of what distinguishes Dask from other systems like Hadoop and Spark. Dask is incredibly flexible in the kinds of algorithms it can run. This is because, at its core, it can run any graph of tasks and not just map, reduce, groupby, join, etc.. Users can do this natively, without having to subclass anything or extend Dask to get this extra power.

There are significant performance advantages to this. For example:

1. Dask.dataframe can easily represent nearest neighbor computations for fast time-series algorithms
2. Dask.array can implement complex linear algebra solvers or SVD algorithms from the latest research
3. Complex Machine Learning algorithms are often easier to implement in Dask, allowing it to be more efficient through smarter algorithms, as well as through scalable computing.
4. Complex hierarchies from bespoke data storage solutions can be explicitly modeled and loaded in to other Dask systems

This doesn’t come for free. Dask’s scheduler has to be very intelligent to smoothly schedule arbitrary graphs while still optimizing for data locality, worker failure, minimal communication, load balancing, scarce resources like GPUs and more. It’s a tough job.

Dask.delayed

So let’s go ahead and run the data ingestion job described with Dask.

We craft some fake functions to simulate actual work:

importrandomfromtimeimportsleepdefload(address):sleep(random.random()/2)defload_from_sql(address):sleep(random.random()/2+0.5)defprocess(data,reference):sleep(random.random()/2)defroll(a,b,c):sleep(random.random()/5)defcompare(a,b):sleep(random.random()/10)defreduction(seq):sleep(random.random()/1)

We annotate these functions with dask.delayed, which changes a function so that instead of running immediately it captures its inputs and puts everything into a task graph for future execution.

fromdaskimportdelayedload=delayed(load)load_from_sql=delayed(load_from_sql)process=delayed(process)roll=delayed(roll)compare=delayed(compare)reduction=delayed(reduction)

Now we just call our normal Python for-loopy code from before. However now rather than run immediately our functions capture a computational graph that can be run elsewhere.

filenames=['mydata-%d.dat'%iforiinrange(100)]data=[load(fn)forfninfilenames]reference=load_from_sql('sql://mytable')processed=[process(d,reference)fordindata]rolled=[]foriinrange(len(processed)-2):a=processed[i]b=processed[i+1]c=processed[i+2]r=roll(a,b,c)rolled.append(r)compared=[]foriinrange(200):a=random.choice(rolled)b=random.choice(rolled)c=compare(a,b)compared.append(c)best=reduction(compared)

Here is an image of that graph for a smaller input of only 10 files and 20 random pairs

We can connect to a small cluster with 20 cores

fromdask.distributedimportClientclient=Client('scheduler-address:8786')

We compute the result and see the trace of the computation running in real time.

result=best.compute()

The completed Bokeh image below is interactive. You can pan and zoom by selecting the tools in the upper right. You can see every task, which worker it ran on and how long it took by hovering over the rectangles.

We see that we use all 20 cores well. Intermediate results are transferred between workers as necessary (these are the red rectangles). We can scale this up as necessary. Dask scales to thousands of cores.

Final Thoughts

Dask’s ability to write down arbitrary computational graphs Celery/Luigi/Airflow-style and yet run them with the scalability promises of Hadoop/Spark allows for a pleasant freedom to write comfortably and yet still compute scalably. This ability opens up new possibilities both to support more sophisticated algorithms and also to handle messy situations that arise in the real world (enterprise data systems are sometimes messy) while still remaining within the bounds of “normal and supported” Dask operation.

This work is supported by Continuum Analytics the XDATA Program and the Data Driven Discovery Initiative from the Moore Foundation

To increase transparency I’m blogging weekly about the work done on Dask and related projects during the previous week. This log covers work done between 2017-01-17 and 2016-01-30. Nothing here is ready for production. This blogpost is written in haste, so refined polish should not be expected.

Themes of the last couple of weeks:

1. Micro-release of distributed scheduler
2. as_completed for asynchronous algorithms
3. Testing ZeroMQ and communication refactor
4. Persist, and Dask-GLM

Stability enhancements and micro-release

We’ve released dask.distributed version 1.15.2, which includes some important performance improvements for communicating multi-dimensional arrays, cleaner scheduler shutdown of workers for adaptive deployments, an improved as_completed iterator that can accept new futures in flight, and a few other smaller features.

The feature here that excites me the most is improved communication of multi-dimensional arrays across the network. In a recent blogpost about image processing on a cluster we noticed that communication bandwidth was far lower than expected. This led us to uncover a flaw in our compression heuristics. Dask doesn’t compress all data, instead it takes a few 10kB samples of the data, compresses them, and if that goes well, decides to compress the entire thing. Unfortunately, due to our mishandling of memoryviews we ended up taking much larger samples than 1kB when dealing with multi-dimensional arrays.

XArray release

This improvement is particularly timely because a new release of XArray (a project that wraps around Dask.array for large labeled arrays) is now available with better data ingestion support for NetCDF data on distributed clusters. This opens up distributed array computing to Dask’s first (and possibly largest) scientific user community, atmospheric scientists.

as_completed accepts futures.

In addition to arrays, dataframes, and the delayed decorator, Dask.distributed also implements the concurrent.futures interface from the standard library (except that Dask’s version parallelizes across a cluster and has a few other benefits). Part of this interface is the as_completed function, which takes in a list of futures and yields those futures in the order in which they finish. This enables the construction of fairly responsive asynchronous computations. As soon as some work finishes you can look at the result and submit more work based on the current state.

That has been around in Dask for some time.

What’s new is that you can now push more futures into as_completed

futures=client.map(my_function,sequence)ac=as_completed(futures)forfutureinac:result=future.result()# future is already finished, so this is fastifcondition:new_future=client.submit(function,*args,**kwargs)ac.add(new_future)# <<---- This is the new ability

So the as_completed iterator can keep going for quite a while with a set of futures always in flight. This relatively simple change allows for the easy expression of a broad set of asynchronous algorithms.

ZeroMQ and Communication Refactor

As part of a large effort, Antoine Pitrou has been refactoring Dask’s communication system. One sub-goal of this refactor was to allow us to explore other transport mechanisms than Tornado IOStreams in a pluggable way.

One such alternative is ZeroMQ sockets. We’ve gotten both incredibly positive and incredibly negative praise/warnings about using ZeroMQ. It’s not a great fit because Dask mostly just does point-to-point communication, so we don’t benefit from all of ZeroMQ’s patterns, which now become more of a hindrance than a benefit. However, we were interested in the performance impact of managing all of our network communication in a completely separately managed C++ thread unaffected by GIL issues.

Whether or you hate or love ZeroMQ you can now pick and choose. Antoine’s branch allows for easy swapping between transport mechanisms and opens the possibility for more in the future like intra-process communication with Queues, MPI, etc.. This doesn’t affect most users, but some Dask deployments are on supercomputers with exotic networks capable of very fast speeds. The possibility that we might tap into Infiniband someday and have the ability to manage data locally without copies (Tornado does not achieve this) is very attractive to some user communities.

After very preliminary benchmarking we’ve found that ZeroMQ offers a small speedup, but results in a lack of stability in the cluster under complex workloads (likely our fault, not ZeroMQs, but still). ZeroMQ support is strictly experimental and will likely stay that way for a long time. Readers should not get excited about this.

Perist and Dask-GLM

In collaboration with researchers at Capital One we’ve been working on a set of parallel solvers for first and second order methods, such as are commonly used in a broad class of statistical and machine learning algorithms.

One challenge in this process has been building algorithms that are simultaneously optimal for both the single machine and distributed schedulers. The distributeed scheduler requires that we think about where data is, on the client or on the cluster, where for the single machine scheudler this is less of a problem. The distrbuted scheduler appropriately has a new verb, persist, which keeps data as a Dask collection, but triggers all of the internal computation

compute(daskarray)->numpyarraypersist(daskarray)->daskarray

We have now mirrored this verb to the single machine scheduler in dask/dask #1927 and we get very nice performance on dask-glm’s algorithms in both cases now.

Working with the developers at Capital One has been very rewarding. I would like to find more machine learning groups that fit the following criteria:

1. Are focused on performance
2. Need parallel computing
3. Are committed to built good open source software
4. Are sufficiently expert in their field to understand correct algorithms

If you know such a person or such a group, either in industry or just a grad student in university, please encourage them to raise an issue at http://github.com/dask/dask/issues/new . I will likely write a larger blogpost on this topic in the near future.

This work is supported by Continuum Analytics the XDATA Program and the Data Driven Discovery Initiative from the Moore Foundation

Summary

This post describes two simple ways to use Dask to parallelize Scikit-Learn operations either on a single computer or across a cluster.

1. Use the Dask Joblib backend
2. Use the dklearn projects drop-in replacements for Pipeline, GridSearchCV, and RandomSearchCV

For the impatient, these look like the following:

### Joblibfromjoblibimportparallel_backendwithparallel_backend('dask.distributed',scheduler_host='scheduler-address:8786'):# your now-cluster-ified sklearn code here### Dask-learn pipeline and GridSearchCV drop-in replacements# from sklearn.grid_search import GridSearchCVfromdklearn.grid_searchimportGridSearchCV# from sklearn.pipeline import Pipelinefromdklearn.pipelineimportPipeline

However, neither of these techniques are perfect. These are the easiest things to try, but not always the best solutions. This blogpost focuses on low-hanging fruit.

Joblib

Scikit-Learn already parallelizes across a multi-core CPU using Joblib, a simple but powerful and mature library that provides an extensible map operation. Here is a simple example of using Joblib on its own without sklearn:

# Sequential codefromtimeimportsleepdefslowinc(x):sleep(1)# take a bit of time to simulate real workreturnx+1>>>[slowinc(i)foriinrange(10)]# this takes 10 seconds[1,2,3,4,5,6,7,8,9,10]# Parallel codefromjoblibimportParallel,delayed>>>Parallel(n_jobs=4)(delayed(slowinc)(i)foriinrange(10))# this takes 3 seconds[1,2,3,4,5,6,7,8,9,10]

Dask users will recognize the delayed function modifier. Dask stole the delayed decorator from Joblib.

Many of Scikit-learn’s parallel algorithms use Joblib internally. If we can extend Joblib to clusters then we get some added parallelism from joblib-enabled Scikit-learn functions immediately.

Distributed Joblib

Fortunately Joblib provides an interface for other parallel systems to step in and act as an execution engine. We can do this with the parallel_backend context manager to run with hundreds or thousands of cores in a nearby cluster:

importdistributed.joblibfromjoblibimportparallel_backendwithparallel_backend('dask.distributed',scheduler_host='scheduler-address:8786'):print(Parallel()(delayed(slowinc)(i)foriinlist(range(100))))

The main value for Scikit-learn users here is that Scikit-learn already uses joblib.Parallel within its code, so this trick works with the Scikit-learn code that you already have.

So we can use Joblib to parallelize normally on our multi-core processor:

estimator=GridSearchCV(n_jobs=4,...)# use joblib on local multi-core processor

or we can use Joblib together with Dask.distributed to parallelize across a multi-node cluster:

withparallel_backend('dask.distributed',scheduler_host='scheduler-address:8786'):estimator=GridSearchCV(...)# use joblib with Dask cluster

(There will be a more thorough example towards the end)

Limitations

Joblib is used throughout many algorithms in Scikit-learn, but not all. Generally any operation that accepts an n_jobs= parameter is a possible choice.

From Dask’s perspective Joblib’s interface isn’t ideal. For example it will always collect intermediate results back to the main process, rather than leaving them on the cluster until necessary. For computationally intense operations this is fine but does add some unnecessary communication overhead. Also Joblib doesn’t allow for operations more complex than a parallel map, so the range of algorithms that this can parallelize is somewhat limited.

Still though, given the wide use of Joblib-accelerated workflows (particularly within Scikit-learn) this is a simple thing to try if you have a cluster nearby with a possible large payoff.

Dask-learn Pipeline and Gridsearch

In July 2016, Jim Crist built and wrote about a small project, dask-learn. This project was a collaboration with SKLearn developers and an attempt to see which parts of Scikit-learn were trivially and usefully parallelizable. By far the most productive thing to come out of this work were Dask variants of Scikit-learn’s Pipeline, GridsearchCV, and RandomSearchCV objects that better handle nested parallelism. Jim observed significant speedups over SKLearn code by using these drop-in replacements.

So if you replace the following imports you may get both better single-threaded performance and the ability to scale out to a cluster:

# from sklearn.grid_search import GridSearchCVfromdklearn.grid_searchimportGridSearchCV# from sklearn.pipeline import Pipelinefromdklearn.pipelineimportPipeline

Here is a simple example from Jim’s more in-depth blogpost:

fromsklearn.datasetsimportmake_classificationX,y=make_classification(n_samples=10000,n_features=500,n_classes=2,n_redundant=250,random_state=42)fromsklearnimportlinear_model,decompositionfromsklearn.pipelineimportPipelinefromdklearn.pipelineimportPipelinelogistic=linear_model.LogisticRegression()pca=decomposition.PCA()pipe=Pipeline(steps=[('pca',pca),('logistic',logistic)])#Parameters of pipelines can be set using ‘__’ separated parameter names:grid=dict(pca__n_components=[50,100,150,250],logistic__C=[1e-4,1.0,10,1e4],logistic__penalty=['l1','l2'])# from sklearn.grid_search import GridSearchCVfromdklearn.grid_searchimportGridSearchCVestimator=GridSearchCV(pipe,grid)estimator.fit(X,y)

SKLearn performs this computation in around 40 seconds while the dask-learn drop-in replacements take around 10 seconds. Also, if you add the following lines to connect to a running cluster the whole thing scales out:

fromdask.distributedimportClientc=Client('scheduler-address:8786')

Here is a live Bokeh plot of the computation on a tiny eight process “cluster” running on my own laptop. I’m using processes here to highlight the costs of communication between processes (red). It’s actually about 30% faster to run this computation within the same single process.

Conclusion

This post showed a couple of simple mechanisms for scikit-learn users to accelerate their existing workflows with Dask. These aren’t particularly sophisticated, nor are they performance-optimal, but they are easy to understand and easy to try out. In a future blogpost I plan to cover more complex ways in which Dask can accelerate sophisticated machine learning workflows.

What we could have done better

As always, I include a brief section on what went wrong or what we could have done better with more time.

• See the bottom of Jim’s post for a more thorough explanation of “what we could have done better” for dask-learn’s pipeline and gridsearch
• Joblib + Dask.distributed interaction is convenient, but leaves some performance on the table. It’s not clear how Dask can help the sklearn codebase without being too invasive.
• It would have been nice to spin up an actual cluster on parallel hardware for this post. I wrote this quickly (in a few hours) so decided to skip this. If anyone wants to write a follow-on experiment I would be happy to publish it.

This work is supported by Continuum Analytics the XDATA Program and the Data Driven Discovery Initiative from the Moore Foundation

Summary

This post briefly describes potential interactions between Dask and TensorFlow and then goes through a concrete example using them together for distributed training with a moderately complex architecture.

This post was written in haste, see disclaimers below.

Introduction

Dask and TensorFlow both provide distributed computing in Python. TensorFlow excels at deep learning applications while Dask is more generic. We can combine both together in a few applications:

1. Simple data parallelism: hyper-parameter searches during training and predicting already-trained models against large datasets are both trivial to distribute with Dask as they would be trivial to distribute with any distributed computing system (Hadoop/Spark/Flink/etc..) We won’t discuss this topic much. It should be straightforward.
2. Deployment: A common pain point with TensorFlow is that setup isn’t well automated. This plagues all distributed systems, especially those that are run on a wide variety of cluster managers (see cluster deployment blogpost for more information). Fortunately, if you already have a Dask cluster running it’s trivial to stand up a distributed TensorFlow network on top of it running within the same processes.
3. Pre-processing: We pre-process data with dask.dataframe or dask.array, and then hand that data off to TensorFlow for training. If Dask and TensorFlow are co-located on the same processes then this movement is efficient. Working together we can build efficient and general use deep learning pipelines.

In this blogpost we look very briefly at the first case of simple parallelism. Then go into more depth on an experiment that uses Dask and TensorFlow in a more complex situation. We’ll find we can accomplish a fairly sophisticated workflow easily, both due to how sensible TensorFlow is to set up and how flexible Dask can be in advanced situations.

Motivation and Disclaimers

Distributed deep learning is fundamentally changing the way humanity solves some very hard computing problems like natural language translation, speech-to-text transcription, image recognition, etc.. However, distributed deep learning also suffers from public excitement, which may distort our image of its utility. Distributed deep learning is not always the correct choice for most problems. This is for two reasons:

1. Focusing on single machine computation is often a better use of time. Model design, GPU hardware, etc. can have a more dramatic impact than scaling out. For newcomers to deep learning, watching online video lecture series may be a better use of time than reading this blogpost.
2. Traditional machine learning techniques like logistic regression, and gradient boosted trees can be more effective than deep learning if you have finite data. They can also sometimes provide valuable interpretability results.

Regardless, there are some concrete take-aways, even if distributed deep learning is not relevant to your application:

1. TensorFlow is straightforward to set up from Python
2. Dask is sufficiently flexible out of the box to support complex settings and workflows
3. We’ll see an example of a typical distributed learning approach that generalizes beyond deep learning.

Additionally the author does not claim expertise in deep learning and wrote this blogpost in haste.

Simple Parallelism

Most parallel computing is simple. We easily apply one function to lots of data, perhaps with slight variation. In the case of deep learning this can enable a couple of common workflows:

1. Build many different models, train each on the same data, choose the best performing one. Using dask’s concurrent.futures interface, this looks something like the following:

   # Hyperparameter searchclient=Client('dask-scheduler-address:8786')scores=client.map(train_and_evaluate,hyper_param_list,data=data)best=client.submit(max,scores)best.result()

2. Given an already-trained model, use it to predict outcomes on lots of data. Here we use a big data collection like dask.dataframe:

   # Distributed predictiondf=dd.read_parquet('...')...# do some preprocessing heredf['outcome']=df.map_partitions(predict)

These techniques are relatively straightforward if you have modest exposure to Dask and TensorFlow (or any other machine learning library like scikit-learn), so I’m going to ignore them for now and focus on more complex situations.

Interested readers may find this blogpost on TensorFlow and Spark of interest. It is a nice writeup that goes over these two techniques in more detail.

A Distributed TensorFlow Application

We’re going to replicate this TensorFlow example which uses multiple machines to train a model that fits in memory using parameter servers for coordination. Our TensorFlow network will have three different kinds of servers:

1. Workers: which will get updated parameters, consume training data, and use that data to generate updates to send back to the parameter servers
2. Parameter Servers: which will hold onto model parameters, synchronizing with the workers as necessary
3. Scorer: which will periodically test the current parameters against validation/test data and emit a current cross_entropy score to see how well the system is running.

This is a fairly typical approach when the model can fit in one machine, but when we want to use multiple machines to accelerate training or because data volumes are too large.

We’ll use TensorFlow to do all of the actual training and scoring. We’ll use Dask to do everything else. In particular, we’re about to do the following:

1. Prepare data with dask.array
2. Set up TensorFlow workers as long-running tasks
3. Feed data from Dask to TensorFlow while scores remain poor
4. Let TensorFlow handle training using its own network

Prepare Data with Dask.array

For this toy example we’re just going to use the mnist data that comes with TensorFlow. However, we’ll artificially inflate this data by concatenating it to itself many times across a cluster:

defget_mnist():fromtensorflow.examples.tutorials.mnistimportinput_datamnist=input_data.read_data_sets('/tmp/mnist-data',one_hot=True)returnmnist.train.images,mnist.train.labelsimportdask.arrayasdafromdaskimportdelayeddatasets=[delayed(get_mnist)()foriinrange(20)]# 20 versions of same datasetimages=[d[0]fordindatasets]labels=[d[1]fordindatasets]images=[da.from_delayed(im,shape=(55000,784),dtype='float32')foriminimages]labels=[da.from_delayed(la,shape=(55000,10),dtype='float32')forlainlabels]images=da.concatenate(images,axis=0)labels=da.concatenate(labels,axis=0)>>>imagesdask.array<concate...,shape=(1100000,784),dtype=float32,chunksize=(55000,784)>images,labels=c.persist([images,labels])# persist data in memory

This gives us a moderately large distributed array of around a million tiny images. If we wanted to we could inspect or clean up this data using normal dask.array constructs:

im=images[1].compute().reshape((28,28))plt.imshow(im,cmap='gray')

im=images.mean(axis=0).compute().reshape((28,28))plt.imshow(im,cmap='gray')

im=images.var(axis=0).compute().reshape((28,28))plt.imshow(im,cmap='gray')

This shows off how one can use Dask collections to clean up and provide pre-processing and feature generation on data in parallel before sending it to TensorFlow. In our simple case we won’t actually do any of this, but it’s useful in more real-world situations.

Finally, after doing our preprocessing on the distributed array of all of our data we’re going to collect images and labels together and batch them into smaller chunks. Again we use some dask.array constructs and dask.delayed when things get messy.

images=images.rechunk((10000,784))labels=labels.rechunk((10000,10))images=images.to_delayed().flatten().tolist()labels=labels.to_delayed().flatten().tolist()batches=[delayed([im,la])forim,lainzip(images,labels)]batches=c.compute(batches)

Now we have a few hundred pairs of NumPy arrays in distributed memory waiting to be sent to a TensorFlow worker.

Setting up TensorFlow workers alongside Dask workers

Dask workers are just normal Python processes. TensorFlow can launch itself from a normal Python process. We’ve made a small function here that launches TensorFlow servers alongside Dask workers using Dask’s ability to run long-running tasks and maintain user-defined state. All together, this is about 80 lines of code (including comments and docstrings) and allows us to define our TensorFlow network on top of Dask as follows:

$ pip install git+https://github.com/mrocklin/dask-tensorflow

fromdask.distibutedimportClient# we already had this aboveclient=Client('dask-scheduler-address:8786')fromdask_tensorflowimportstart_tensorflowtf_spec,dask_spec=start_tensorflow(client,ps=1,worker=4,scorer=1)>>>tf_spec.as_dict(){'ps':['192.168.100.1:2227'],'scorer':['192.168.100.2:2222'],'worker':['192.168.100.3:2223','192.168.100.4:2224','192.168.100.5:2225','192.168.100.6:2226']}>>>dask_spec{'ps':['tcp://192.168.100.1:34471'],'scorer':['tcp://192.168.100.2:40623'],'worker':['tcp://192.168.100.3:33075','tcp://192.168.100.4:37123','tcp://192.168.100.5:32839','tcp://192.168.100.6:36822']}

This starts three groups of TensorFlow servers in the Dask worker processes. TensorFlow will manage its own communication but co-exist right alongside Dask in the same machines and in the same shared memory spaces (note that in the specs above the IP addresses match but the ports differ).

This also sets up a normal Python queue along which Dask can safely send information to TensorFlow. This is how we’ll send those batches of training data between the two services.

Define TensorFlow Model and Distribute Roles

Now is the part of the blogpost where my expertise wanes. I’m just going to copy-paste-and-modify a canned example from the TensorFlow documentation. This is a simplistic model for this problem and it’s entirely possible that I’m making transcription errors. But still, it should get the point across. You can safely ignore most of this code. Dask stuff gets interesting again towards the bottom:

importmathimporttempfileimporttimefromqueueimportEmptyIMAGE_PIXELS=28hidden_units=100learning_rate=0.01sync_replicas=Falsereplicas_to_aggregate=len(dask_spec['worker'])defmodel(server):worker_device="/job:%s/task:%d"%(server.server_def.job_name,server.server_def.task_index)task_index=server.server_def.task_indexis_chief=task_index==0withtf.device(tf.train.replica_device_setter(worker_device=worker_device,ps_device="/job:ps/cpu:0",cluster=tf_spec)):global_step=tf.Variable(0,name="global_step",trainable=False)# Variables of the hidden layerhid_w=tf.Variable(tf.truncated_normal([IMAGE_PIXELS*IMAGE_PIXELS,hidden_units],stddev=1.0/IMAGE_PIXELS),name="hid_w")hid_b=tf.Variable(tf.zeros([hidden_units]),name="hid_b")# Variables of the softmax layersm_w=tf.Variable(tf.truncated_normal([hidden_units,10],stddev=1.0/math.sqrt(hidden_units)),name="sm_w")sm_b=tf.Variable(tf.zeros([10]),name="sm_b")# Ops: located on the worker specified with task_indexx=tf.placeholder(tf.float32,[None,IMAGE_PIXELS*IMAGE_PIXELS])y_=tf.placeholder(tf.float32,[None,10])hid_lin=tf.nn.xw_plus_b(x,hid_w,hid_b)hid=tf.nn.relu(hid_lin)y=tf.nn.softmax(tf.nn.xw_plus_b(hid,sm_w,sm_b))cross_entropy=-tf.reduce_sum(y_*tf.log(tf.clip_by_value(y,1e-10,1.0)))opt=tf.train.AdamOptimizer(learning_rate)ifsync_replicas:ifreplicas_to_aggregateisNone:replicas_to_aggregate=num_workerselse:replicas_to_aggregate=replicas_to_aggregateopt=tf.train.SyncReplicasOptimizer(opt,replicas_to_aggregate=replicas_to_aggregate,total_num_replicas=num_workers,name="mnist_sync_replicas")train_step=opt.minimize(cross_entropy,global_step=global_step)ifsync_replicas:local_init_op=opt.local_step_init_opifis_chief:local_init_op=opt.chief_init_opready_for_local_init_op=opt.ready_for_local_init_op# Initial token and chief queue runners required by the sync_replicas modechief_queue_runner=opt.get_chief_queue_runner()sync_init_op=opt.get_init_tokens_op()init_op=tf.global_variables_initializer()train_dir=tempfile.mkdtemp()ifsync_replicas:sv=tf.train.Supervisor(is_chief=is_chief,logdir=train_dir,init_op=init_op,local_init_op=local_init_op,ready_for_local_init_op=ready_for_local_init_op,recovery_wait_secs=1,global_step=global_step)else:sv=tf.train.Supervisor(is_chief=is_chief,logdir=train_dir,init_op=init_op,recovery_wait_secs=1,global_step=global_step)sess_config=tf.ConfigProto(allow_soft_placement=True,log_device_placement=False,device_filters=["/job:ps","/job:worker/task:%d"%task_index])# The chief worker (task_index==0) session will prepare the session,# while the remaining workers will wait for the preparation to complete.ifis_chief:print("Worker %d: Initializing session..."%task_index)else:print("Worker %d: Waiting for session to be initialized..."%task_index)sess=sv.prepare_or_wait_for_session(server.target,config=sess_config)ifsync_replicasandis_chief:# Chief worker will start the chief queue runner and call the init op.sess.run(sync_init_op)sv.start_queue_runners(sess,[chief_queue_runner])returnsess,x,y_,train_step,global_step,cross_entropydefps_task():withlocal_client()asc:c.worker.tensorflow_server.join()defscoring_task():withlocal_client()asc:# Scores Channelscores=c.channel('scores',maxlen=10)# Make Modelserver=c.worker.tensorflow_serversess,_,_,_,_,cross_entropy=model(c.worker.tensorflow_server)# Testing Datafromtensorflow.examples.tutorials.mnistimportinput_datamnist=input_data.read_data_sets('/tmp/mnist-data',one_hot=True)test_data={x:mnist.validation.images,y_:mnist.validation.labels}# Main LoopwhileTrue:score=sess.run(cross_entropy,feed_dict=test_data)scores.append(float(score))time.sleep(1)defworker_task():withlocal_client()asc:scores=c.channel('scores')num_workers=replicas_to_aggregate=len(dask_spec['worker'])server=c.worker.tensorflow_serverqueue=c.worker.tensorflow_queue# Make modelsess,x,y_,train_step,global_step,_=model(c.worker.tensorflow_server)# Main loopwhilenotscoresorscores.data[-1]>1000:try:batch=queue.get(timeout=0.5)exceptEmpty:continuetrain_data={x:batch[0],y_:batch[1]}sess.run([train_step,global_step],feed_dict=train_data)

The last three functions defined here, ps_task, scorer_task and worker_task are functions that we want to run on each of our three groups of TensorFlow server types. The parameter server task just starts a long-running task and passively joins the TensorFlow network:

defps_task():withlocal_client()asc:c.worker.tensorflow_server.join()

The scorer task opens up an inter-worker channel of communication named “scores”, creates the TensorFlow model, then every second scores the current state of the model against validation data. It reports the score on the inter-worker channel:

defscoring_task():withlocal_client()asc:scores=c.channel('scores')#  inter-worker channel# Make Modelsess,_,_,_,_,cross_entropy=model(c.worker.tensorflow_server)...whileTrue:score=sess.run(cross_entropy,feed_dict=test_data)scores.append(float(score))time.sleep(1)

The worker task makes the model, listens on the Dask-TensorFlow Queue for new training data, and continues training until the last reported score is good enough.

defworker_task():withlocal_client()asc:scores=c.channel('scores')queue=c.worker.tensorflow_queue# Make modelsess,x,y_,train_step,global_step,_=model(c.worker.tensorflow_server)whilescores.data[-1]>1000:batch=queue.get()train_data={x:batch[0],y_:batch[1]}sess.run([train_step,global_step],feed_dict=train_data)

We launch these tasks on the Dask workers that have the corresponding TensorFlow servers (see tf_spec and dask_spec above):

ps_tasks=[c.submit(ps_task,workers=worker)forworkerindask_spec['ps']]worker_tasks=[c.submit(worker_task,workers=addr,pure=False)foraddrindask_spec['worker']]scorer_task=c.submit(scoring_task,workers=dask_spec['scorer'][0])

This starts long-running tasks that just sit there, waiting for external stimulation:

Finally we construct a function to dump each of our batches of data from our Dask.array (from the very beginning of this post) into the Dask-TensorFlow queues on our workers. We make sure to only run these tasks where the Dask-worker has a corresponding TensorFlow training worker:

fromdistributed.worker_clientimportget_workerdeftransfer_dask_to_tensorflow(batch):worker=get_worker()worker.tensorflow_queue.put(batch)dump=c.map(transfer_dask_to_tensorflow,batches,workers=dask_spec['worker'],pure=False)

If we want to we can track progress in our local session by subscribing to the same inter-worker channel:

scores=c.channel('scores')

We can use this to repeatedly dump data into the workers over and over again until they converge.

whilescores.data[-1]>1000:dump=c.map(transfer_dask_to_tensorflow,batches,workers=dask_spec['worker'],pure=False)wait(dump)

Conclusion

We discussed a non-trivial way to use TensorFlow to accomplish distributed machine learning. We used Dask to support TensorFlow in a few ways:

1. Trivially setup the TensorFlow network
2. Prepare and clean data
3. Coordinate progress and stopping criteria

We found it convenient that Dask and TensorFlow could play nicely with each other. Dask supported TensorFlow without getting in the way. The fact that both libraries play nicely within Python and the greater PyData stack (NumPy/Pandas) makes it trivial to move data between them without costly or complex tricks.

Additionally, we didn’t have to work to integrate these two systems. There is no need for a separate collaborative effort to integrate Dask and TensorFlow at a core level. Instead, they are designed in such a way so as to foster this type of interaction without special attention or effort.

This is also the first blogpost that I’ve written that, from a Dask perspective, uses some more complex features like long running tasks or publishing state between workers with channels. These more advanced features are invaluable when creating more complex/bespoke parallel computing systems, such as are often found within companies.

What we could have done better

From a deep learning perspective this example is both elementary and incomplete. It would have been nice to train on a dataset that was larger and more complex than MNIST. Also it would be nice to see the effects of training over time and the performance of using different numbers of workers. In defense of this blogpost I can only claim that Dask shouldn’t affect any of these scaling results, because TensorFlow is entirely in control at these stages and TensorFlow already has plenty of published scaling information.

Generally speaking though, this experiment was done in a weekend afternoon and the blogpost was written in a few hours shortly afterwards. If anyone is interested in performing and publishing about a more serious distributed deep learning experiment with TensorFlow and Dask I would be happy to support them on the Dask side. I think that there is plenty to learn here about best practices.

Acknowledgements

The following individuals contributed to the construction of this blogpost:

• Stephan Hoyer contributed with conversations about how TensorFlow is used in practice and with concrete experience on deployment.
• Will Warner and Erik Welch both provided valuable editing and language recommendations

This work is supported by Continuum Analytics the XDATA Program and the Data Driven Discovery Initiative from the Moore Foundation

To increase transparency I’m blogging weekly(ish) about the work done on Dask and related projects during the previous week. This log covers work done between 2017-02-01 and 2017-02-20. Nothing here is ready for production. This blogpost is written in haste, so refined polish should not be expected.

Themes of the last couple of weeks:

1. Profiling experiments with Dask-GLM
2. Subsequent graph optimizations, both non-linear fusion and avoiding repeatedly creating new graphs
3. Tensorflow and Keras experiments
4. XGBoost experiments
5. Dask tutorial refactor
6. Google Cloud Storage support
7. Cleanup of Dask + SKLearn project

Dask-GLM and iterative algorithms

Dask-GLM is currently just a bunch of solvers like Newton, Gradient Descent, BFGS, Proximal Gradient Descent, and ADMM. These are useful in solving problems like logistic regression, but also several others. The mathematical side of this work is mostly done by Chris White and Hussain Sultan at Capital One.

We’ve been using this project also to see how Dask can scale out machine learning algorithms. To this end we ran a few benchmarks here: https://github.com/dask/dask-glm/issues/26 . This just generates and solves some random problems, but at larger scales.

What we found is that some algorithms, like ADMM perform beautifully, while for others, like gradient descent, scheduler overhead can become a substantial bottleneck at scale. This is mostly just because the actual in-memory NumPy operations are so fast; any sluggishness on Dask’s part becomes very apparent. Here is a profile of gradient descent:

Notice all the white space. This is Dask figuring out what to do during different iterations. We’re now working to bring this down to make all of the colored parts of this graph squeeze together better. This will result in general overhead improvements throughout the project.

Graph Optimizations - Aggressive Fusion

We’re approaching this in two ways:

1. More aggressively fuse tasks together so that there are fewer blocks for the scheduler to think about
2. Avoid repeated work when generating very similar graphs

In the first case, Dask already does standard task fusion. For example, if you have the following to tasks:

x=f(w)y=g(x)z=h(y)

Dask (along with every other compiler-like project since the 1980’s) already turns this into the following:

z=h(g(f(w)))

What’s tricky with a lot of these mathematical or optimization algorithms though is that they are mostly, but not entirely linear. Consider the following example:

y=exp(x)-1/x

Visualized as a node-link diagram, this graph looks like a diamond like the following:

         o  exp(x) - 1/x
        / \
exp(x) o   o   1/x
        \ /
         o  x

Graphs like this generally don’t get fused together because we could compute both exp(x) and 1/x in parallel. However when we’re bound by scheduling overhead and when we have plenty of parallel work to do, we’d prefer to fuse these into a single task, even though we lose some potential parallelism. There is a tradeoff here and we’d like to be able to exchange some parallelism (of which we have a lot) for less overhead.

PR here dask/dask #1979 by Erik Welch (Erik has written and maintained most of Dask’s graph optimizations).

Graph Optimizations - Structural Sharing

Additionally, we no longer make copies of graphs in dask.array. Every collection like a dask.array or dask.dataframe holds onto a Python dictionary holding all of the tasks that are needed to construct that array. When we perform an operation on a dask.array we get a new dask.array with a new dictionary pointing to a new graph. The new graph generally has all of the tasks of the old graph, plus a few more. As a result, we frequently make copies of the underlying task graph.

y=(x+1)assertset(y.dask).issuperset(x.dask)

Normally this doesn’t matter (copying graphs is usually cheap) but it can become very expensive for large arrays when you’re doing many mathematical operations.

Now we keep dask graphs in a custom mapping (dict-like object) that shares subgraphs with other arrays. As a result, we rarely make unnecessary copies and some algorithms incur far less overhead. Work done in dask/dask #1985.

TensorFlow and Keras experiments

Two weeks ago I gave a talk with Stan Seibert (Numba developer) on Deep Learning (Stan’s bit) and Dask (my bit). As part of that talk I decided to launch tensorflow from Dask and feed Tensorflow from a distributed Dask array. See this blogpost for more information.

That experiment was nice in that it showed how easy it is to deploy and interact with other distributed servies from Dask. However from a deep learning perspective it was immature. Fortunately, it succeeded in attracting the attention of other potential developers (the true goal of all blogposts) and now Brett Naul is using Dask to manage his GPU workloads with Keras. Brett contributed code to help Dask move around Keras models. He seems to particularly value Dask’s ability to manage resources to help him fully saturate the GPUs on his workstation.

XGBoost experiments

After deploying Tensorflow we asked what would it take to do the same for XGBoost, another very popular (though very different) machine learning library. The conversation for that is here: dmlc/xgboost #2032 with prototype code here mrocklin/dask-xgboost. As with TensorFlow, the integration is relatively straightforward (if perhaps a bit simpler in this case). The challenge for me is that I have little concrete experience with the applications that these libraries were designed to solve. Feedback and collaboration from open source developers who use these libraries in production is welcome.

Dask tutorial refactor

The dask/dask-tutorial project on Github was originally written or PyData Seattle in July 2015 (roughly 19 months ago). Dask has evolved substantially since then but this is still our only educational material. Fortunately Martin Durant is doing a pretty serious rewrite, both correcting parts that are no longer modern API, and also adding in new material around distributed computing and debugging.

Google Cloud Storage

Dask developers (mostly Martin) maintain libraries to help Python users connect to distributed file systems like HDFS (with hdfs3, S3 (with s3fs, and Azure Data Lake (with adlfs), which subsequently become usable from Dask. Martin has been working on support for Google Cloud Storage (with gcsfs) with another small project that uses the same API.

Cleanup of Dask+SKLearn project

Last year Jim Crist published threegreatblogposts about using Dask with SKLearn. The result was a small library dask-learn that had a variety of features, some incredibly useful, like a cluster-ready Pipeline and GridSearchCV, other less so. Because of the experimental nature of this work we had labeled the library “not ready for use”, which drew some curious responses from potential users.

Jim is now busy dusting off the project, removing less-useful parts and generally reducing scope to strictly model-parallel algorithms.

This work is supported by Continuum Analytics the XDATA Program and the Data Driven Discovery Initiative from the Moore Foundation

Summary

Dask just released version 0.14.0. This release contains some significant internal changes as well as the usual set of increased API coverage and bug fixes. This blogpost outlines some of the major changes since the last release January, 27th 2017.

1. Structural sharing of graphs between collections
2. Refactor communications system
3. Many small dataframe improvements
4. Top-level persist function

You can install new versions using Conda or Pip

conda install -c conda-forge dask distributed

or

pip install dask[complete] distributed --upgrade

Share Graphs between Collections

Dask collections (arrays, bags, dataframes, delayed) hold onto task graphs that have all of the tasks necessary to create the desired result. For larger datasets or complex calculations these graphs may have thousands, or sometimes even millions of tasks. In some cases the overhead of handling these graphs can become significant.

This is especially true because dask collections don’t modify their graphs in place, they make new graphs with updated computations. Copying graph data structures with millions of nodes can take seconds and interrupt interactive workflows.

To address this dask.arrays and dask.delayed collections now use special graph data structures with structural sharing. This significantly cuts down on the amount of overhead when building repetitive computations.

importdask.arrayasdax=da.ones(1000000,chunks=(1000,))# 1000 chunks of size 1000

Version 0.13.0

%timeforiinrange(100):x=x+1CPUtimes:user2.69s,sys:96ms,total:2.78sWalltime:2.78s

Version 0.14.0

%timeforiinrange(100):x=x+1CPUtimes:user756ms,sys:8ms,total:764msWalltime:763ms

The difference in this toy problem is moderate but for real world cases this can difference can grow fairly large. This was also one of the blockers identified by the climate science community stopping them from handling petabyte scale analyses.

We chose to roll this out for arrays and delayed first just because those are the two collections that typically produce large task graphs. Dataframes and bags remain as before for the time being.

Communications System

Dask communicates over TCP sockets. It uses Tornado’s IOStreams to handle non-blocking communication, framing, etc.. We’ve run into some performance issues with Tornado when moving large amounts of data. Some of this has been improved upstream in Tornado directly, but we still want the ability to optionally drop Tornado’s byte-handling communication stack in the future. This is especially important as dask gets used in institutions with faster and more exotic interconnects (supercomputers). We’ve been asked a few times to support other transport mechanisms like MPI.

The first step (and probably hardest step) was to make Dask’s communication system is pluggable so that we can use different communication options without significant source-code changes. We managed this a month ago and now it is possible to add other transports to Dask relatively easily. TCP remains the only real choice today though there is also an experimental ZeroMQ option (which provides little-to-no performance benefit over TCP) as well as a fully in-memory option in development.

For users the main difference you’ll see is that tcp:// is now prepended many places. For example:

$ dask-scheduler
distributed.scheduler - INFO - -----------------------------------------------
distributed.scheduler - INFO -   Scheduler at:  tcp://192.168.1.115:8786
...

Variety of Dataframe Changes

As usual the Pandas API has been more fully covered by community contributors. Some representative changes include the following:

1. Support non-uniform categoricals: We no longer need to do a full pass through the data when categorizing a column. Instead we categorize each partition independently (even if they have different category values) and then unify these categories only when necessary

   df['x']=df['x'].astype('category')# this is now fast

2. Groupby cumulative reductions

   df.groupby('x').cumsum()

3. Support appending to Parquet collections

   df.to_parquet('/path/to/foo.parquet',append=True)

4. A new string and HTML representation of dask.dataframes. Typically Pandas prints dataframes on the screen by rendering the first few rows of data. However, Because Dask.dataframes are lazy we don’t have this data and so typically render some metadata about the dataframe

   >>>df# version 0.13.0dd.DataFrame<make-ti...,npartitions=366,divisions=(Timestamp('2000-01-0100:00:00',freq='D'),Timestamp('2000-01-02 00:00:00',freq='D'),Timestamp('2000-01-03 00:00:00',freq='D'),...,Timestamp('2000-12-3100:00:00',freq='D'),Timestamp('2001-01-01 00:00:00',freq='D'))>

   This rendering, while informative, can be improved. Now we render dataframes as a Pandas dataframe, but place metadata in the dataframe instead of the actual data.

   >>>df# version 0.14.0DaskDataFrameStructure:xyznpartitions=3662000-01-01float64float64int642000-01-02.....................2000-12-31.........2001-01-01.........DaskName:make-timeseries,366tasks

   Additionally this renders nicely as an HTML table in a Jupyter notebook

Variety of Distributed System Changes

There have also been a wide variety of changes to the distributed system. I’ll include a representative sample here to give a flavor of what has been happening:

1. Ensure first-come-first-served priorities when dealing with multiple clients

2. Send small amounts of data through Channels. Channels are a way for multiple clients/users connected to the same scheduler to publish and exchange data between themselves. Previously they only transmitted Futures (which could in trun point to larger data living on the cluster). However we found that it was useful to communicate small bits of metadata as well, for example to signal progress or stopping critera between clients collaborating on the same workloads. Now you can publish any msgpack serializable data on Channels.

   # Publishing Clientscores=client.channel('scores')scores.append(123.456)# Subscribing Clientscores=client.channel('scores')whilescores.data[-1]<THRESHOLD:...continueworking...

3. We’re better at estimating the size in data of SciPy Sparse matrices and Keras models. This allows Dask to make smarter choices about when it should and should not move data around for load balancing. Additionally Dask can now also serialize Keras models.

4. To help people deploying on clusters that have a shared network file system (as is often the case in scientific or academic institutions) the scheduler and workers can now communicate connection information using the --scheduler-file keyword

   dask-scheduler --scheduler-file /path/to/scheduler.json
   dask-worker --scheduler-file /path/to/scheduler.json
   dask-worker --scheduler-file /path/to/scheduler.json
   
   >>> client = Client(scheduler_file='/path/to/scheudler.json')
   

   Previously we needed to communicate the address of the scheduler, which could be challenging when we didn’t know on which node the scheduler would be run.

Other

There are a number of smaller details not mentioned in this blogpost. For more information visit the changelogs and documentation

• Dask/dask docs
• Dask/dask changelog
• Dask/distributed docs
• Dask/distributed changelog

Additionally a great deal of Dask work over the last month has happened outside of these core dask repositories.

You can install or upgrade using Conda or Pip

conda install -c conda-forge dask distributed

or

pip install dask[complete] distributed --upgrade

Acknowledgements

Since the last 0.13.0 release on January 27th the following developers have contributed to the dask/dask repository:

• Antoine Pitrou
• Chris Barber
• Daniel Davis
• Elmar Ritsch
• Erik Welch
• jakirkham
• Jim Crist
• John Crickett
• jspreston
• Juan Luis Cano Rodríguez
• kayibal
• Kevin Ernst
• Markus Gonser
• Matthew Rocklin
• Martin Durant
• Nir
• Sinhrks
• Talmaj Marinc
• Vlad Frolov
• Will Warner

And the following developers have contributed to the dask/distributed repository:

• Antoine Pitrou
• Ben Schreck
• bmaisonn
• Brett Naul
• Demian Wassermann
• Israel Saeta Pérez
• John Crickett
• Joseph Crail
• Malte Gerken
• Martin Durant
• Matthew Rocklin
• Min RK
• strets123

This work is supported by Continuum Analytics the XDATA Program and the Data Driven Discovery Initiative from the Moore Foundation

Summary

Performing benchmarks to compare software is surprisingly difficult to do fairly, even assuming the best intentions of the author. Technical developers can fall victim to a few natural human failings:

1. We judge other projects by our own objectives rather than the objectives under which that project was developed
2. We fail to use other projects with the same expertise that we have for our own
3. We naturally gravitate towards cases at which our project excels
4. We improve our software during the benchmarking process
5. We don’t release negative results

We discuss each of these failings in the context of current benchmarks I’m working on comparing Dask and Spark Dataframes.

Introduction

Last week I started comparing performance between Dask Dataframes (a project that I maintain) and Spark Dataframes (the current standard). My initial results showed that Dask.dataframes were overall much faster, somewhere like 5x.

These results were wrong. They’re weren’t wrong in a factual sense, the experiments that I ran were clear and reproducible, but there was so much bias in how I selected, set up, and ran those experiments that the end result was misleading. After checking results myself and then having other experts come in and check my results I now see much more sensible numbers. At the moment both projects within a factor of two most of the time, with some interesting exceptions either way.

This blogpost outlines the ways in which library authors can fool themselves when performing benchmarks, using my recent experience as an anecdote. I hope that this encourages authors to check themselves, and encourages readers to be more critical of numbers that they see in the future.

This problem exists as well in academic research. For a pop-science rendition I recommend “The Experiment Experiment” on the Planet Money Podcast.

Skewed Objectives

Feature X is so important. I wonder how the competition fares?

Every project is developed with different applications in mind and so has different strengths and weaknesses. If we approach a benchmark caring only about our original objectives and dismissing the objectives of the other projects then we’ll very likely trick ourselves.

For example consider reading CSV files. Dask’s CSV reader is based off of Pandas’ CSV reader, which was the target of a lot of effort and love; this is because CSV was so important to the finance community where Pandas grew up. Spark’s CSV solution is less awesome, but that’s less about the quality of Spark and more a statement to the fact that Spark users tend not to use CSV. If they’re using text-based formats they’re much more likely to use line-delimited JSON, which is typical in Spark’s common use cases (web diagnostics, click logs, etc..) Pandas/Dask came from the scientific and finance worlds where CSV is king while Spark came from the web world where JSON reigns.

Conversely, Dask.dataframe hasn’t bothered to hook up the pandas.read_json function to Dask.dataframe yet. Surprisingly it never comes up. Both projects can correctly say that the other project’s solution to what they consider the standard text-based file format is less-than-awesome. Comparing performance here either way will likely lead to misguided conclusions.

So when benchmarking data ingestion maybe we look around a bit, see that both claim to support Parquet well, and use that as the basis for comparison.

Skewed Experience

Whoa, this other project has a lot of configuration parameters! Lets just use the defaults

Software is often easy to set up, but requires experience set up optimally. Authors are naturally more adept at setting up their own software than the software of their competition.

My original (and flawed) solution to this was to “just use the defaults” on both projects. Given my inability to tune Spark (there are several dozen parameters) I decided to also not tune Dask and run under default settings. I figured that this would be a good benchmark not only of the software, but on choices for sane defaults, which is a good design principle in itself.

However this failed spectacularly because I was making unconscious decisions like the size of machines that I was using for the experiment, CPU/memory ratios, etc.. It turns out that Spark’s defaults are optimized for very small machines (or more likely, small YARN containers) and use only 1GB of memory per executor by default while Dask is typically run on larger boxes or has the full use of a single machine in a single shared-memory process. My standard cluster configurations were biased towards Dask before I even considered running a benchmark.

Similarly the APIs of software projects are complex and for any given problem there is often both a fast way and a general-but-slow way. Authors naturally choose the fast way on their own system but inadvertently choose the general way that comes up first when reading documentation for the other project. It often takes months of hands-on experience to understand a project well enough to definitely say that you’re not doing things in a dumb way.

In both cases here I think the only solution is to collaborate with someone that primarily uses the other system.

Preference towards strengths

Oh hey, we’re doing really well here. This is great! Lets dive into this a bit more.

It feels great to see your project doing well. This emotional pleasure response is powerful. It’s only natural that we pursue that feeling more, exploring different aspects of it. This can skew our writing as well. We’ll find that we’ve decided to devote 80% of the text to what originally seemed like a small set of features, but which now seems like the main point.

It’s important that we define a set of things we’re going to study ahead of time and then stick to those things. When we run into cases where our project fails we should take that as an opportunity to raise an issue for future (though not current) development.

Tuning during experimentation

Oh, I know why this is slow. One sec, let me change something in the code.

I’m doing this right now. Dask dataframe shuffles are generally slower than Spark dataframe shuffles. On numeric data this used to be around a 2x difference, now it’s more like a 1.2x difference (at least on my current problem and machine). Overall this is great, seeing that another project was beating Dask motivated me to dive in (see dask/distributed #932) and this will result in a better experience for users in the future. As a developer this is also how I operate. I define a benchmark, profile my code, identify hot-spots, and optimize. Business as usual.

However as an author of a comparative benchmark this also somewhat dishonest; I’m not giving the Spark developers the same opportunity to find and fix similar performance issues in their software before I publish my results. I’m also giving a biased picture to my readers. I’ve made all of the pieces that I’m going to show off fast while neglecting the others. Picking benchmarks, optimizing the project to make them fast, and then publishing those results gives the incorrect impression that the entire project has been optimized to that level.

Omission

So, this didn’t go as planned. Lets wait a few months until the next release.

There is no motivation to publish negative results. Unless of course you’ve just written a blogpost announcing that you plan to release benchmarks in the near future. Then you’re really forced to release numbers, even if they’re mixed.

That’s ok, mixed numbers can be informative. They build trust and community. And we all talk about open source community driven software, so these should be welcome.

Straight up bias

Look, we’re not in grad-school any more. We’ve got to convince companies to actually use this stuff.

Everything we’ve discussed so far assumes best intentions, that the author is acting in good faith, but falling victim to basic human failings.

However many developers today (including myself) are paid and work for for-profit companies that need to make money. To an increasing extent making this money depends on community mindshare, which means publishing benchmarks that sway users to our software. Authors have bosses that they’re trying to impress or the content and tone of an article may be influenced by other people within the company other than the stated author.

I’ve been pretty lucky working with Continuum Analytics (my employer) in that they’ve been pretty hands-off with technical writing. For other employers that may be reading, we’ve actually had an easier time getting business because of the honest tone in these blogposts in some cases. Potential clients generally have the sense that we’re trustworthy.

Technical honesty goes a surprisingly long way towards implying technical proficiency.

This work is supported by Continuum Analytics, the XDATA Program, and the Data Driven Discovery Initiative from the Moore Foundation.

Summary

We build distributed optimization algorithms with Dask.  We show both simple examples and also benchmarks from a nascent dask-glm library for generalized linear models.  We also talk about the experience of learning Dask to do this kind of work.

This blogpost is co-authored by Chris White (Capital One) who knows optimization and Matthew Rocklin (Continuum Analytics) who knows distributed computing.

Introduction

Many machine learning and statistics models (such as logistic regression) depend on convex optimization algorithms like Newton’s method, stochastic gradient descent, and others.  These optimization algorithms are both pragmatic (they’re used in many applications) and mathematically interesting.  As a result these algorithms have been the subject of study by researchers and graduate students around the world for years both in academia and in industry.

Things got interesting about five or ten years ago when datasets grew beyond the size of working memory and “Big Data” became a buzzword.  Parallel and distributed solutions for these algorithms have become the norm, and a researcher’s skillset now has to extend beyond linear algebra and optimization theory to include parallel algorithms and possibly even network programming, especially if you want to explore and create more interesting algorithms.

However, relatively few people understand both mathematical optimization theory and the details of distributed systems. Typically algorithmic researchers depend on the APIs of distributed computing libraries like Spark or Flink to implement their algorithms. In this blogpost we explore the extent to which Dask can be helpful in these applications. We approach this from two perspectives:

1. Algorithmic researcher (Chris): someone who knows optimization and iterative algorithms like Conjugate Gradient, Dual Ascent, or GMRES but isn’t so hot on distributed computing topics like sockets, MPI, load balancing, and so on
2. Distributed systems developer (Matt): someone who knows how to move bytes around and keep machines busy but doesn’t know the right way to do a line search or handle a poorly conditioned matrix

Prototyping Algorithms in Dask

Given knowledge of algorithms and of NumPy array computing it is easy to write parallel algorithms with Dask. For a range of complicated algorithmic structures we have two straightforward choices:

1. Use parallel multi-dimensional arrays to construct algorithms from common operations like matrix multiplication, SVD, and so on. This mirrors mathematical algorithms well but lacks some flexibility.
2. Create algorithms by hand that track operations on individual chunks of in-memory data and dependencies between them. This is very flexible but requires a bit more care.

Coding up either of these options from scratch can be a daunting task, but with Dask it can be as simple as writing NumPy code.

Let’s build up an example of fitting a large linear regression model using both built-in array parallelism and fancier, more customized parallelization features that Dask offers. The dask.array module helps us to easily parallelize standard NumPy functionality using the same syntax – we’ll start there.

Data Creation

Dask has many ways to create dask arrays; to get us started quickly prototyping let’s create some random data in a way that should look familiar to NumPy users.

importdaskimportdask.arrayasdaimportnumpyasnpfromdask.distributedimportClientclient=Client()## create inputs with a bunch of independent normalsbeta=np.random.random(100)# random beta coefficients, no interceptX=da.random.normal(0,1,size=(1000000,100),chunks=(100000,100))y=X.dot(beta)+da.random.normal(0,1,size=1000000,chunks=(100000,))## make sure all chunks are ~equally sizedX,y=dask.persist(X,y)client.rebalance([X,y])

Observe that X is a dask array stored in 10 chunks, each of size (100000, 100). Also note that X.dot(beta) runs smoothly for both numpy and dask arrays, so we can write code that basically works in either world.

Caveat: If X is a numpy array and beta is a dask array, X.dot(beta) will output an in-memorynumpy array. This is usually not desirable as you want to carefully choose when to load something into memory. One fix is to use multipledispatch to handle odd edge cases; for a starting example, check out the dot code here.

Dask also has convenient visualization features built in that we will leverage; below we visualize our data in its 10 independent chunks:

Array Programming

If you can write iterative array-based algorithms in NumPy, then you can write iterative parallel algorithms in Dask

As we’ve already seen, Dask inherits much of the NumPy API that we are familiar with, so we can write simple NumPy-style iterative optimization algorithms that will leverage the parallelism dask.array has built-in already. For example, if we want to naively fit a linear regression model on the data above, we are trying to solve the following convex optimization problem:

Recall that in non-degenerate situations this problem has a closed-form solution that is given by:

We can compute $\beta^*$ using the above formula with Dask:

## naive solutionbeta_star=da.linalg.solve(X.T.dot(X),X.T.dot(y))>>>abs(beta_star.compute()-beta).max()0.0024817567237768179

Sometimes a direct solve is too costly, and we want to solve the above problem using only simple matrix-vector multiplications. To this end, let’s take this one step further and actually implement a gradient descent algorithm which exploits parallel matrix operations. Recall that gradient descent iteratively refines an initial estimate of beta via the update:

where $\alpha$ can be chosen based on a number of different “step-size” rules; for the purposes of exposition, we will stick with a constant step-size:

## quick step-size calculation to guarantee convergence_,s,_=np.linalg.svd(2*X.T.dot(X))step_size=1/s-1e-8## define some parametersmax_steps=100tol=1e-8beta_hat=np.zeros(100)# initial guessforkinrange(max_steps):Xbeta=X.dot(beta_hat)func=((y-Xbeta)**2).sum()gradient=2*X.T.dot(Xbeta-y)## Updateobeta=beta_hatbeta_hat=beta_hat-step_size*gradientnew_func=((y-X.dot(beta_hat))**2).sum()beta_hat,func,new_func=dask.compute(beta_hat,func,new_func)# <--- Dask code## Check for convergencechange=np.absolute(beta_hat-obeta).max()ifchange<tol:break>>>abs(beta_hat-beta).max()0.0024817567259038942

It’s worth noting that almost all of this code is exactly the same as the equivalent NumPy code. Because Dask.array and NumPy share the same API it’s pretty easy for people who are already comfortable with NumPy to get started with distributed algorithms right away. The only thing we had to change was how we produce our original data (da.random.normal instead of np.random.normal) and the call to dask.compute at the end of the update state. The dask.compute call tells Dask to go ahead and actually evaluate everything we’ve told it to do so far (Dask is lazy by default). Otherwise, all of the mathematical operations, matrix multiplies, slicing, and so on are exactly the same as with Numpy, except that Dask.array builds up a chunk-wise parallel computation for us and Dask.distributed can execute that computation in parallel.

To better appreciate all the scheduling that is happening in one update step of the above algorithm, here is a visualization of the computation necessary to compute beta_hat and the new function value new_func:

Each rectangle is an in-memory chunk of our distributed array and every circle is a numpy function call on those in-memory chunks. The Dask scheduler determines where and when to run all of these computations on our cluster of machines (or just on the cores of our laptop).

Array Programming + dask.delayed

Now that we’ve seen how to use the built-in parallel algorithms offered by Dask.array, let’s go one step further and talk about writing more customized parallel algorithms. Many distributed “consensus” based algorithms in machine learning are based on the idea that each chunk of data can be processed independently in parallel, and send their guess for the optimal parameter value to some master node. The master then computes a consensus estimate for the optimal parameters and reports that back to all of the workers. Each worker then processes their chunk of data given this new information, and the process continues until convergence.

From a parallel computing perspective this is a pretty simple map-reduce procedure. Any distributed computing framework should be able to handle this easily. We’ll use this as a very simple example for how to use Dask’s more customizable parallel options.

One such algorithm is the Alternating Direction Method of Multipliers, or ADMM for short. For the sake of this post, we will consider the work done by each worker to be a black box.

We will also be considering a regularized version of the problem above, namely:

At the end of the day, all we will do is:

• create NumPy functions which define how each chunk updates its parameter estimates
• wrap those functions in dask.delayed
• call dask.compute and process the individual estimates, again using NumPy

First we need to define some local functions that the chunks will use to update their individual parameter estimates, and import the black box local_update step from dask_glm; also, we will need the so-called shrinkage operator (which is the proximal operator for the $l1$-norm in our problem):

fromdask_glm.algorithmsimportlocal_updatedeflocal_f(beta,X,y,z,u,rho):return((y-X.dot(beta))**2).sum()+(rho/2)*np.dot(beta-z+u,beta-z+u)deflocal_grad(beta,X,y,z,u,rho):return2*X.T.dot(X.dot(beta)-y)+rho*(beta-z+u)defshrinkage(beta,t):returnnp.maximum(0,beta-t)-np.maximum(0,-beta-t)## set some algorithm parametersmax_steps=10lamduh=7.2rho=1.0(n,p)=X.shapenchunks=X.npartitionsXD=X.to_delayed().flatten().tolist()# A list of pointers to remote numpy arraysyD=y.to_delayed().flatten().tolist()# ... one for each chunk# the initial consensus estimatez=np.zeros(p)# an array of the individual "dual variables" and parameter estimates,# one for each chunk of datau=np.array([np.zeros(p)foriinrange(nchunks)])betas=np.array([np.zeros(p)foriinrange(nchunks)])forkinrange(max_steps):# process each chunk in parallel, using the black-box 'local_update' magicnew_betas=[dask.delayed(local_update)(xx,yy,bb,z,uu,rho,f=local_f,fprime=local_grad)forxx,yy,bb,uuinzip(XD,yD,betas,u)]new_betas=np.array(dask.compute(*new_betas))# everything else is NumPy code occurring at "master"beta_hat=0.9*new_betas+0.1*z# create consensus estimatezold=z.copy()ztilde=np.mean(beta_hat+np.array(u),axis=0)z=shrinkage(ztilde,lamduh/(rho*nchunks))# update dual variablesu+=beta_hat-z>>># Number of coefficients zeroed out due to L1 regularization>>>print((z==0).sum())12

There is of course a little bit more work occurring in the above algorithm, but it should be clear that the distributed operations are not one of the difficult pieces. Using dask.delayed we were able to express a simple map-reduce algorithm like ADMM with similarly simple Python for loops and delayed function calls. Dask.delayed is keeping track of all of the function calls we wanted to make and what other function calls they depend on. For example all of the local_update calls can happen independent of each other, but the consensus computation blocks on all of them.

We hope that both parallel algorithms shown above (gradient descent, ADMM) were straightforward to someone reading with an optimization background. These implementations run well on a laptop, a single multi-core workstation, or a thousand-node cluster if necessary. We’ve been building somewhat more sophisticated implementations of these algorithms (and others) in dask-glm. They are more sophisticated from an optimization perspective (stopping criteria, step size, asynchronicity, and so on) but remain as simple from a distributed computing perspective.

Experiment

We compare dask-glm implementations against Scikit-learn on a laptop, and then show them running on a cluster.

Reproducible notebook is available here

We’re building more sophisticated versions of the algorithms above in dask-glm.  This project has convex optimization algorithms for gradient descent, proximal gradient descent, Newton’s method, and ADMM.  These implementations extend the implementations above by also thinking about stopping criteria, step sizes, and other niceties that we avoided above for simplicity.

In this section we show off these algorithms by performing a simple numerical experiment that compares the numerical performance of proximal gradient descent and ADMM alongside Scikit-Learn’s LogisticRegression and SGD implementations on a single machine (a personal laptop) and then follows up by scaling the dask-glm options to a moderate cluster.  

Disclaimer: These experiments are crude. We’re using artificial data, we’re not tuning parameters or even finding parameters at which these algorithms are producing results of the same accuracy. The goal of this section is just to give a general feeling of how things compare.

We create data

## size of problem (no. observations)N=8e6chunks=1e6seed=20009beta=(np.random.random(15)-0.5)*3X=da.random.random((N,len(beta)),chunks=chunks)y=make_y(X,beta=np.array(beta),chunks=chunks)X,y=dask.persist(X,y)client.rebalance([X,y])

And run each of our algorithms as follows:

# Dask-GLM Proximal Gradientresult=proximal_grad(X,y,lamduh=alpha)# Dask-GLM ADMMX2=X.rechunk((1e5,None)).persist()# ADMM prefers smaller chunksy2=y.rechunk(1e5).persist()result=admm(X2,y2,lamduh=alpha)# Scikit-Learn LogisticRegressionnX,ny=dask.compute(X,y)# sklearn wants numpy arraysresult=LogisticRegression(penalty='l1',C=1).fit(nX,ny).coef_# Scikit-Learn Stochastic Gradient Descentresult=SGDClassifier(loss='log',penalty='l1',l1_ratio=1,n_iter=10,fit_intercept=False).fit(nX,ny).coef_

We then compare with the $L_{\infty}$ norm (largest different value).

abs(result-beta).max()

Times and $L_\infty$ distance from the true “generative beta” for these parameters are shown in the table below:

Again, please don’t take these numbers too seriously: these algorithms all solve regularized problems, so we don’t expect the results to necessarily be close to the underlying generative beta (even asymptotically). The numbers above are meant to demonstrate that they all return results which were roughly the same distance from the beta above. Also, Dask-glm is using a full four-core laptop while SKLearn is restricted to use a single core.

In the sections below we include profile plots for proximal gradient and ADMM. These show the operations that each of eight threads was doing over time. You can mouse-over rectangles/tasks and zoom in using the zoom tools in the upper right. You can see the difference in complexity of the algorithms. ADMM is much simpler from Dask’s perspective but also saturates hardware better for this chunksize.

Profile Plot for Proximal Gradient Descent

Profile Plot for ADMM

The general takeaway here is that dask-glm performs comparably to Scikit-Learn on a single machine. If your problem fits in memory on a single machine you should continue to use Scikit-Learn and Statsmodels. The real benefit to the dask-glm algorithms is that they scale and can run efficiently on data that is larger-than-memory by operating from disk on a single computer or on a cluster of computers working together.

Cluster Computing

As a demonstration, we run a larger version of the data above on a cluster of eight m4.2xlarges on EC2 (8 cores and 30GB of RAM each.)

We create a larger dataset with 800,000,000 rows and 15 columns across eight processes.

N=8e8chunks=1e7seed=20009beta=(np.random.random(15)-0.5)*3X=da.random.random((N,len(beta)),chunks=chunks)y=make_y(X,beta=np.array(beta),chunks=chunks)X,y=dask.persist(X,y)

We then run the same proximal_grad and admm operations from before:

# Dask-GLM Proximal Gradientresult=proximal_grad(X,y,lamduh=alpha)# Dask-GLM ADMMX2=X.rechunk((1e6,None)).persist()# ADMM prefers smaller chunksy2=y.rechunk(1e6).persist()result=admm(X2,y2,lamduh=alpha)

Proximal grad completes in around seventeen minutes while ADMM completes in around four minutes. Profiles for the two computations are included below:

Profile Plot for Proximal Gradient Descent

We include only the first few iterations here. Otherwise this plot is several megabytes.

Link to fullscreen plot

Profile Plot for ADMM

Link to fullscreen plot

These both obtained similar $L_{\infty}$ errors to what we observed before.

Although this time we had to be careful about a couple of things:

1. We explicitly deleted the old data after rechunking (ADMM prefers different chunksizes than proximal_gradient) because our full dataset, 100GB, is close enough to our total distributed RAM (240GB) that it’s a good idea to avoid keeping replias around needlessly. Things would have run fine, but spilling excess data to disk would have negatively affected performance.
2. We set the OMP_NUM_THREADS=1 environment variable to avoid over-subscribing our CPUs. Surprisingly not doing so led both to worse performance and to non-deterministic results. An issue that we’re still tracking down.

Analysis

The algorithms in Dask-GLM are new and need development, but are in a usable state by people comfortable operating at this technical level. Additionally, we would like to attract other mathematical and algorithmic developers to this work. We’ve found that Dask provides a nice balance between being flexible enough to support interesting algorithms, while being managed enough to be usable by researchers without a strong background in distributed systems. In this section we’re going to discuss the things that we learned from both Chris’ (mathematical algorithms) and Matt’s (distributed systems) perspective and then talk about possible future work. We encourage people to pay attention to future work; we’re open to collaboration and think that this is a good opportunity for new researchers to meaningfully engage.

Chris’s perspective

1. Creating distributed algorithms with Dask was surprisingly easy; there is still a small learning curve around when to call things like persist, compute, rebalance, and so on, but that can’t be avoided. Using Dask for algorithm development has been a great learning environment for understanding the unique challenges associated with distributed algorithms (including communication costs, among others).
2. Getting the particulars of algorithms correct is non-trivial; there is still work to be done in better understanding the tolerance settings vs. accuracy tradeoffs that are occurring in many of these algorithms, as well as fine-tuning the convergence criteria for increased precision.
3. On the software development side, reliably testing optimization algorithms is hard. Finding provably correct optimality conditions that should be satisfied which are also numerically stable has been a challenge for me.
4. Working on algorithms in isolation is not nearly as fun as collaborating on them; please join the conversation and contribute!
5. Most importantly from my perspective, I’ve found there is a surprisingly large amount of misunderstanding in “the community” surrounding what optimization algorithms do in the world of predictive modeling, what problems they each individually solve, and whether or not they are interchangeable for a given problem. For example, Newton’s method can’t be used to optimize an l1-regularized problem, and the coefficient estimates from an l1-regularized problem are fundamentally (and numerically) different from those of an l2-regularized problem (and from those of an unregularized problem). My own personal goal is that the API for dask-glm exposes these subtle distinctions more transparently and leads to more thoughtful modeling decisions “in the wild”.

Matt’s perspective

This work triggered a number of concrete changes within the Dask library:

1. We can convert Dask.dataframes to Dask.arrays. This is particularly important because people want to do pre-processing with dataframes but then switch to efficient multi-dimensional arrays for algorithms.
2. We had to unify the single-machine scheduler and distributed scheduler APIs a bit, notably adding a persist function to the single machine scheduler. This was particularly important because Chris generally prototyped on his laptop but we wanted to write code that was effective on clusters.
3. Scheduler overhead can be a problem for the iterative dask-array algorithms (gradient descent, proximal gradient descent, BFGS). This is particularly a problem because NumPy is very fast. Often our tasks take only a few milliseconds, which makes Dask’s overhead of 200us per task become very relevant (this is why you see whitespace in the profile plots above). We’ve started resolving this problem in a few ways like more aggressive task fusion and lower overheads generally, but this will be a medium-term challenge. In practice for dask-glm we’ve started handling this just by choosing chunksizes well. I suspect that for the dask-glm in particular we’ll just develop auto-chunksize heuristics that will mostly solve this problem. However we expect this problem to recur in other work with scientists on HPC systems who have similar situations.
4. A couple of things can be tricky for algorithmic users:
   1. Placing the calls to asynchronously start computation (persist, compute). In practice Chris did a good job here and then I came through and tweaked things afterwards. The web diagnostics ended up being crucial to identify issues.
   2. Avoiding accidentally calling NumPy functions on dask.arrays and vice versa. We’ve improved this on the dask.array side, and they now operate intelligently when given numpy arrays. Changing this on the NumPy side is harder until NumPy protocols change (which is planned).

Future work

There are a number of things we would like to do, both in terms of measurement and for the dask-glm project itself. We welcome people to voice their opinions (and join development) on the following issues:

1. Asynchronous Algorithms
2. User APIs
3. Extend GLM families
4. Write more extensive rigorous algorithm testing - for satisfying provable optimality criteria, and for robustness to various input data
5. Begin work on smart initialization routines

What is your perspective here, gentle reader? Both Matt and Chris can use help on this project. We hope that some of the issues above provide seeds for community engagement. We welcome other questions, comments, and contributions either as github issues or comments below.

Acknowledgements

Thanks also go to Hussain Sultan (Capital One) and Tom Augspurger for collaboration on Dask-GLM and to Will Warner (Continuum) for reviewing and editing this post.

This work is supported by Continuum Analytics, the XDATA Program, and the Data Driven Discovery Initiative from the Moore Foundation.

I’m pleased to announce the release of Dask version 0.14.1. This release contains a variety of performance and feature improvements. This blogpost includes some notable features and changes since the last release on February 27th.

As always you can conda install from conda-forge

conda install -c conda-forge dask distributed

or you can pip install from PyPI

pip install dask[complete] --upgrade

Arrays

Recent work in distributed computing and machine learning have motivated new performance-oriented and usability changes to how we handle arrays.

Automatic chunking and operation on NumPy arrays

Many interactions between Dask arrays and NumPy arrays work smoothly. NumPy arrays are made lazy and are appropriately chunked to match the operation and the Dask array.

>>>x=np.ones(10)# a numpy array>>>y=da.arange(10,chunks=(5,))# a dask array>>>z=x+y# combined become a dask.array>>>zdask.array<add,shape=(10,),dtype=float64,chunksize=(5,)>>>>z.compute()array([1.,2.,3.,4.,5.,6.,7.,8.,9.,10.])

Reshape

Reshaping distributed arrays is simple in simple cases, and can be quite complex in complex cases. Reshape now supports a much more broad set of shape transformations where any dimension is collapsed or merged to other dimensions.

>>>x=da.ones((2,3,4,5,6),chunks=(2,2,2,2,2))>>>x.reshape((6,2,2,30,1))dask.array<reshape,shape=(6,2,2,30,1),dtype=float64,chunksize=(3,1,2,6,1)>

This operation ends up being quite useful in a number of distributed array cases.

Optimize Slicing to Minimize Communication

Dask.array slicing optimizations are now careful to produce graphs that avoid situations that could cause excess inter-worker communication. The details of how they do this is a bit out of scope for a short blogpost, but the history here is interesting.

Historically dask.arrays were used almost exclusively by researchers with large on-disk arrays stored as HDF5 or NetCDF files. These users primarily used the single machine multi-threaded scheduler. We heavily tailored Dask array optimizations to this situation and made that community pretty happy. Now as some of that community switches to cluster computing on larger datasets the optimization goals shift a bit. We have tons of distributed disk bandwidth but really want to avoid communicating large results between workers. Supporting both use cases is possible and I think that we’ve achieved that in this release so far, but it’s starting to require increasing levels of care.

Micro-optimizations

With distributed computing also comes larger graphs and a growing importance of graph-creation overhead. This has been optimized somewhat in this release. We expect this to be a focus going forward.

DataFrames

Set_index

Set_index is smarter in two ways:

1. If you set_index on a column that happens to be sorted then we’ll identify that and avoid a costly shuffle. This was always possible with the sorted= keyword but users rarely used this feature. Now this is automatic.
2. Similarly when setting the index we can look at the size of the data and determine if there are too many or too few partitions and rechunk the data while shuffling. This can significantly improve performance if there are too many partitions (a common case).

• dask/dask #2025
• dask/dask #2091

Shuffle performance

We’ve micro-optimized some parts of dataframe shuffles. Big thanks to the Pandas developers for the help here. This accelerates set_index, joins, groupby-applies, and so on.

• dask/dask #2032

Fastparquet

The fastparquet library has seen a lot of use lately and has undergone a number of community bugfixes.