Manthan Thakar

:sparkles: Optimizing joins in Apache Spark using Scala compiler plugin and broadcast join

2017-12-22T00:00:00+00:00

TLDR With our Scala compiler plugin, in the best case we were able to decrease shuffled bytes by 89% and runtime by 24%.

Join operation on RDDs can be expensive. We suspect that one of the biggest factors that affects join performance is the amount of data shuffled in the process. In order to alleviate excessive shuffling of data, we propose and implement two types of optimization, namely, Column Pruning and Broadcast Join. In the following sections, we discuss both of these approaches and present our findings.

Optimization 1: Column Pruning

Hypothesis: Pruning unused columns (values in Pair-RDD) of an RDD before performing a join reduces the amount of data shuffled for join and improves join performance.

Approach

At a higher-level, we need following steps to perform column pruning on a spark join.

Identify target RDDs Identify RDDs on which join is being performed.
Obtain column Usage The columns from each RDD that are used after join operation.
Transform target RDDs Using the column usage information, transform the target RDDs before join is performed.

In order to capture information required to perform steps above, we build a compiler-plugin for Scala compiler. Scala compiler plugin gives us the facility to access the abstract syntax tree (AST) of the source code after different phases of the compiler. This information would be either completely missing or hard to obtain inside Spark’s DAG scheduler. Hence, we believe that a compiler plugin is a better choice for our purposes.

Scala Compiler Plugin

Scala compiler can be modified by building compiler plugins. Scala compiler has 25 phases including phases like parser, typer, erasure, etc. The compiler plugin can be placed before or after any of these phases. It can then be used to analyze and modify AST from the previous phase. We use this capability and place our compiler plugin JoinOptimizer after Scala compiler’s parser phase.

When a Scala program is compiled using JoinOptimizer, the plugin obtains the AST constructed by compiler’s parser, at which point JoinOptimizer performs following steps:

Identify join operations in a Scala program using Scala’s quasiquotes.
Iff a join operation is encountered:
- Capture the trees for target RDDs and the next Transformation that directly follows the join.
- Analyze this Transformation’s lambda and obtain column usage of target RDDs.
- Transform the tree before the join operation to insert a mapValues on target RDD that emits only used columns.

Two PairRDDs rdd1 and rdd2 are defined in line 1 and 2 in fig 1, both with key data type of Long and value data type of Tuple5[Long, ...]. Line 5 contains code that performs join on rdd1 and rdd2 and then uses the first column from rdd1 and second column from rdd2 inside the lambda enclosed in mapValues. It is quite obvious that 4 columns in both RDDs are unused after join. Hence, these unused columns can be pruned.

Lines 8-10 show the code generated by our compiler plugin. It prunes the columns from both rdd1 and rdd2 before the join by adding an extra mapValues stage. Note that, for rdd2 it uses a null value for first column since it is not used. To avoid rewriting user code Transformations following joins we replace all unused indices in a Tuple with null values. We hypothesize that this would help save bytes and shuffles.

Benchmarks

To measure the impact of JoinOptimizer plugin we compile same spark programs with and without the plugin. The spark program used for benchmarking performs joins between two RDDs, rdd1 and rdd2. We tested multiple combinations of number of columns available in source RDD’s and number columns used post join operation. Note that, these configurations are only varied for rdd1 and for simplicity, we keep rdd2 constant for all benchmarks. For all the configurations, we measure execution times and amount of data shuffled.

All the benchmarks are run on a local machine with configuration: 2.7 GHz Intel Core i5 processor, 8 GB RAM, 4 vCores, 256GB SSD Storage, Spark 2.2.0, Scala 2.11.11

Compilation Time

Fig 2 shows the distribution of compilation time for the same spark program with and without the compiler plugin. From the violin plot, it can be seen that for majority of distribution, compiler plugin adds overhead of about 2 seconds. In the worst case (for the outliers), this overhead can be more than 5 seconds.

Shuffled Data

Fig 3 (shuffle) shows the amount of data shuffled for both spark programs compiled with (depicted as Plugin) and without (depicted as Control) JoinOptimizer. Each block in heatmaps shows median kb shuffled for different configurations of columns. X-axis shows the total number of columns before join and Y-axis shows the used number of columns used post join.

From the Control plot, it can be observed that the amount of data shuffled stays roughly the same for any value on X-axis. This indicates that in Control shuffle seems to be a function of total number of columns in the source RDDs. In contrast, for the Plugin plot, the shuffled data increases only when the used number columns increases. It bolsters our initial hypothesis. If number of columns used are constant, regardless of number of total columns shuffled data stays almost constant. Thus, in Plugin shuffle becomes a function of used number columns in source RDDs.

Fig 3 also shows the delta for shuffle data between both control and plugin benchmarks. The highest gain in network efficiency of 89% is observed when total number of columns in rdd1 is 22% and none of the columns are used after join. This means that the optimized code shuffles 89% less data than its unoptimized counterpart. Moreover, the percentage gain increases horizontally from left to right for each value on Y-axis indicating the decrease in shuffling compared to unoptimized version, when total number of columns increase but number of columns used remains constant. Surprisingly, when there’s only one column in RDD and none of the columns are used, optimized version shuffles 23% more data than unoptimized version which was unexpected. This could be a limitation in our plugin implementation that converts the RDD values and maps the values as Tuple22 with 22 x null values.

Execution Time

Similarly, Fig 3 (Time) shows execution times for optimized (depicted as plugin) and unoptimized (depicted as control) programs. Each block in heatmaps shows median execution time in seconds for different configurations of columns in rdd1. The patterns observed for shuffled data in both control and plugin plots also exist for execution times. This indicates that there is high correlation between shuffled data and execution time.

Optimization 2: Broadcast Join

Hypothesis: Out of two RDDs on which join is to be performed, if one RDD is much smaller than the other and it can fit in memory, then broadcasting the small RDD to all the nodes and then performing map-side join reduces the amount of data being shuffled and improves join performance.

Approach

Spark’s join function by default performs a reduce-side join. Meaning, it shuffles records from both target RDDs having same keys to one node and then combines their data. But in situations where one of the RDDs is much smaller, this smaller RDD can be grouped and broadcast to all the nodes. Once the smaller RDD is transferred to each node, both RDDs can be joined by adding map stages. This eliminates an extra reduce step that is required in reduce-side joins. Although, broadcast join includes sending data to all the nodes upfront, it saves the bigger RDD from being shuffled. This is equivalent to performing a map-side join instead of spark’s reduce-side join.

The key components in this proposed approach are:

1. RDD Size Estimation

Estimating the size of target RDDs is instrumental in performing broadcast join. We have implemented a size estimator for RDDs that uses the following formula:

estimated_size = size(N) * R / N

Where R is total number of rows in the RDD, and size(N) gives total size of N sample rows fetched from each partition.

Since this is an heuristic approach, it is prone to underestimation when rows contain variable number of columns. In such cases broadcasting an RDD might result in ungraceful job failure. Therefore, we keep the estimator configurable for the user, so that user can provide a custom size estimator according to data.

Moreover, broadcast join requires that the small RDD can fit in memory on all the nodes. We assumed this property to be a configuration setting of a cluster and we kept memory threshold configurable as well. If size of an RDD obtained using the given estimator is smaller than the memory threshold, we perform broadcast join; otherwise Spark’s default shuffle-join is performed.

2. Performing map-side join

If one of the target RDDs is determined to be smaller than threshold by Size Estimator, we groupByKey and broadcast it to all nodes to be cached. Once cached, we only need to map through the bigger RDD to lookup its keys in the cache to join records. This in turn eliminates shuffling of bigger RDD and a reduce stage.

Benchmarks

Figure 4 shows benchmark results for our broadcast join implementation for varying sizes of both small RDD (smaller than threshold) and big RDD. The varying RDD size is shown on X-axis and the other RDD is kept constant. Y-axis depicts the execution time taken to perform join.

Fixed size small RDD

To measure the impact of increasing size of bigger RDD on broadcast join, we keep the size of small RDD constant and vary big RDD. Figure 4(a) shows execution times of joins for this experiment in two different scenarios.

The first scenario is when one RDD is small and the other RDD is big marked as squares. BS:Shuffle shows the execution time for shuffle RDD join in this scenario for varying sizes of big RDD. BS:BJ similarly shows the execution time for broadcast join. This is an ideal scenario for a broadcast join. Therefore, as the size of the big RDD increases, broadcast join yields better performance. This is because, shuffle join has to send more data over network as the size of big RDD increases. We observed about 50% of improvement in performance for broadcast join of small RDD to big RDD.

The second scenario is less ideal for broadcast join. In this scenario, both RDDs are big and therefore can’t be broadcast. The execution times for this scenario are marked as triangles in Fig 4(a), where BB:BJ means execution times for broadcast join and BB:Shuffle means execution times for shuffle joins. Here, as the size of big RDD increases, shuffle join yields better performance. We believe that this is because of the estimation overhead incurred by the size estimator. We observed about 50% of decrease in performance for broadcast join of small RDD to big RDD.

Fixed size big RDD

To understand the ideal broadcast size for small RDD, we keep big RDD constant and vary the size of small RDD along with broadcast memory threshold. We plot the execution time of join operation for different sizes of small RDD. Figure 4(b) shows execution times for this set up. Here, broadcast join times are marked as triangles and shuffle join times are marked as squares. With our local configuration, we observed that once the size of broadcast RDD increases more than 256 mb, performance gains obtained using broadcast join starts to diminish.

Conclusion

Why doesn’t Spark perform column pruning on RDD joins? We started our project with this simple question. We found that for doing any optimization like this Spark needs access to source code and recompile lambdas and/or functions to an optimal code.
We also found Scala compiler plugin could prove to be a very useful Spark companion in optimizing job and with considerable shuffle saves for poorly written jobs. Provided appropriate documentations become available for Scala compiler plugin for new developers to get onboarded.
Broadcast Join seems to outperform shuffle join when one RDD is smaller than 256 mb in our tests. It could prove very useful in case of unknown RDD sizes, when actual RDD size is suspected to be small while writing the job.

Future Work

To overcome restriction to view the source code. We would like to explore analyzing generated source from Byte Code decompiler. We could then move this whole logic to spark DAG creation as initially planned.

:airplane: Visualizing flight delays using Hadoop and R

2017-11-15T11:02:34+00:00

:octocat: Source code on GitHub

In this post I talk about my flight delay analysis using flight on-time performance data published by Bureau of Transportation Statistics.

The flight dataset used contains flight data from 1987 to 2015. While this dataset isn’t very large, it still requires some careful data processing. For this purpose, I decided to use Apache Hadoop for data cleaning and processing. The cleaned data is then used for visualizations using R.

If you’d like to dig deep, you can play around with the source code.

Data Processing

As mentioned, Apache hadoop is used to process the flight dataset. Although the dataset contains many interesting fields, we’re only interested in getting the delays at top 5 airports and top 5 airlines for different months and different years.

As expected, the dataset contains quite a few invalid records with corrupt values. The mapreduce job written for filtering out such invalid records roughly functions as follows:

Mapper: The map phase of the mapreduce job is responsible for

Validating each record by performing sanity checks
Emitting valid records as key-value pairs

Reducer: In the reduce phase all the flights having the same output key (same airline, airport, year and month) are combined by aggregating the delay and flight count in DelayWritable. Since, there could be many flights with same key in one file, we apply the same reducer as combiner which significantly reduced the amount of data shuffled for the reduce phase.

Most active airlines and airports

Figure 1 shows the most active airlines and airports from years 1987 to 2015. South West Airlines is the most active airline with astonishing 20 million flights. Atlanta’s Hartsfield-Jackson is the most active airport, while Chicago’s O’Hare closely comes at second place.

Delays

Figure 2 shows mean delay per year for top airlines and airports and apply loess smoothing to obtain a trend line with confidence intervals. Note that, outliers are clipped in the graphs shown below in favor of readable plots.

Among the top 5 airlines in fig 2.1, North West airlines (NW) seems to be having very high variations in mean delays across years. Mean delays for NW seem to be decreasing starting from 1988 to 1993. They increase again starting from 1994.

It can also be observed that United Airlines (UA) has wider error bounds, indicating highly varying mean delays across years, making the pattern less predictable. Moreover, South West airlines (WN) has been seeing steady increase in mean delays.

Similarly in fig 2.2, we can observe that the mean delays for ATL airport increases up to 2006 and then steeply decreases. Interestingly, LAX and ORD seem to be having exactly the opposite trends for mean delays. While the delay trend in ORD across years roughly resembles letter “S”, the trend line in LAX looks like an upside-down version of it. The Denver airport (DEN) seems to be having the least variation in delays with trend line staying around 10 minutes mark.

Figure 3 shows a heat map of mean delays across all years from all airlines to top 5 airports. Each square block in the heat map shows the mean delay from an airline to an airport and the delay is proportional to the intensity of the color in that block. It’s important to note that the white blocks with lines inside show NA values, meaning there’s no flight data available from an airline to a particular destination.

Among top 5 airports, ORD contains many dark blocks across all airlines, suggesting mean delays above 15 minutes while PHX has most of the mean delays close to 10 minutes. As a new airline, you wouldn’t want your first flight to go to ORD.

Out of all airlines, NK - Spirit Airlines, seems to be having highest mean delays while traveling to top 5 airports. The delays increase especially for LAX and ORD. AS seems to be having least delays going to top 5 airports. As a traveler going to Los Angeles, from the historical data, you’d be better off traveling in MQ than NK.

All of the top 5 airlines relatively have lesser delays while going to top 5 airports, despite of having higher volume of flights.

Figure 4 shows mean delays for top 5 airlines and airports across all months of all years, as a heat map. We decided to explore the idea of visualizing mean delays per month per year, because it wouldn’t be useful to aggregate delays for a month across all years. As Figure 3, the darker the block higher the delay. But here, different delay categories have different colors. In general, blue blocks indicate smaller delays and red blocks indicate higher delays. Blocks with shades of green, yellow and orange generally fall into medium range.

Figure 4.1 shows mean delays per month for top 5 airlines for all years. For WN, which is also the most active airline, September seems to be the month having least delays across all years. In fact, for all the top 5 airlines, in terms of delays September seems to be the best month across all years. Conversely, December is the worst month for delays for all of the top 5 airlines.

Figure 4.2 similarly shows mean delays per month for top 5 airports for all years. This graph seems to be agreeing with the previous observation that the delays are smaller in September across years. One interesting observation in figure 4.2 is that all of the top 5 airports seems to be having higher delays in the year of 2000. Moreover, for year 2000 airports DEN and ORD seems to be having similar delay patterns for months Jun, July and August and December to some extent.

This visualization makes it easy to point out particularly bad months in a year. Moreover, we can identify airports that have higher seasonal delays or had higher seasonal delay in any particular year. For example, ORD in general has a lot of orange/red blocks. But one pattern that stands out starts around December 2006 and continues till December 2008. This pattern shows that there very high delays starting from December continuing till February of next year. This could be attributed to stormy winters which would generally occur around those months.

Conclusion

With Apache Hadoop and R, it’s really simple to build reproducible data pipelines. If you’d like to run this pipeline, be sure to check the github repository of the project.

Thread-safe Memory Allocator (Malloc) library

2017-10-26T22:10:00+00:00

:octocat: Source code on GitHub

A thread-safe memory allocator based on Buddy Memory Allocation Alogirithm

Design Overview

The functions provided with this implementation include:

void *malloc(size_t size);
void free(void *ptr);
void *calloc(size_t nmemb, size_t size);
void *realloc(void *ptr, size_t size);
struct mallinfo mallinfo();
void malloc_stats();

Functions malloc(), mallinfo(), malloc_stats() and data structures required by these functions are declared in malloc.h and defined in malloc.c. malloc.h also houses blocks which is an array of Block * (read block.h for more). blocks is responsible for maintaining the free blocks. Per thread arenas are maintained by giving individual copies of blocks to each thread. The memory required to create free nodes is obtained using sbrk and the starting address of that is stored in heap.

Functions free(), realloc() and calloc() can be found respectively in free.c, realloc.c and calloc.c.

freelist.c contains functions that are used to maintain the freelist. block.h and block.c define the structure for the blocks and provide some utility functions for blocks.

thread-test.c and test.c contain some tests for multi-threaded and single-threaded memory requests.

Allocation

Memory is allocated using Buddy Allocation. A normal memory request to the malloc function roughly looks like this.

Check if heap is empty. If so, extend heap by calling sbrk.
Calculate the level for the given memory request. For example, a memory request for 200 bytes will be turned into 256 bytes since, all the blocks are in power of 2 in this allocation scheme.
Once we know the level, we check if there are any free blocks in blocks for that level.
- If so, allocate the block and update the free list accordingly.
- If not, we find the next largest free block in blocks and keep breaking it in two until we can satisfy the memory request.
- If there are no blocks that can be broken down to satisfy the request, we ask for more heap space using the sbrk system call and satisfy the request.

Note that, the request made to the sbrk call is multiple of page size.

If the memory request is much bigger than the biggest block available, the memory is satisfied using mmap system call.

:file_folder: Hitch, a cross-platform AirDrop

2017-08-02T22:10:00+00:00

:octocat: Source code on GitHub

Hitch is a cross-platform version of AirDrop. It allows easy file-transfer among devices in the same network, just like AirDrop. Underneath, it uses ZeroConf to discover all the hitch-enabled devices.

Here’s how simple sending a file using Hitch is:

Here’s how receiving a file looks like:

Story

The inspiration for this project came from the need of AirDrop like easy file transfer app for all platforms (windows, linux, mac e.g.).

Initially, the app was built using Python only because how simple it is to do network programming in Python. But eventually, I decided to rewrite the app in Javascript using the Electron framework in favor of better UI.

:family: Conneige, an IoT device to better connect families [MIT Media Lab Hackathon 2017]

2017-04-24T22:10:00+00:00

Conneige is an IoT device designed to connect students with their families as part of MIT’s connected care hackathon at the Media Lab. We used Particle’s Photon microcontroller and Particle Cloud to connect IoT devices.

In the team of designers and business developers, my responsibilites included programming Photon microcontroller using their Node JS SDK and testing the IoT device.

Here’s the presentation: