Before we start you must:

• Have python 3.5 or older installed with venv installed
• At the moment a Linux-based system or a little bit of knowledge to translate the commands (I don't have a Windows machine close but give me a shout if you want me to translate the non-compatible commands)
• That's it!

Apache Parquet is a file format (a data storage format more specifically) that focus on handling data through columns (as opposite to rows like in a CSV file). It isn't the only columnar format but it is a well known and popular one. In a nutshell, this format stores information about the data allowing to perform certain operations very quickly. It also relies heavily on compression allowing for smaller file sizes.

If you come from a "normal" database or are used to work with Data Frames, it is normal to think in rows. You have a header row naming each column and then pick up the rows one by one. This structure represents very well the relationship that exists between the columns or fields. However, sometimes you want to run a query that affects heavily a particular column rather than each row.

Think for example trying to find a particular date or a range of dates in a timestamp column. In a bad scenario, you need to scan every record in the column to find what you need. Formats such as CSV don't know anything about the information they contain. Parquet will know more about your column and the data will be stored in a way that will make these type of queries perform much better.

This section is very similar to my previous post ( ). If you already followed that post, you just need to install pyarrow by activating your virtual environment and running pip install pyarrow

If you are coming to this post first, carry on reading so you can get your environment and data ready.

We are going to create a virtual environment, install Dask, pyarrow and Jupyter Notebooks (this last one just to run our code).

We will now create the main folder called parquetdask and a virtual environment called venv inside:

mkdir parquetdask
cd parquetdask
python3 -m venv venv

Now we will activate the virtual environment and install the packages we are going to need

source venv/bin/activate
pip install "dask[complete]==2.27.0" pyarrow==1.0.1 jupyter==1.0.0

Start your notebook

jupyter notebook

Then create a new notebook and copy these sections into separated sections.

First we import the libraries we are going to need and start Dask (you can read more about this in my previous post)

import time
import os
import subprocess
import dask
import dask.dataframe as dd
from dask.delayed import delayed
import time
from dask.distributed import Client, progress
client = Client(threads_per_worker=4, n_workers=4)

Let's define some variables we will need

all_data = "data/pp-complete.csv"
columns = ["transaction", "price", "date", "postcode", "prop_type", "old_new", "duration", "paon", "saon", "street", "locality", "city", "district", "county", "ppd_cat", "rec_status"]

Now, we will run some analysis using Dask directly from our csv file.

start = time.time()
df = dd.read_csv(all_data,  blocksize=32 * 1024 * 1024, header=0, names=columns)
df_by_county = df.groupby("county")["price"].sum().compute()
print(df_by_county)
end = time.time()
print("time elapsed {}".format(end-start))

The result I got was:

dask using a csv file - time elapsed 19.78 seconds
(remember from the previous post, pandas took over 50 secs)

Next, we will transform our CSV file to Parquet (it will take some time)

 df.to_parquet(all_data + ".parquet", engine='pyarrow')

Before we move on, let's have a look at the size of the two files (the parquet and the csv one in our data folder). You can use the follow command in a unix-like system or just look at the size in your file browser (note that parquet is a folder, not a file)

print("{}G".format(round(os.path.getsize(all_data)/1000000000, 1)))
print(subprocess.check_output(['du','-sh', all_data + ".parquet"]).split()[0].decode('utf-8'))

The results I got was:

4.4G
2.2G

This is already showing the power of compression implemented in Parquet.

But what about performance? let's re-run the same process again, this time reading from the Parquet version:

start = time.time()
df = dd.read_parquet(all_data + ".parquet",  blocksize=32 * 1024 * 1024, header=0, names=columns, engine='pyarrow')
df_by_county = df.groupby("county")["price"].sum().compute()
print(df_by_county)
end = time.time()
print("time elapsed {}".format(end-start))

My result I got was:

dask using a parquet file - time elapsed 13.65 seconds

That's approximately 30% faster than the csv. Not bad at all considering you also get the storage space improvement.

This article is just a very brief introduction to Apache Parquet. There's a lot more to discover and analyse.

For example, you can explore the folder created as the Parquet "file". In there you can see that the original csv file is split into lots of smaller files allowing for better parallel execution.

The file size (and number of) split is itself something you can modify to test for performance improvement. There will be a relation between the number of files and the number of Dask workers/threads running that will define the optimum performance.

You should notice also that there are metadata files. While these are not really fully human readable, they do show why this format performs so well in what we did. The file "knows" more about the data.

Finally, we are using PyArrow to handle the file but you could use Fastparquet as an alternative. Again, this is something else you can explore.

I hope you found this post interesting. As I said, this is just an introduction and there are lots more to explore!