Interaktive Datenanalyse und Visualisierung mit Python¶
Dr. Yves J. Hilpisch
The Python Quants GmbH
Pycon DE, Köln – 16. Oktober 2013
Python for Analytics¶
Corporations, decision makers and analysts nowadays generally face a number of problems with data:
- sources: data typically comes from different sources, like from the Web, from in-house databases or it is generated in-memory, e.g. for simulation purposes
- formats: data is generally available in different formats, like SQL databases/tables, Excel files, CSV files, arrays, proprietary formats
- structure: data typically comes differently structured, be it unstructured, simply indexed, hierarchically indexed, in table form, in matrix form, in multidimensional arrays
- completeness: real-world data is generally incomplete, i.e. there is missing data (e.g. along an index) or multiple series of data cannot be aligned correctly (e.g. two time series with different time indexes)
- conventions: for some types of data there a many “competing” conventions with regard to formatting, like for dates and time
- interpretation: some data sets contain information that can be easily and intelligently interpreted, like a time index, others not, like texts
- performance: reading, streamlining, aligning, analyzing – i.e. processing – (big) data sets might be slow
In addition to these data-oriented problems, there typically are organizational issues that have to be considered:
- departments: the majority of companies is organized in departments with different technologies, databases, etc., leading to “data silos”
- analytics skills: analytical and business skills in general are possessed by people working in line functions (e.g. production) or administrative functions (e.g. finance)
- technical skills: technical skills, like retrieving data from databases and visualizing them, are generally possessed by people in technology functions (e.g. development, systems operations)
This presentation focuses on
- Python as a general purpose analytics environment
- interactive analytics examples
- prototyping-like Python usage
It does not address such important issues like
- architectural issues regarding hardware and software
- development processes, testing, documentation and production
- real world problem modeling
Some Python Fundamentals¶
Fundamental Python Libraries¶
A fundamental Python stack for interactive data analytics and visualization should at least contain the following libraries tools:
- Python – the Python interpreter itself
- NumPy – high performance, flexible array structures and operations
- SciPy – collection of scientific modules and functions (e.g. for regression, optimization, integration)
- pandas – time series and panel data analysis and I/O
- PyTables – hierarchical, high performance database (e.g. for out-of-memory analytics)
- matplotlib – 2d and 3d visualization
- IPython – interactive data analytics, visualization, publishing
It is best to use either the Python distribution Anaconda or the Web-based analytics environment Wakari. Both provide almost "complete" Python environments.
For example, pandas can, among others, help with the following data-related problems:
- sources: pandas reads data directly from different data sources such as SQL databases, Excel files or JSON based APIs
- formats: pandas can process input data in different formats like CSV files or Excel files; it can also generate output in different formats like CSV, XLS, HTML or JSON
- structure: pandas' strength lies in structured data formats, like time series and panel data
- completeness: pandas automatically deals with missing data in most circumstances, e.g. computing sums even if there are a few or many “not a number”, i.e. missing, values
- conventions/interpretation: for example, pandas can interpret and convert different date-time formats to Python datetime objects and/or timestamps
- performance: the majority of pandas classes, methods and functions is implemented in a performance-oriented fashion making heavy use of the Python/C compiler Cython
First Interactive Steps with Python¶
As a simple example let's generate a NumPy array with five sets of 1000 (pseudo-)random numbers each.
import numpy as np # this imports the NumPy library
data = np.random.standard_normal((5, 1000)) # generate 5 sets with 1000 rn each
data[:, :5].round(3) # print first five values of each set rounded to 3 digits
Let's plot a histogram of the 1st, 2nd and 3rd data set.
import matplotlib.pyplot as plt # this imports matplotlib.pyplot
plt.hist([data[0], data[1], data[2]], label=['Set 0', 'Set 1', 'Set 2'])
plt.grid(True) # grid for better readability
plt.legend()
We then want to plot the 'running' cumulative sum of each set.
plt.figure() # initialize figure object
plt.grid(True)
for data_set in enumerate(data): # iterate over all rows
plt.plot(data_set[1].cumsum(), label='Set %s' % data_set[0])
# plot the running cumulative sums for each row
plt.legend(loc=0) # write legend with labels
Some fundamental statistics from our data sets.
data.mean(axis=1) # average value of the 5 sets
data.std(axis=1) # standard deviation of the 5 sets
np.corrcoef(data) # correltion matrix of the 5 data sets
General I/O Capabilities of Python¶
SQL Databases¶
Currently, and in the foreseeable future as well, most corporate data is stored in SQL databases. We take this as a starting point and generate a SQL table with dummy data.
filename = 'numbers'
import sqlite3 as sq3 # imports the SQLite library
# Remove files if they exist
import os
try:
os.remove(filename + '.db')
os.remove(filename + '.h5')
os.remove(filename + '.csv')
os.remove(filename + '.xls')
except:
pass
We create a table and populate the table with the dummy data.
query = 'CREATE TABLE numbs (no1 real, no2 real, no3 real, no4 real, no5 real)'
con = sq3.connect(filename + '.db')
con.execute(query)
con.commit()
# Writing Random Numbers to SQLite3 Database
rows = 5000000 # 5,000,000 rows for the SQL table (ca. 250 MB in size)
nr = np.random.standard_normal((rows, 5))
%time con.executemany('INSERT INTO numbs VALUES(?, ?, ?, ?, ?)', nr)
con.commit()
We can now read the data into a NumPy array.
# Reading a couple of rows from SQLite3 database table
array(con.execute('SELECT * FROM numbs').fetchmany(3)).round(3)
# Reading the whole table into an NumPy array
%time result = array(con.execute('SELECT * FROM numbs').fetchall())
Reading and Writing Data with pandas¶
pandas is optimized both for convenience and performance when it comes to analytics and I/O operations.
import pandas as pd
import pandas.io.sql as pds
%time data = pds.read_frame('SELECT * FROM numbs', con)
con.close()
data.head()
The plotting of larger data sets often is a problem. This can easily be accomplished here by only plotting every 500th data row, for instance.
%time data[::500].cumsum().plot(grid=True, legend=True) # every 500th row is plotted
There are some reasons why you may want to write the DataFrame to disk:
- Database Performance: you do not want to query the SQL database everytime anew
- IO Speed: you want to increase data/information retrievel
- Data Structure: maybe you want to store additional data with the original data set or you have generated a DataFrame out of multiple sources
pandas is tightly integrated with PyTables, a library that implements the HDF5 standard for Python. It makes data storage and I/O operations not only convenient but quite fast as well.
h5 = pd.HDFStore(filename + '.h5', 'w')
%time h5['data'] = data # writes the DataFrame to HDF5 database
Now, we can read the data from the HDF5 file into another DataFrame object. Performance of such an operation generally is quite high.
%time data_new = h5['data']
%time data.sum() - data_new.sum() # checking if the DataFrames are the same
The data is easily exported to different standard data file formats like comma separated value files (CSV).
%time data.to_csv(filename + '.csv') # writes the whole data set as CSV file
csv_file = open(filename + '.csv')
for i in range(5):
print csv_file.readline(),
pandas also interacts easily with Microsoft Excel spreadsheet files (XLS/XLSX).
%time data.ix[:65000].to_excel(filename + '.xls') # first 65,000 rows written in a Excel file
With pandas you can in one step read data from an Excel file and e.g. plot it.
%time pd.read_excel(filename + '.xls', 'sheet1').hist(grid=True)
One can also generate HTML code, e.g. for automatic Web-based report generation.
html_code = data.ix[:2].to_html()
html_code
from IPython.core.display import HTML
HTML(html_code)
Some final checks and data cleaning.
ll numb*
Parallel Execution of Option Pricing Code¶
The algorithm we (arbitrarily) choose is the Monte Carlo estimator for a European option value in the Black-Scholes-Merton set-up.
\[dS_t = r S_t dt + \sigma S_t dZ_t\]
def BSM_MCS(S0):
''' Black-Scholes-Merton MCS Estimator for European Calls. '''
import numpy as np
K = 100.; T = 1.0; r = 0.05; vola = 0.2
M = 50; I = 20000
dt = T / M
rand = np.random.standard_normal((M + 1, I))
S = np.zeros((M + 1, I)); S[0] = S0
for t in range(1, M + 1):
S[t] = S[t-1] * np.exp((r - 0.5 * vola ** 2) * dt + vola * np.sqrt(dt) * rand[t])
BSM_MCS_Value = np.sum(np.maximum(S[-1] - K, 0)) / I
return BSM_MCS_Value
The Sequential Calculation¶
The benchmark case of sequential execution.
from time import time
n = 100
def seqValue():
optionValues = []
for S in linspace(80, 100, n):
optionValues.append(BSM_MCS(S))
return optionValues
t0 = time()
seqValue()
t1 = time(); d1 = t1 - t0
print "Number of Options %8d" % n
print "Duration in Seconds %8.3f" % d1
print "Options per Second %8.3f" % (n / d1)
The Parallel Calculation¶
First, start a local cluster, if you have multiple cores in your machine.
# in the shell ...
####
# ipcluster start -n 4
####
# or maybe more cores
Enable parallel computing capabilities.
from IPython.parallel import Client
cluster_profile = "default"
# the profile is set to the default one
# see IPython docs for further details
c = Client(profile=cluster_profile)
view = c.load_balanced_view()
The function for the parallel execution of a number of different options valuations.
def parValue():
optionValues = []
for S in linspace(80, 100, n):
value = view.apply_async(BSM_MCS, S)
# asynchronously calculate the option values
optionValues.append(value)
return optionValues
def execution():
optionValues = parValue() # calculate all values
print "Submitted tasks %8d" % len(optionValues)
c.wait(optionValues)
return optionValues
t0 = time()
optionValues = execution()
t1 = time(); d2 = t1 - t0
print "Duration in Seconds %8.3f" % d2
print "Options per Second %8.3f" % (n / d2)
print "Speed-up Factor %8.3f" % (d1 / d2)
Analyzing Financial Time Series Data¶
Historical Correlation between EURO STOXX 50 and VSTOXX¶
It is a stylized fact that stock indexes and related volatility indexes are highly negatively correlated. The following example analyzes this stylized fact based on the EURO STOXX 50 stock index and the VSTOXX volatility index using Ordinary Least-Squares regession (OLS).
First, we collect historical data for both the EURO STOXX 50 stock and the VSTOXX volatility index.
import pandas as pd
from urllib import urlretrieve
es_url = 'http://www.stoxx.com/download/historical_values/hbrbcpe.txt'
vs_url = 'http://www.stoxx.com/download/historical_values/h_vstoxx.txt'
urlretrieve(es_url, 'es.txt')
urlretrieve(vs_url, 'vs.txt')
The EURO STOXX 50 data is not yet in the right format. Some house cleaning is necessary.
lines = open('es.txt').readlines() # reads the whole file line-by-line
new_file = open('es50.txt', 'w') # opens a new file
new_file.writelines('date' + lines[3][:-1].replace(' ', '') + ';DEL' + lines[3][-1])
# writes the corrected third line of the orginal file as first line of new file
new_file.writelines(lines[4:-1]) # writes the remaining lines of the orginial file
print list(open('es50.txt'))[:5] # opens the new file for inspection
Now, the data can be safely read into a DataFrame object.
es = pd.read_csv('es50.txt', index_col=0, parse_dates=True, sep=';', dayfirst=True)
del es['DEL'] # delete the helper column
es
The VSTOXX data can be read without touching the raw data.
vs = pd.read_csv('vs.txt', index_col=0, header=2, parse_dates=True, sep=',', dayfirst=True)
# you can alternatively read from the Web source directly
# without saving the csv file to disk:
# vs = pd.read_csv(vs_url, index_col=0, header=2,
# parse_dates=True, sep=',', dayfirst=True)
We now merge the data for further analysis.
from datetime import datetime
data = pd.DataFrame({'EUROSTOXX' :
es['SX5E'][es.index > datetime(1999, 12, 31)]})
data = data.join(pd.DataFrame({'VSTOXX' :
vs['V2TX'][vs.index > datetime(1999, 12, 31)]}))
data
Let's inspect the two time series.
data.head()
A picture can tell the complete story.
data.plot(subplots=True, grid=True, style='b')
We now generate log returns for both time series.
rets = np.log(data / data.shift(1))
rets.head()
To this new data set, also stored in a DataFrame object, we apply OLS.
xdat = rets['EUROSTOXX']
ydat = rets['VSTOXX']
model = pd.ols(y=ydat, x=xdat)
model
Again, we want to see how our results look graphically.
plt.plot(xdat, ydat, 'r.')
ax = plt.axis() # grab axis values
x = np.linspace(ax[0], ax[1] + 0.01)
plt.plot(x, model.beta[1] + model.beta[0] * x, 'b', lw=2)
plt.grid(True)
plt.axis('tight')
Analyzing High Frequency Data¶
Using pandas, the code that follows reads intraday, high-frequency data from a Web source, plots it and resamples it.
date = datetime.now()
url = 'http://hopey.netfonds.no/posdump.php?date=%s%s%s&paper=AAPL.O&csv_format=csv' % ('2013', '10', '07')
# you may have to adjust the date since only recent dates are available
urlretrieve(url, 'aapl.csv')
AAPL = pd.read_csv('aapl.csv', index_col=0, header=0, parse_dates=True)
AAPL
The intraday evolution of the Apple stock price.
AAPL['bid'].plot()
A resampling of the data is easily accomplished with pandas.
import datetime as dt
# this resamples the record frequency to 5 minutes, using mean as aggregation rule
AAPL_5min = AAPL.resample(rule='5min', how='mean')
AAPL_5min.head()
Let's have a graphical look at the new data set.
AAPL_5min['bid'].plot()
Obvioulsly, there hasn't been trading data for every interval.
pandas has a number of approaches to fill in missing data.
AAPL_5min['bid'].fillna(method='ffill').plot() # this forward fills missing data
Analyzing Non-Financial Real-World Data¶
The Example and First Analyses¶
This example analyzes the frequency of bikes on 7 different bike paths in Montreal.
The data is from the Web site Bike Data.
Thanks to Julia Evans for providing this example (cf. Julia's Notebook).
After having downloaded and extracted the data zip file from the source, we can read the data for 2012 with pandas into a DataFrame.
bikes = pd.read_csv('./Data/2012.csv', encoding='latin1', sep=';', index_col='Date', parse_dates=True, dayfirst=True)
bikes.head()
We want to get rid of those columns without data.
bikes = bikes.dropna(axis=1) # delete empty ('NaN') columns
bikes # the DataFrame object
Let's see how the first three different paths are used by bikers.
rcParams['figure.figsize'] = (7, 7) # enlarge figure size
bikes[bikes.columns[:3]].plot(subplots=True, grid=True, color='b')
# plot the first three columns of the DataFrame
pandas provides a number of convenience functions/methods to generate meta data for DataFrame objects.
np.round(bikes.describe(), 1)
Which one is overall the most popular bike path?
bikes.sum().plot(kind='barh', grid=True)
Grouping and Aggregation¶
You often have to group data sets and aggregate data values accordingly. In this case, it is natural to ask for the different frequencies of bikes for different weekdays. pandas can handle such tasks really well. First, we add a weekday column to the DataFrame object.
bikes['weekday'] = bikes.index.weekday
bikes.tail()
Now, we can use this column for grouping operations.
day_grouping = bikes.groupby('weekday').mean()
day_grouping.index = ['Mon', 'Tue', 'Wed', 'Thu', 'Fri', 'Sat', 'Sun']
np.round(day_grouping)
Which is the most popular day for biking? We generate a stacked bar plot to derive the answer graphically.
day_grouping.plot(kind='bar', stacked=True, legend=False)
Obviously, Thursday wins the race.
Combining Two Data Sets¶
We next want to see which role the weather plays in explaining biker frequency in each month. To collect weather data we use a function provided in Julia Evans' Notebook.
# cf. http://nbviewer.ipython.org/urls/raw.github.com/jvns/talks/master/pyconca2013/pistes-cyclables.ipynb
def get_weather_data(year):
url1 = "http://climate.weather.gc.ca/climateData/"
url2 = "bulkdata_e.html?format=csv&stationID=5415"
url3 = "&Year={year}&Month={month}&timeframe=1&submit=Download+Data"
url_template = url1 + url2 + url3
# mctavish station: 10761, airport station: 5415
data_by_month = []
for month in range(1, 13):
url = url_template.format(year=year, month=month)
weather_data = pd.read_csv(url, skiprows=16, index_col='Date/Time',
parse_dates=True).dropna(axis=1)
weather_data.columns = map(lambda x: x.replace('\xb0', ''),
weather_data.columns)
weather_data = weather_data.drop(['Year', 'Day', 'Month', 'Time',
'Data Quality'], axis=1)
data_by_month.append(weather_data.dropna())
# Concatenate and drop any empty columns
return pd.concat(data_by_month).dropna(axis=1, how='all').dropna()
Let's retrieve the weather data for 2012.
try:
h5 = pd.HDFStore('./data/weather.h5', 'r')
weather = h5['weather']
h5.close()
except:
weather = get_weather_data(2012)
weather.head()
Now, we can resample the time series data sets to look for correlation between e.g. the average temperature and the average bike frequency.
interval = '1m' # resampling period, e.g. 1 month
bikes_int = bikes.resample(rule=interval, how='sum') # bike frequency per interval
del bikes_int['weekday']
bikes_int.tail()
We do the same with the temperature data.
temp_int = weather['Temp (C)'].resample(rule=interval, how='mean')
np.round(temp_int.tail(), 1)
for col in bikes_int.columns:
cor = bikes_int[col].corr(temp_int)
print "Correlation av. %22s path bike frequency and av. temp. %4.3f" % (col, cor)
We briefly check graphically if we can trust our numerical results.
new_df = pd.DataFrame({'Bikes': bikes_int['Berri 1'] , 'Temp': temp_int})
new_df = (new_df - new_df.mean()) / new_df.mean() * 100
new_df.plot(grid=True)
Why Python for Data Analytics & Visualization?¶
10 years ago, Python was considered exotic in the analytics space – at best. Languages/packages like R and Matlab dominated the scene. Today, Python has become a major force in data analytics & visualization due to a number of characteristics:
- multi-purpose: prototyping, development, production, sytems administration – Python is one for all
- libraries: there is a library for almost any task or problem you face
- efficiency: Python speeds up all IT development tasks for analytics applications and reduces maintenance costs
- performance: Python has evolved from a scripting language to a 'meta' language with bridges to all high performance environments (e.g. LLVM, multi-core CPUs, GPUs, clusters)
- interoperalbility: Python seamlessly integrates with almost any other language and technology
- interactivity: Python allows domain experts to get closer to their business and financial data pools and to do real-time analytics
- collaboration: solutions like Wakari with IPython Notebook allow the easy sharing of code, data, results, graphics, etc.
One of the easiest ways to deploy Python today across a whole organization with a heterogenous IT infrastructure is via Wakari, Continuum's Web-/Browser- and Python-based Data Analytics environment. It is availble both as a cloud as well as an enterprise solution for in-house deployment.
The Python Quants GmbH – Python Data Exploration & Visualization¶
The Python Quants – the company Web site
Dr. Yves J. Hilpisch – my personal Web site
Derivatives Analytics with Python – my new book
Read an Excerpt and Order the Book
Contact Us