New Python Data Analytics¶
TsTables – High Performance Times Series Management
TsTables is a Python library by Andy Fiedler built on top of the popular PyTables HDF5 database library. It is meant to handle large amounts of high frequency time series data in append once, retrieve many times scenarios (cf. Gihub page). The focus lies on retrieving chunks of data from large data sets as quickly as possible.
import numpy as np
import pandas as pd
import tables as tb
import tstables as tstb
import random
from time import time
from datetime import datetime
%matplotlib inline
Sample Time Series Data¶
Let us generate a decent amount of sample data points.
no = 30000000
co = 3
dt = 1. / (12 * 30 * 24 * 60)
vol = 0.2
We generate one second intervals of data.
dr = pd.date_range('2015-1-1', periods=no, freq='1s')
dr
In memory generation is quite quick.
%%time
da = 100 * np.exp(np.cumsum(-0.5 * vol ** 2 * dt +
vol * np.sqrt(dt) * np.random.standard_normal((no, co)), axis=0))
da[0] = 100
df = pd.DataFrame(da, index=dr, columns=['ts1', 'ts2', 'ts3'])
df.count()
The starting values of the three time series.
df.head()
And a plot of the time series data (every 100000th point).
df[::100000].plot()
Storage and Retrieval with TsTables¶
To store the time series data in a PyTables table we first define the table structure.
class TS(tb.IsDescription):
timestamp = tb.Int64Col(pos=0)
ts1 = tb.Float64Col(pos=1)
ts2 = tb.Float64Col(pos=1)
ts3 = tb.Float64Col(pos=1)
Second, open a database file and create the table object.
h5 = tb.open_file('ts.h5','w')
TsTables adds a new function create_ts to PyTables.
ts = h5.create_ts('/','TS', TS)
Third, we append the time series data to the table object.
%time ts.append(df)
ls -n *.h5
The approach of TsTables is to apply a highly structured storage hierarchy.
a = str(h5)
print a[:508]
The strength of TsTables lies in retrieving chunks of time series data defined by a start date and an end date (which obviously is a typical case in finance, e.g. in backtesting strategies or risk management).
read_start_dt = datetime(2015, 2, 1, 0, 0)
read_end_dt = datetime(2015, 2, 2, 0, 0)
TsTables tries to make such an operation as fast as possible.
%time rows = ts.read_range(read_start_dt, read_end_dt)
Let us try it with random intervals.
t0 = time()
its = 100
for _ in xrange(its):
day = random.randint(1, 27)
read_start_dt = datetime(2015, 2, day, 0, 0)
read_end_dt = datetime(2015, 2, day + 1, 0, 0)
rows = ts.read_range(read_start_dt, read_end_dt)
t1 = time()
The results are convincing.
print "time for %d random accesses %5.3f seconds" %(its, t1 - t0)
print "average time for random access %5.3f seconds" %((t1 - t0) / its)
Conveniently, the returned object is a pandas DataFrame.
rows.count()
rows.head()
A look at a data sub-set.
rows[::500].plot()
h5.close()
!rm ts.h5
bcolz – High Performance Columnar Data Store
bcolz is a columnar data store for fast data storage and retrieval with built-in high performance compression. It supports both in-memory and out-of-memory storage and operations. Cf. http://bcolz.blosc.org/.
import bcolz
ctable Example¶
The first example is based on the ctable class for data in table format. The example data set is 1 GB in size.
N = 100000 * 1000
print N
In-Memory Storage¶
We generate first an in-memory object using high compression. Since we work with integers, good compression ratios are to be expected. It takes about 24 sec to generate the ctable object from a generator via the fromiter method.
%%time
ct = bcolz.fromiter(((i, i ** 2) for i in xrange(N)),
dtype="i4, i8",
count=N,
cparams=bcolz.cparams(clevel=9))
The in-memory size is about 150 MB only, which translates in to a compression ratio of 7+.
ct
You can now implement fast numerical operations on this data object (note that the output is a carray object).
%time ct.eval('f0 ** 2 + sqrt(f1)')
Disk-Based Storage¶
The same tasks can be implemented with disk-based storage. To this end, only specify the rootdir parameter. With about 30 sec the generation takes a bit longer on disk. everything else (especially the object handling) remaining the same however.
%%time
ct = bcolz.fromiter(((i, i ** 2) for i in xrange(N)),
dtype="i4, i8",
count=N, rootdir='ct',
cparams=bcolz.cparams(clevel=9))
Everything else (especially the object handling) remains almost the same however.
ct
The numerical operations work in the same fashion and hardly take longer due to native multi threading and optimized caching.
%time ct.eval('f0 ** 2 + sqrt(f1)')
Let us finally verify system disk usage.
!ls ct
!du -h ct
# system disk usage
!rm -r ct
carray Example¶
This example is about mid data which does not fit (in general) into memory (without compression).
import numpy as np
We generte as basis a NumPy ndarray object of size 32 MB.
n = 2000
a = np.arange(n * n).reshape(n, n)
a.nbytes
In-Memory Storage¶
Let us first again work in-memory. Our carray object contains 4,000 versions of the ndarray object. The in-memory generation of the object takes about 25 sec.
%%time
it = 4000
ca = bcolz.carray(a, cparams=bcolz.cparams(clevel=9))
for i in range(it):
ca.append(a)
The carray object stores 120 GB worth of data in less than 1 GB of memory, for a compression ratio of more than 130.
ca
Let us implement the evaluation of a numerical expression on this data set. The syntax and handling are the same as with NumPy ndarray objects.
%time ca[:5000] ** 2 + np.sqrt(ca[10000:15000])
Another approach is to use the eval function of bcolz.
x = ca[:10000] # 10,000 rows as sub-set
%time bcolz.eval('x ** 2 + sqrt(x)', cparams=bcolz.cparams(clevel=9))
# output carray object compressed
Disk-Based Storage¶
Disk-based storage of multiple versions of the array object. We write the object 4,000 times to disk in a single carray object. It takes only about 1 min to compress and store 120 GB worth of data on disk.
%%time
it = 4000
ca = bcolz.carray(a, rootdir='ca',
cparams=bcolz.cparams(clevel=9))
for i in range(it):
ca.append(a)
The compression ratio in this case is again 130+.
ca
Simple numerical operations are easy to implement.
%time np.sum(ca[:1000] + ca[4000:5000])
Let us try the previous, mathematically more demanding operation – again with a sub-set of the data.
x = ca[:10000] # 10,000 rows as sub-set
First, with an in-memory carray results object.
%time bcolz.eval('x ** 2 + sqrt(x)', cparams=bcolz.cparams(9))
Second, with an on-disk results object. The time difference is not that huge.
%time bcolz.eval('x ** 2 + sqrt(x)', cparams=bcolz.cparams(9), rootdir='out')
Finally, we verify system disk usage.
!du -hs ca
# system disk usage
!du -hs out
!rm -r ca
!rm -r out
blaze – Data Blending and Analysis
blaze allows Python users a familiar interface to query data living in diverse data storage systems.
Cf. http://blaze.pydata.org/.
import blaze as bz
Simple Example¶
The first example constructs a blaze.Data object from native Python objects.
t = bz.Data([('Henry', 'boy', 8),
('Lilli', 'girl', 14)],
fields=['name', 'gender', 'age'])
t
t[t.age > 10]
Data from NumPy Array¶
Let us read data from an in-memory NumPy ndarray object.
import numpy as np
a = np.random.standard_normal((1000000, 5))
# 1mn data rows, 5 columns
df = bz.DataFrame(a, columns=['f0', 'f1', 'f2', 'f3', 'f4'])
# blaze DataFrame constructor
A look at the data structure.
df.head()
Data itself is stored as NumPy ndarray object.
df.values
Data from CSV File¶
We generate first a CSV file using the random data from before.
%time df.to_csv('data.csv', index=False)
Let us read the data with blaze. Actually, we only generate a view.
%time csv = bz.CSV('data.csv')
%time t1 = bz.Data(csv)
Now, we can work with the data. Note, however, that iterating, slicing, etc. are not (yet) implemented.
%time t1.count()
The backend is a CSV object. And a look at the first 10 rows.
t1.data
t1
Data from SQL¶
We now generate a SQLite3 table with the dummy data from before.
import sqlite3 as sq3
con = sq3.connect('data.sql')
try:
con.execute('DROP TABLE numbers')
# delete in case it exists
except:
pass
We write the data into an appropriate table.
con.execute(
'CREATE TABLE numbers (f0 real, f1 real, f2 real, f3 real, f4 real)'
)
%time con.executemany('INSERT INTO numbers VALUES (?, ?, ?, ?, ?)', a)
con.commit()
con.close()
Now reading the data with blaze (i.e. just generating a view).
%time t2 = bz.Data('sqlite:///data.sql::numbers')
The SQL backend and first 10 rows again.
t2.data
t2
Working with the blaze Objects¶
blaze provides an abstraction logic for computations/queries.
ts = bz.TableSymbol('ts',
'{f0: float64, f1: float64, f2: float64, f3: float64, f4: float64}')
# generic table description -- independent of the target data structure
expr = ts[ts['f0'] + ts['f3'] > 2.5]['f1']
# generic expression -- independent of the target data structure
The blaze compiler specializes the generic objects to different data structures.
%time np.array(bz.compute(expr, a)) # NumPy ndarray object
%time np.array(bz.compute(expr, df)) # DataFrame object
%time np.array(bz.compute(expr, csv)) # CSV file representation
In similar fashion, blaze allows unified expression evaluations for different backends (I).
%time t1[t1['f0'] + t1['f3'] > 2.5]['f1'].head()
# table representation 1
# from CSV
In similar fashion, blaze allows unified expression evaluations for different backends (II).
%time t2[t2['f0'] + t2['f3'] > 2.5]['f1'].head()
# table representation 2
# from SQL database
Typical aggregational operations work as well.
%time t1.f0.sum()
%time t2.f3.max()
Transforming Data Formats¶
If you work intensively with data sets, it might be beneficial to transform them once into highly performant binary data formats (eg bcolz, HDF5).
Using bcolz as Data Store¶
%time bz.into('data.bcolz', 'data.csv')
# natively done by blaze
# cparams=bcolz.cparams(9) could be added
# no effect here due to random floats
We can now connect to the bcolz disk-based ctable object.
import bcolz as bc
b = bc.ctable(rootdir='data.bcolz')
Now, the power of bcolz for numerical computations can be played out.
%time nex = b.eval('sqrt(abs(f0)) + log(abs(f1))')
nex
Using HDF5¶
Similarly, we can use PyTables and HDF5 as an efficient binary store.
import pandas as pd
%%time
con = sq3.connect('data.sql')
pd.HDFStore('data.h5')['sql'] = pd.read_sql('SELECT * FROM numbers', con)
# simultaneously reading whole SQL table and writing it to HDF5 store
con.close()
Now, data can be efficiently retrieved.
%%time
%matplotlib inline
pd.HDFStore('data.h5')['sql'][::1000].cumsum().plot(figsize=(10, 5))
# simultaneously reading data from HDF5 store and plotting it
Cleaning Up¶
!du -h dat*
!ls -n dat*.*
# cleaning up
!rm -r dat*.*
Conclusion¶
High performance (hardware-bound) I/O operations and highly efficient data blending and analytics are among Python's key strengths.
Python Quant Platform | http://quant-platform.com
Python for Finance | Python for Finance @ O'Reilly
Derivatives Analytics with Python | Derivatives Analytics @ Wiley Finance