Eikon Data API¶
Cross-Asset Financial Analytics — Discovering Statistical Inefficiencies via Deep Learning
Dr. Yves J. Hilpisch | The Python Quants GmbH
http://tpq.io | @dyjh | training@tpq.io
The Agenda¶
This tutorial shows
- how to retrieve historical data across asset classes via the Eikon Data API,
- how to work with such data using
pandas,PlotlyandCufflinksand - how to discover stastical inefficiencies in financial time series with machine learning.
Importing Required Packages¶
In [1]:
import time
import eikon as ek # the Eikon package
import numpy as np # NumPy
import pandas as pd # pandas
import cufflinks as cf # Cufflinks
import configparser as cp
The following Python and package versions are used.
In [2]:
import sys
print(sys.version)
In [3]:
ek.__version__
Out[3]:
In [4]:
np.__version__
Out[4]:
In [5]:
pd.__version__
Out[5]:
In [6]:
cf.__version__
Out[6]:
Connecting to Eikon Data API¶
This code sets the app_id to connect to the Eikon Data API Proxy which needs to be running locally.
In [7]:
cfg = cp.ConfigParser()
cfg.read('eikon.cfg')
Out[7]:
In [8]:
# ek.set_app_key(cfg['eikon']['app_id'])
Retrieving Cross-Asset Data¶
We first define a small universe of RICS for which to retrieve data.
In [9]:
rics = [
'GE', # General Electric stock
'AAPL.O', # Apple stock
'.SPX', # S&P 500 stock index
'.VIX', # VIX volatility index
'EUR=', # EUR/USD exchange rate
'XAU=', # Gold price
'GLD', # Gold ETF
'BTC=', # Bitcoin in USD
]
Second, end-of-day (EOD) data is retrieved.
In [10]:
fn = 'eikon_eod_data.csv'
try:
data = pd.read_csv(fn, index_col=0, parse_dates=True)
except:
first = True
for ric in rics:
print(ric)
d = ek.get_timeseries(ric, # the RIC
fields='CLOSE', # the required fields
start_date='2015-01-01', # start date
end_date='2019-09-30') # end date
if first:
data = d
data.columns = [ric]
first = False
else:
data[ric] = d
time.sleep(2)
data.to_csv(fn)
In [11]:
data.head() # first five rows
Out[11]:
In [12]:
data.tail() # final five rows
Out[12]:
In [13]:
data.info() # DataFrame meta information
Preparing Features¶
To discover statistical inefficiencies we work with different features extracted from the time series data.
In [14]:
def add_lags(data, ric, lags):
cols = []
df = pd.DataFrame(data[ric])
df['r'] = np.log(df / df.shift())
df['sma'] = df[ric].rolling(20).mean()
df['min'] = df[ric].rolling(20).min()
df['max'] = df[ric].rolling(20).max()
df['mom'] = df['r'].rolling(20).mean()
df['vol'] = df['r'].rolling(20).std()
df.dropna(inplace=True)
df['d'] = np.where(df['r'] > 0, 1, 0)
features = [ric, 'r', 'd', 'sma', 'min', 'max', 'mom', 'vol']
for f in features:
for lag in range(1, lags + 1):
col = f'{f}_lag_{lag}' # defines the column name
df[col] = df[f].shift(lag) # creates the lagged data column
cols.append(col) # stores the column name
df.dropna(inplace=True) # gets rid of incomplete data rows
return df, cols
Second, the iterations over all RICs, using the add_lags function and storing the resulting DataFrame objects in a dictionary.
In [15]:
lags = 7 # historical lags
In [16]:
dfs = {}
for ric in rics:
print(ric)
df, cols = add_lags(data, ric, lags)
dfs[ric] = df.dropna(), cols
In [17]:
dfs.keys() # the keys of the dictonary
Out[17]:
In [18]:
# dfs['AAPL.O'].head(7)
Machine Learning¶
The matrix consisting of the lagged data columns is used to "predict" the next day's value of the RIC via Machine Learning.
In [19]:
from sklearn.naive_bayes import GaussianNB
from sklearn.linear_model import LogisticRegression
from sklearn.tree import DecisionTreeClassifier
from sklearn.ensemble import RandomForestClassifier
from sklearn.ensemble import BaggingClassifier
from sklearn.neural_network import MLPClassifier
from sklearn.metrics import accuracy_score
In [20]:
np.set_printoptions(precision=4, suppress=True)
In [21]:
def create_model(algo=3, n_estimators=75):
base_estimators = [
GaussianNB(),
LogisticRegression(C=1),
DecisionTreeClassifier(random_state=100, max_depth=3,
min_samples_leaf=12),
MLPClassifier(hidden_layer_sizes=2 * [128,],
early_stopping=True, shuffle=False,
random_state=100,
activation='relu',
max_iter=250),
RandomForestClassifier(n_estimators=5, max_depth=3,
min_samples_leaf=8, random_state=100)
]
model = BaggingClassifier(base_estimator=base_estimators[4],
n_estimators=n_estimators,
bootstrap=True,
max_features=0.75,
max_samples=0.75,
n_jobs=4,
random_state=100
)
return model
First, training and prediction in-sample only.
In [22]:
%%time
for ric in rics:
model = create_model(n_estimators=10)
df, cols = dfs[ric] # getting data for the RIC
model.fit(df[cols], df['d']) # the fitting step
pred = model.predict(df[cols]) # the prediction step
acc = accuracy_score(df['d'], pred) # prediction accuracy
print(f'IN-SAMPLE | {ric:7s} | acc={acc:.4f}')
Second, training in-sample and prediction out-of-sample.
In [23]:
split = int(len(dfs[ric][0]) * 0.7)
In [24]:
%%time
for ric in rics:
model = create_model()
df, cols = dfs[ric] # getting data for the RIC
train = df.iloc[:split] # training data set
mu, std = train[cols].mean(), train[cols].std()
train[cols] = (train[cols] - mu) / std # normalization
model.fit(train[cols], train['d']) # the fitting step
test = df.iloc[split:] # test data set
test[cols] = (test[cols] - mu) / std # normalization
pred = model.predict(test[cols]) # the prediction step
acc = accuracy_score(test['d'], pred) # prediction accuracy
print(f'OUT-OF-SAMPLE | {ric:7s} | acc={acc:.4f}')
Analyzing Intraday Data¶
Let us quickly check, whether the results are similar on an intraday basis.
In [25]:
fn = 'eikon_id_data.csv'
try:
data = pd.read_csv(fn, index_col=0, parse_dates=True)
except:
first = True
for ric in rics:
print(ric)
d = ek.get_timeseries(ric, # the RIC
fields='CLOSE', # the required fields
start_date='2019-10-07', # start date
end_date='2019-10-12', # end date
interval='minute')
if first:
data = d
data.columns = [ric]
first = False
else:
data[ric] = d
time.sleep(2)
data.to_csv(fn)
In [26]:
data.dropna(inplace=True)
In [27]:
data.info()
In [28]:
data.tail()
Out[28]:
In [29]:
dfs = {}
for ric in rics:
df, cols = add_lags(data, ric, lags)
dfs[ric] = df.dropna(), cols
In [30]:
split = int(len(dfs[ric][0]) * 0.8)
In [31]:
%%time
for ric in rics:
model = create_model(n_estimators=75)
df, cols = dfs[ric] # getting data for the RIC
train = df.iloc[:split] # training data set
mu, std = train[cols].mean(), train[cols].std()
train[cols] = (train[cols] - mu) / std # normalization
model.fit(train[cols], train['d']) # the fitting step
test = df.iloc[split:] # test data set
test[cols] = (test[cols] - mu) / std # normalization
pred = model.predict(test[cols]) # the prediction step
acc = accuracy_score(test['d'], pred) # prediction accuracy
print(f'OUT-OF-SAMPLE | {ric:7s} | acc={acc:.4f}')
Conclusions¶
Based on this tutorial, we can conclude that
- it is easy to retrieve historical end-of-day and intraday data across asset classes via the Eikon Data API,
PlotlyandCufflinksmake financial data visualization convenient and- there is some evidence for statistical inefficiencies based on a machine learning-based prediction approach.
Eikon Data API Developer Resources¶
Data Item Browser Application: Type DIB into Eikon Search Bar.