Dr. Yves J. Hilpisch
The Python Quants GmbH
Risk Disclaimer¶
Trading forex/CFDs on margin carries a high level of risk and may not be suitable for all investors as you could sustain losses in excess of deposits. Leverage can work against you. Due to the certain restrictions imposed by the local law and regulation, German resident retail client(s) could sustain a total loss of deposited funds but are not subject to subsequent payment obligations beyond the deposited funds. Be aware and fully understand all risks associated with the market and trading. Prior to trading any products, carefully consider your financial situation and experience level. Any opinions, news, research, analyses, prices, or other information is provided as general market commentary, and does not constitute investment advice. FXCM & TPQ will not accept liability for any loss or damage, including without limitation to, any loss of profit, which may arise directly or indirectly from use of or reliance on such information.
Reading FXCM Tick Data¶
In [1]:
import pandas as pd
import datetime as dt
import cufflinks as cf # Cufflinks
cf.set_config_file(offline=True) # set the plotting mode to offline
The Tick Reader Class¶
In [2]:
from fxcm_tick_reader import fxcm_tick_reader
Available Symbols¶
In [3]:
fxcm_tick_reader.get_available_symbols()
Out[3]:
Retrieving Tick Data¶
In [4]:
start = dt.datetime(2018, 2, 1)
stop = dt.datetime(2018, 2, 2)
In [5]:
%time td = fxcm_tick_reader('EURUSD', start, stop)
In [6]:
type(td)
Out[6]:
In [7]:
td.get_raw_data().info()
In [8]:
td.get_raw_data().tail(10)
Out[8]:
Working with the Tick Data¶
In [9]:
%time td.get_data().info()
In [10]:
%time td.get_data().info()
In [11]:
%%time
data = td.get_data(start='2018-02-01 08:00:00', end='2018-02-01 16:00:00')
data.info()
In [12]:
data['Bid'].iplot()
Connecting to the FXCM RESTful API¶
In [13]:
import fxcm
In [14]:
api = fxcm.fxcm(config_file='fxcm.cfg')
In [15]:
instruments = api.get_instruments()
print(instruments)
Historical Data¶
Retrieving Historical Data¶
In [16]:
candles = api.get_candles('USD/JPY', period='D1', number=10)
In [17]:
# 10 most recent days | daily
candles
Out[17]:
In [18]:
start = dt.datetime(2017, 1, 1)
end = dt.datetime(2018, 1, 1)
In [19]:
candles = api.get_candles('USD/JPY', period='D1',
start=start, stop=end)
In [20]:
candles['askclose'].iplot()
The parameter period must be one of m1, m5, m15, m30, H1, H2, H3, H4, H6, H8, D1, W1 or M1.
In [21]:
# 50 most recent 1 minute bars
candles = api.get_candles('EUR/USD', period='m1', number=50)
candles.tail(10)
Out[21]:
Visualization of the Data¶
In [22]:
data = candles[['askopen', 'askhigh', 'asklow', 'askclose']]
data.columns = ['open', 'high', 'low', 'close']
data.info()
In [23]:
qf = cf.QuantFig(data, title='EUR/USD', legend='top',
name='EUR/USD', datalegend=False)
In [24]:
qf.iplot()
In [25]:
qf.add_bollinger_bands(periods=10, boll_std=2, colors=['magenta', 'grey'], fill=True)
qf.data.update()
In [26]:
qf.iplot()
Using Machine Learning for Market Prediction¶
The following example is simplified and for illustration purposes only. Among others, it does not consider transactions costs or bid-ask spreads.
In [27]:
import datetime
import numpy as np
import pandas as pd
Data Retrieval¶
In [28]:
candles = api.get_candles('EUR/USD', period='m5',
start=dt.datetime(2018, 2, 7),
stop=dt.datetime(2018, 2, 9))
In [29]:
data = pd.DataFrame(candles[['askclose', 'bidclose']].mean(axis=1),
columns=['midclose'])
In [30]:
data.info()
In [31]:
data.tail()
Out[31]:
In [32]:
data.iplot()
Feature Preparation¶
In [33]:
data['returns'] = np.log(data / data.shift(1))
In [34]:
lags = 5
cols = []
for lag in range(1, lags + 1):
col = 'lag_%s' % lag
data[col] = data['returns'].shift(lag)
cols.append(col)
In [35]:
col = 'momentum'
data[col] = data['returns'].rolling(5).mean().shift(1)
cols.append(col)
In [36]:
cols
Out[36]:
In [37]:
from pylab import plt
plt.style.use('seaborn')
%matplotlib inline
In [38]:
data['direction'] = np.sign(data['returns'])
to_plot = ['midclose', 'returns', 'direction']
data[to_plot].iloc[:100].plot(figsize=(10, 6),
subplots=True, style=['-', '-', 'ro'], title='EUR/USD');
In [39]:
# the "patterns" = 2 ** lags
np.digitize(data[cols], bins=[0])[:10]
Out[39]:
In [40]:
2 ** len(cols)
Out[40]:
In [41]:
data.dropna(inplace=True)
Support Vector Machines¶
In [42]:
from sklearn import svm
In [43]:
model = svm.SVC(C=100)
In [44]:
data.info()
In [45]:
%time model.fit(np.sign(data[cols]), np.sign(data['returns']))
Out[45]:
Predicting Market Direction¶
In [46]:
pred = model.predict(np.sign(data[cols]))
pred[:15]
Out[46]:
Vectorized Backtesting¶
In [47]:
data['position'] = pred
In [48]:
data['strategy'] = data['position'] * data['returns']
In [49]:
# unleveraged | no bid-ask spread or transaction costs | only in-sample
data[['returns', 'strategy']].cumsum().apply(np.exp).iplot()
In [50]:
data['position'].value_counts()
Out[50]:
Train Test Split¶
In [51]:
from sklearn.metrics import accuracy_score
from sklearn.model_selection import train_test_split
Split Feature Sets¶
In [52]:
mu = data['returns'].mean()
v = data['returns'].std()
bins = [mu - v, mu, mu + v]
train_x, test_x, train_y, test_y = train_test_split(
data[cols].apply(lambda x: np.digitize(x, bins=bins)),
np.sign(data['returns']),
test_size=0.50, random_state=111)
In [53]:
train_x.sort_index(inplace=True)
train_y.sort_index(inplace=True)
test_x.sort_index(inplace=True)
test_y.sort_index(inplace=True)
In [54]:
# the patterns = buckets ** lags
train_x.head(5)
Out[54]:
In [55]:
test_x.head(5)
Out[55]:
In [56]:
4 ** len(cols)
Out[56]:
In [57]:
ax = data['midclose'][train_x.index].plot(style=['b.'], figsize=(10, 6))
data['midclose'][test_x.index].plot(style=['r.'], ax=ax)
data['midclose'].plot(ax=ax, lw=0.5, style=['k--']);
In [58]:
ax = data['midclose'].iloc[-100:][train_x.index].plot(style=['bo'], figsize=(10, 6))
data['midclose'].iloc[-100:][test_x.index].plot(style=['ro'], ax=ax)
data['midclose'].iloc[-100:].plot(ax=ax, lw=0.5, style=['k--']);
Model Fitting & Prediction¶
In [59]:
model.fit(train_x, train_y)
Out[59]:
In [60]:
train_pred = model.predict(train_x)
In [61]:
accuracy_score(train_y, train_pred)
Out[61]:
In [62]:
test_pred = model.predict(test_x)
In [63]:
accuracy_score(test_y, test_pred)
Out[63]:
Vectorized Backtesting — Direct Predictions
In [64]:
pred = model.predict(np.digitize(data[cols], bins=bins))
pred[:15]
Out[64]:
In [65]:
data['position'] = pred
In [66]:
data['strategy'] = data['position'] * data['returns']
In [67]:
# in-sample | unleveraged | no bid-ask spread or transaction costs
data.loc[train_x.index][['returns', 'strategy']].cumsum().apply(np.exp).iplot()
In [68]:
# out-of-sample | unleveraged | no bid-ask spread or transaction costs
data.loc[test_x.index][['returns', 'strategy']].cumsum().apply(np.exp).iplot()
In [69]:
# number of trades
sum(data['position'].diff() != 0)
Out[69]:
Retrieving Streaming Data¶
In [70]:
def output(data, dataframe):
print('%3d | %s | %s | %6.4f, %6.4f, %6.4f, %6.4f'
% (len(dataframe), data['Symbol'],
pd.to_datetime(int(data['Updated']), unit='ms'),
data['Rates'][0], data['Rates'][1],
data['Rates'][2], data['Rates'][3]))
In [71]:
api.subscribe_market_data('EUR/USD', (output,))
In [72]:
api.get_last_price('EUR/USD')
Out[72]:
In [73]:
api.unsubscribe_market_data('EUR/USD')
Placing Orders via the RESTful API¶
In [74]:
api.get_open_positions()
Out[74]:
In [75]:
order = api.create_market_buy_order('EUR/USD', 100)
In [76]:
order.get_currency()
Out[76]:
In [77]:
order.get_isBuy()
Out[77]:
In [78]:
cols = ['tradeId', 'amountK', 'currency', 'grossPL', 'isBuy']
In [79]:
api.get_open_positions()[cols]
Out[79]:
In [80]:
order = api.create_market_buy_order('USD/JPY', 50)
In [81]:
api.get_open_positions()[cols]
Out[81]:
In [82]:
order = api.create_market_sell_order('EUR/USD', 25)
In [83]:
order = api.create_market_buy_order('USD/JPY', 50)
In [84]:
api.get_open_positions()[cols]
Out[84]:
In [85]:
order = api.create_market_sell_order('EUR/USD', 50)
In [86]:
api.get_open_positions()[cols]
Out[86]:
In [87]:
api.close_all_for_symbol('USD/JPY')
In [88]:
api.get_open_positions()[cols]
Out[88]:
In [89]:
api.close_all()
In [90]:
api.get_open_positions()
Out[90]:
A Time-Based, Automated Algorithm¶
In [91]:
import time
In [92]:
for _ in range(5):
print(50 * '=')
print('TRADE NO {}'.format(_))
order = api.create_market_buy_order('EUR/USD', 100)
print('POSITIONS\n', api.get_open_positions()[cols])
time.sleep(7)
api.close_all_for_symbol('EUR/USD')
print('POSITIONS\n', api.get_open_positions())
time.sleep(7)