The Python Quants

FXCM Algorithmic Trading Initiative

Algo Trading: REST API & Python Wrapper

Dr. Yves J. Hilpisch

The Python Quants GmbH

Risk Disclaimer

Trading forex/CFDs on margin carries a high level of risk and may not be suitable for all investors as you could sustain losses in excess of deposits. Leverage can work against you. Due to the certain restrictions imposed by the local law and regulation, German resident retail client(s) could sustain a total loss of deposited funds but are not subject to subsequent payment obligations beyond the deposited funds. Be aware and fully understand all risks associated with the market and trading. Prior to trading any products, carefully consider your financial situation and experience level. Any opinions, news, research, analyses, prices, or other information is provided as general market commentary, and does not constitute investment advice. FXCM & TPQ will not accept liability for any loss or damage, including without limitation to, any loss of profit, which may arise directly or indirectly from use of or reliance on such information.

Speaker Disclaimer

The speaker is neither an employee, agent nor representative of FXCM and is therefore acting independently. The opinions given are their own, constitute general market commentary, and do not constitute the opinion or advice of FXCM or any form of personal or investment advice. FXCM assumes no responsibility for any loss or damage, including but not limited to, any loss or gain arising out of the direct or indirect use of this or any other content. Trading forex/CFDs on margin carries a high level of risk and may not be suitable for all investors as you could sustain losses in excess of deposits.

Some Basic Imports

In [1]:
import time
import numpy as np
import pandas as pd
import datetime as dt
import cufflinks as cf
from pylab import plt
cf.set_config_file(offline=True)
plt.style.use('seaborn')
%matplotlib inline

You can install fxcmpy via

pip install fxcmpy

The documentation is currently found under http://fxcmpy.tpq.io

Read the license & risk warning carefully.

In [2]:
import fxcmpy
In [3]:
fxcmpy.__version__
Out[3]:
'1.1.11'

Connecting to the FXCM RESTful API

For this approach, a configuration file is needed (see Quick Start).

In [4]:
api = fxcmpy.fxcmpy(config_file='../fxcm.cfg')
In [5]:
instruments = api.get_instruments()
In [6]:
print(instruments)
['EUR/USD', 'USD/JPY', 'GBP/USD', 'USD/CHF', 'EUR/CHF', 'AUD/USD', 'USD/CAD', 'NZD/USD', 'EUR/GBP', 'EUR/JPY', 'GBP/JPY', 'CHF/JPY', 'GBP/CHF', 'EUR/AUD', 'EUR/CAD', 'AUD/CAD', 'AUD/JPY', 'CAD/JPY', 'NZD/JPY', 'GBP/CAD', 'GBP/NZD', 'GBP/AUD', 'AUD/NZD', 'USD/SEK', 'EUR/SEK', 'EUR/NOK', 'USD/NOK', 'USD/MXN', 'AUD/CHF', 'EUR/NZD', 'USD/ZAR', 'USD/HKD', 'ZAR/JPY', 'USD/TRY', 'EUR/TRY', 'NZD/CHF', 'CAD/CHF', 'NZD/CAD', 'TRY/JPY', 'USD/CNH', 'AUS200', 'ESP35', 'FRA40', 'GER30', 'HKG33', 'JPN225', 'NAS100', 'SPX500', 'UK100', 'US30', 'Copper', 'CHN50', 'EUSTX50', 'USDOLLAR', 'USOil', 'UKOil', 'SOYF', 'NGAS', 'Bund', 'XAU/USD', 'XAG/USD']

Historical Data via RESTful API

In [7]:
candles = api.get_candles('USD/JPY', period='D1', number=10)
In [8]:
# 10 most recent days | daily
candles
Out[8]:
bidopen bidclose bidhigh bidlow askopen askclose askhigh asklow tickqty
date
2018-03-22 21:00:00 106.046 105.273 106.089 105.260 106.055 105.289 106.092 105.262 456441
2018-03-23 21:00:00 105.273 104.712 105.411 104.636 105.289 104.766 105.446 104.639 523620
2018-03-25 21:00:00 104.578 104.641 104.712 104.575 104.612 104.693 104.750 104.592 206
2018-03-26 21:00:00 104.641 105.393 105.482 104.629 104.693 105.418 105.482 104.642 314101
2018-03-27 21:00:00 105.393 105.328 105.903 105.316 105.418 105.353 105.905 105.327 336213
2018-03-28 21:00:00 105.328 106.838 107.012 105.325 105.353 106.864 107.014 105.333 440760
2018-03-29 21:00:00 106.838 106.414 106.933 106.267 106.864 106.443 106.935 106.268 303237
2018-03-30 21:00:00 106.414 106.260 106.537 106.122 106.443 106.312 106.547 106.127 171834
2018-04-02 21:00:00 106.243 105.877 106.450 105.657 106.279 105.910 106.451 105.658 264943
2018-04-03 21:00:00 105.877 106.600 106.658 105.693 105.910 106.620 106.659 105.695 597173
In [9]:
start = dt.datetime(2017, 1, 1)
end = dt.datetime(2018, 1, 1)
In [10]:
candles = api.get_candles('USD/JPY', period='D1',
                         start=start, stop=end)
In [11]:
candles['askclose'].iplot()

The parameter period must be one of m1, m5, m15, m30, H1, H2, H3, H4, H6, H8, D1, W1 or M1.

In [12]:
# 50 most recent 1 minute bars
candles = api.get_candles('EUR/USD', period='m1', number=150)
candles.tail(10)
Out[12]:
bidopen bidclose bidhigh bidlow askopen askclose askhigh asklow tickqty
date
2018-04-04 09:37:00 1.22897 1.22907 1.22907 1.22897 1.22899 1.22907 1.22909 1.22899 97
2018-04-04 09:38:00 1.22907 1.22897 1.22907 1.22897 1.22907 1.22899 1.22907 1.22897 34
2018-04-04 09:39:00 1.22898 1.22898 1.22898 1.22893 1.22900 1.22899 1.22900 1.22893 85
2018-04-04 09:40:00 1.22898 1.22896 1.22900 1.22891 1.22899 1.22897 1.22901 1.22893 98
2018-04-04 09:41:00 1.22896 1.22898 1.22903 1.22891 1.22897 1.22898 1.22904 1.22892 97
2018-04-04 09:42:00 1.22898 1.22891 1.22902 1.22891 1.22898 1.22893 1.22903 1.22893 62
2018-04-04 09:43:00 1.22891 1.22901 1.22902 1.22891 1.22893 1.22904 1.22904 1.22892 47
2018-04-04 09:44:00 1.22901 1.22907 1.22909 1.22892 1.22904 1.22909 1.22910 1.22893 113
2018-04-04 09:45:00 1.22907 1.22909 1.22912 1.22902 1.22909 1.22910 1.22913 1.22903 91
2018-04-04 09:46:00 1.22909 1.22902 1.22917 1.22902 1.22910 1.22903 1.22917 1.22903 126

Retrieving Streaming Data

In [13]:
def output(data, dataframe):
    print('%3d | %s | %s | %6.4f, %6.4f, %6.4f, %6.4f' 
          % (len(dataframe), data['Symbol'],
             pd.to_datetime(int(data['Updated']), unit='ms'), 
             data['Rates'][0], data['Rates'][1],
             data['Rates'][2], data['Rates'][3]))
In [14]:
api.subscribe_market_data('EUR/USD', (output,))
  1 | EUR/USD | 2018-04-04 09:47:13.650000 | 1.2290, 1.2291, 1.2315, 1.2257
  2 | EUR/USD | 2018-04-04 09:47:15.461000 | 1.2290, 1.2290, 1.2315, 1.2257
  3 | EUR/USD | 2018-04-04 09:47:15.736000 | 1.2290, 1.2290, 1.2315, 1.2257
  4 | EUR/USD | 2018-04-04 09:47:18.941000 | 1.2290, 1.2290, 1.2315, 1.2257
  5 | EUR/USD | 2018-04-04 09:47:20.702000 | 1.2290, 1.2290, 1.2315, 1.2257
  6 | EUR/USD | 2018-04-04 09:47:21.260000 | 1.2290, 1.2290, 1.2315, 1.2257
In [15]:
api.get_last_price('EUR/USD')
Out[15]:
Bid     1.22903
Ask     1.22904
High    1.23149
Low     1.22569
Name: 2018-04-04 09:47:21.260000, dtype: float64
In [16]:
api.unsubscribe_market_data('EUR/USD')

Placing Orders via the RESTful API

In [17]:
api.get_open_positions()
Out[17]:
In [18]:
order = api.create_market_buy_order('EUR/USD', 100)
Order_id: 74241808
{'response': {'executed': True}, 'data': {'type': 0, 'orderId': 74241808}}
0
In [19]:
sel = ['tradeId', 'amountK', 'currency', 'grossPL', 'isBuy']
In [20]:
api.get_open_positions()[sel]
Out[20]:
tradeId amountK currency grossPL isBuy
0 49311194 100 EUR/USD -2 True
In [21]:
order = api.create_market_buy_order('USD/JPY', 50)
Order_id: 74241809
{'response': {'executed': True}, 'data': {'type': 0, 'orderId': 74241809}}
0
In [22]:
api.get_open_positions()[sel]
Out[22]:
tradeId amountK currency grossPL isBuy
0 49311194 100 EUR/USD -2.00000 True
1 49311195 50 USD/JPY -0.47112 True
In [23]:
order = api.create_market_sell_order('EUR/USD', 25)
Order_id: 74241810
{'response': {'executed': True}, 'data': {'type': 0, 'orderId': 74241810}}
1
In [24]:
order = api.create_market_buy_order('USD/JPY', 50)
Order_id: 74241811
{'response': {'executed': True}, 'data': {'type': 0, 'orderId': 74241811}}
0
In [25]:
api.get_open_positions()[sel]
Out[25]:
tradeId amountK currency grossPL isBuy
0 49311195 50 USD/JPY 3.29762 True
1 49311317 75 EUR/USD -2.25000 True
2 49311197 50 USD/JPY 1.41327 True
In [26]:
order = api.create_market_sell_order('EUR/USD', 50)
Order_id: 74241812
{'response': {'executed': True}, 'data': {'type': 0, 'orderId': 74241812}}
1
In [27]:
api.get_open_positions()[sel]
Out[27]:
tradeId amountK currency grossPL isBuy
0 49311195 50 USD/JPY 3.76868 True
1 49311197 50 USD/JPY 1.88434 True
2 49311318 25 EUR/USD -0.75000 True
In [28]:
api.close_all_for_symbol('USD/JPY')
In [29]:
api.get_open_positions()[sel]
Out[29]:
tradeId amountK currency grossPL isBuy
0 49311318 25 EUR/USD -0.75 True
In [30]:
api.close_all()
In [31]:
api.get_open_positions()
Out[31]:

Backtesting AI-Based Algorithmic Trading Strategy

The following example is simplified and for illustration purposes only. Among others, it does not consider transactions costs or bid-ask spreads.

Data Retrieval

In [32]:
candles = api.get_candles('EUR/USD', period='m5',
                         start=dt.datetime(2018, 3, 26),
                          stop=dt.datetime(2018, 3, 30))
In [33]:
data = pd.DataFrame(candles[['askclose', 'bidclose']].mean(axis=1),
                    columns=['midclose'])
In [34]:
data.info()
<class 'pandas.core.frame.DataFrame'>
DatetimeIndex: 1153 entries, 2018-03-26 00:00:00 to 2018-03-30 00:00:00
Data columns (total 1 columns):
midclose    1153 non-null float64
dtypes: float64(1)
memory usage: 18.0 KB
In [35]:
data.iplot()

Feature Preparation

In [36]:
data['returns'] = np.log(data / data.shift(1))
In [37]:
lags = 5
cols = []
for lag in range(1, lags + 1):
    col = 'lag_%s' % lag
    data[col] = data['returns'].shift(lag)
    cols.append(col)
In [38]:
cols
Out[38]:
['lag_1', 'lag_2', 'lag_3', 'lag_4', 'lag_5']
In [39]:
data.dropna(inplace=True)
In [40]:
# the "patterns" = 2 ** lags
np.digitize(data[cols], bins=[0])[:10]
Out[40]:
array([[0, 0, 1, 1, 1],
       [1, 0, 0, 1, 1],
       [0, 1, 0, 0, 1],
       [1, 0, 1, 0, 0],
       [1, 1, 0, 1, 0],
       [1, 1, 1, 0, 1],
       [1, 1, 1, 1, 0],
       [0, 1, 1, 1, 1],
       [0, 0, 1, 1, 1],
       [1, 0, 0, 1, 1]])
In [41]:
2 ** len(cols)
Out[41]:
32

Support Vector Machines

In [42]:
from sklearn import svm
In [43]:
model = svm.SVC(C=100)

Train Test Split

In [44]:
from sklearn.metrics import accuracy_score
from sklearn.model_selection import train_test_split
In [45]:
train_x, test_x, train_y, test_y = train_test_split(
    data[cols].apply(lambda x: np.digitize(x, bins=[0])),                   
    np.sign(data['returns']),
    test_size=0.50, random_state=111)
In [46]:
train_x.sort_index(inplace=True)
train_y.sort_index(inplace=True)
test_x.sort_index(inplace=True)
test_y.sort_index(inplace=True)

Model Fitting & Prediction

In [47]:
model.fit(train_x, train_y)
Out[47]:
SVC(C=100, cache_size=200, class_weight=None, coef0=0.0,
  decision_function_shape='ovr', degree=3, gamma='auto', kernel='rbf',
  max_iter=-1, probability=False, random_state=None, shrinking=True,
  tol=0.001, verbose=False)
In [48]:
train_pred = model.predict(train_x)
In [49]:
accuracy_score(train_y, train_pred)
Out[49]:
0.5898778359511344
In [50]:
test_pred = model.predict(test_x)
In [51]:
accuracy_score(test_y, test_pred)
Out[51]:
0.49477351916376305

Vectorized Backtesting

In [52]:
data['position'] = model.predict(data[cols].apply(lambda x: np.digitize(x, bins=[0])).dropna())
In [53]:
data['strategy'] = data['position'] * data['returns']
In [54]:
# in-sample | unleveraged | no bid-ask spread or transaction costs
data.loc[train_x.index][['returns', 'strategy']].cumsum().apply(np.exp).iplot()
In [55]:
# out-of-sample | unleveraged | no bid-ask spread or transaction costs
data.loc[test_x.index][['returns', 'strategy']].cumsum().apply(np.exp).iplot()
In [56]:
# number of trades
sum(data['position'].diff() != 0)
Out[56]:
568

Automating A Simple Time-Based Algorithm

In [57]:
for _ in range(5):
    print(50 * '=')
    print('TRADE NO {}'.format(_))
    order = api.create_market_buy_order('EUR/USD', 100)
    time.sleep(1)
    print('POSITIONS\n', api.get_open_positions()[sel])
    time.sleep(7)
    api.close_all_for_symbol('EUR/USD')
    print('POSITIONS\n', api.get_open_positions())
    time.sleep(7)
==================================================
TRADE NO 0
Order_id: 74243485
{'response': {'executed': True}, 'data': {'type': 0, 'orderId': 74243485}}
0
POSITIONS
     tradeId  amountK currency  grossPL  isBuy
0  49311489      100  EUR/USD       -1   True
POSITIONS
 Empty DataFrame
Columns: []
Index: []
==================================================
TRADE NO 1
Order_id: 74243492
{'response': {'executed': True}, 'data': {'type': 0, 'orderId': 74243492}}
1
POSITIONS
     tradeId  amountK currency  grossPL  isBuy
0  49311496      100  EUR/USD        0   True
POSITIONS
 Empty DataFrame
Columns: []
Index: []
==================================================
TRADE NO 2
Order_id: 74243500
{'response': {'executed': True}, 'data': {'type': 0, 'orderId': 74243500}}
1
POSITIONS
     tradeId  amountK currency  grossPL  isBuy
0  49311502      100  EUR/USD       -2   True
POSITIONS
 Empty DataFrame
Columns: []
Index: []
==================================================
TRADE NO 3
Order_id: 74243508
{'response': {'executed': True}, 'data': {'type': 0, 'orderId': 74243508}}
1
POSITIONS
     tradeId  amountK currency  grossPL  isBuy
0  49311509      100  EUR/USD       -1   True
POSITIONS
 Empty DataFrame
Columns: []
Index: []
==================================================
TRADE NO 4
Order_id: 74243875
{'response': {'executed': True}, 'data': {'type': 0, 'orderId': 74243875}}
1
POSITIONS
     tradeId  amountK currency  grossPL  isBuy
0  49311529      100  EUR/USD       -2   True
POSITIONS
 Empty DataFrame
Columns: []
Index: []

The Python Quants