Linux GCC	Windows MSVC

• Sections
  • 1. Introduction and features
  • 2. Overview of architecture and multithreading system
  • 3. Low latency features , benchmarks and how it can be improved
  • 4. Build and runtime dependencies
  • 5. How to build
  • 6. Configuring and running the server
  • 7. Offline order entry mode
  • 8. Test harness / FIX client automation
  • 9. Unit tests
  • 10. Utility scripts
  • 11. Coding conventions
  • 12. Code map

This project is my "core tech" programming lab / playground.

It is essentially a minimal traditional stock exchange. It uses FIX protocol for order entry and also market data ( highly likely mold udp 64/ ITCH ) and dropcopies will be included in future.

A nice video which explains components of an exchange : https://www.youtube.com/watch?v=b1e4t2k2KJY

For FIX protocol , see https://en.wikipedia.org/wiki/Financial_Information_eXchange

It is cross platform (Linux/Windows) , written in C++11. Its test harness is written in stock Python.

• Named as "lite" because :

  • Does not use any 3rd party library. Only GoogleTest is used for unit testing.
  • No runtime dependency except C++ runtime and socket access.
  • All configurations are mananaged in one ini file.
  • It is meant to run on a single server , therefore designed to be only vertically-scalable meaning that the more CPU cores you add , a better performance you might get.
  • It supports minimum electronic trading features and also does not do many FIX validations. All existing features can be seen in the tables below.

Its architecture overview is as below :

The core of order matching layer is called the central order book, which keeps order books per security symbol.

1. Main thread is a single-thread epoll FIX server which passes orders to the central order book`s SPSC lockfree queues per symbol
2. Central order book uses a thread pool in which there is a thread per symbol and ideally pinned to a CPU core. They process messages and push results to SPSC queues of outgoing message processor per symbol.
3. Outgoing message procesor sends execution reports to the FIX clients.

Some features are tunable through the configuration file ( liteExchange.ini ) and some are not.

Non-tunable low latency features are as below :

The tunable low latency features :

• Planning to have benchmarks after completing some key performance killers / jitter sources below :

  • Object pools : Currently no pooling for FIX message or order object instances

  • Memory allocations : Project can benefit from preallocating everything in critical path.

  • Data oriented design : Currently order class in order books suffers from cache misses. If the order class is split into core order ( price-side-symbol ) and other order data, the matching engine would gain a lot of speed by avoiding cache misses

  • Cache-aware algorithms : Order book implementation uses std::multimap which is known to use red-black tree which is controversial when it comes to be cache friendly : https://news.ycombinator.com/item?id=7513896

For Linux , the project is built and tested with GCC4.8 and GCC7 on CentOS7 and Ubuntu.

For running on Linux , make sure you have GNU Libstd C++ 6 runtime in your Linux distribution

CentOS :

Find out package name for your architecture : yum whatprovides libstdc++.so.6
Then yum install the package you found

Ubuntu :

sudo apt-get install libstdc++6

As for Windows you can build with MSVC141(VS2017).

For running on Windows , you need to install MSVC141 ( VS2017 ) C++ runtime :

Go to https://support.microsoft.com/en-gb/help/2977003/the-latest-supported-visual-c-downloads
Download and install VS2017 x86 or x64 version

How to build the project on Linux :

cd build/linux_gcc
make clean
make debug  OR make release
or if you want to make a parallel build :
./build_in_parallel.sh release

How to debug the project on Linux using VisualStudio code : Build the project for debug mode and open the source directory and VisualStudioCode. Source directory provides a launch.json supporting GCC on Linux and MSVC on Windows. Therefore choose GDB in VSCode debug options before starting debugging and then press F5.

How to build the project on Linux from Windows with Visual Studio 2017 : You will need to install Linux C++ feature during VS2017 installation. Then after making sure that your Linux has SSH, you will need your Linux machine SSH details to your Visual Studio. You will also need to copy lieExchange directory to your home directory ( ~/ ). After that you will be able to build and debug on remote Linux. For details please see https://nativecoding.wordpress.com/2018/02/24/visual-studio-for-existing-remote-linux-c-projects/

How to build the project on Windows with Visual Studio 2017 : Go to "build/windows_msvc_visual_studio" directory and use SLN file to launch VS with the project

How to build the project on Windows with Visual Studio in command line : Go to "build/windows_msvc_command_line" directory and execute one of batch files : build_debug.bat or build_release.bat

Warning levels : Using -Wall for GCC and /W3 for MSVC

Precompiled header file usage : On Windows , the project is using /FI ( Force include parameter, therefore no need to include the pch header everywhere ) and specified the pch header to be precompiled_header.h. Note that this breaks edit-and-continue in Visual Studio. Also precompiled header setting of all projects is "use" whereas precompiled_header.cpp is "create". For Linux , there is pch rule to enable it in the makefile ( build/linux/Makefile) , but currently that rule is not being used since it doesn`t appear as it is doing much improvement as on Windows.

For GCC see https://gcc.gnu.org/onlinedocs/gcc/Precompiled-Headers.html

For MSVC see https://msdn.microsoft.com/en-us/library/8c5ztk84(v=vs.140).aspx

The engine executable looks for "liteExchange.ini" file. There is a few categories of configuration parameters.

You will also need to specify security symbols. The order matching engine`s thread pool will create a worker thread for each symbol. For specifying symbols in ini file, you need to use brackets as below :

    SYMBOL[]=MSFT
    SYMBOL[]=AAPL
    SYMBOL[]=INTC
    SYMBOL[]=GOOGL

Once you start the executable , initially you will see a screen like this :

    06-02-2016 16:22:00 : INFO , Main thread , starting
    06-02-2016 16:22:00 : INFO , Thread pool , Thread(0) MSFT starting
    06-02-2016 16:22:00 : INFO , Thread pool , Thread(1) AAPL starting
    06-02-2016 16:22:00 : INFO , Thread pool , Thread(2) INTC starting
    06-02-2016 16:22:00 : INFO , Thread pool , Thread(3) GOOGL starting
    06-02-2016 16:22:00 : INFO , Thread pool , Thread(4) QCOM starting
    06-02-2016 16:22:00 : INFO , Thread pool , Thread(5) QQQ starting
    06-02-2016 16:22:00 : INFO , Thread pool , Thread(6) BBRY starting
    06-02-2016 16:22:00 : INFO , Thread pool , Thread(7) SIRI starting
    06-02-2016 16:22:00 : INFO , Thread pool , Thread(8) ZNGA starting
    06-02-2016 16:22:00 : INFO , Thread pool , Thread(9) ARCP starting
    06-02-2016 16:22:00 : INFO , Thread pool , Thread(10) XIV starting
    06-02-2016 16:22:00 : INFO , Thread pool , Thread(11) FOXA starting
    06-02-2016 16:22:00 : INFO , Thread pool , Thread(12) TVIX starting
    06-02-2016 16:22:00 : INFO , Thread pool , Thread(13) YHOO starting
    06-02-2016 16:22:00 : INFO , Thread pool , Thread(14) HBAN starting
    06-02-2016 16:22:00 : INFO , Thread pool , Thread(15) BARC starting
    06-02-2016 16:22:00 : INFO , Incoming message dispatcher , Thread starting
    06-02-2016 16:22:00 : INFO , Outgoing message processor , Thread starting
    06-02-2016 16:22:00 : INFO , FIX Engine , Acceptor started

    Available commands :

            display : Shows all order books in the central order book
            quit : Shutdowns the server

The default mode is FIX server mode. However, if you specify an order file in liteExchange.ini as below :

				ORDER_ENTRY_MODE=OFFLINE
				OFFLINE_ORDER_ENTRY_FILE=sample_offline_order_file.txt
				OFFLINE_ORDER_ENTRY_OUTPUT_FILE=offline_order_entry_output.txt

Then the order matcher will process all the orders in that file bypassing FIX protocol and produce offline_order_entry_output.txt as result file. For an example offline order file see :

https://github.com/akhin/multithreaded_order_matching_engine/blob/master/bin/sample_offline_order_file.txt

The output of offline order matching will have timestamps with microsecond precision and FIX execution report messages as below :

	16-09-2017 03:12:27.739729
	8=FIX.4.2�9=68�35=8�6=0�11=�14=0�17=1�20=0�37=�38=2�39=0�54=2�55=GOOGL�150=0�151=2�10=001�
	16-09-2017 03:12:27.755355
	8=FIX.4.2�9=68�35=8�6=0�11=�14=0�17=2�20=0�37=�38=1�39=0�54=1�55=GOOGL�150=0�151=1�10=255�
	16-09-2017 03:12:27.773486
	8=FIX.4.2�9=78�35=8�6=1�11=�14=1�17=3�20=0�31=1�32=1�37=�38=1�39=2�54=1�55=GOOGL�150=2�151=0�10=173�
	16-09-2017 03:12:27.777489

Under "test_harness" directory :

The project has a Python script which can send orders from FIX files using Fix protocol.

1. Modify test_cases.txt which has the orders to send to the engine as FIX messages.

2. To run it : python ./fix_client_automation.py

3. You can set automation parameters in the commandline :

The project uses GoogleTest 1.7. You can find a makefile and vcproj under "unit_test" directory.

Building and running unit test on Linux : You can use Makefile under "unit_test" directory.

You can find them under "utility_scripts" directory :

Source code and file/directory naming conventions :

Source code indentations and new line usage :

- 4 spaces, no tabs ( This needs to be setup in VS )
- By default Netbeans editor uses spaces for tabs
- Needs to be set in VS2013 : https://msdn.microsoft.com/en-gb/library/ms165330(v=vs.90).aspx
- New lines : Unix CR only ( \n ) , VisualStudio can handle it even though Windows is \r\n

utility/source_code_formatter.sh : It is a Bash script that scans all cpp,h,hpp files in project directory and converts Windows end of lines to Linux, converts tabs to 4 spaces and removes trailing whitespace. It requires dos2unix.

Inclusions : Using forward slash as it works for both Linux and Windows :

		#include <core/concurrency/thread.h>

core : It is the core library. Has subdirectories such as as network, concurrency etc

fix : It mainly uses core/network and core/concurrency. It is the FIX implemenatation

order_matcher : Order matcher layer is the heart of project. It knows nothing about FIX protocol but has its own order, incoming message and outgoing message classes.

server : This layer is the layer that uses FIX layer to receive FIX orders and submit to the central order book in the order matcher layer. Then it receives results from the central order book and sends them to the outgoing message processor.