Paxi is the framework that implements WPaxos and other Paxos protocol variants. Paxi provides most of the elements that any Paxos implementation or replication protocol needs, including network communication, state machine of a key-value store, client API and multiple types of quorum systems.

Warning: Paxi project is still under heavy development, with more features and protocols to include. Paxi API may change too.

Paxi paper (SIGMOD) can be found in https://dl.acm.org/doi/abs/10.1145/3299869.3319893. BibTex:

@inproceedings{ailijiang2019dissecting,
  title={Dissecting the Performance of Strongly-Consistent Replication Protocols},
  author={Ailijiang, Ailidani and Charapko, Aleksey and Demirbas, Murat},
  booktitle={Proceedings of the 2019 International Conference on Management of Data},
  pages={1696--1710},
  year={2019}
}

WPaxos is a multileader Paxos protocol that provides low-latency and high-throughput consensus across wide-area network (WAN) deployments. Unlike statically partitioned multiple Paxos deployments, WPaxos perpetually adapts to the changing access locality through object stealing. Multiple concurrent leaders coinciding in different zones steal ownership of objects from each other using phase-1 of Paxos, and then use phase-2 to commit update-requests on these objects locally until they are stolen by other leaders. To achieve fast phase-2 commits, WPaxos adopts the flexible quorums idea in a novel manner, and appoints phase-2 acceptors to be close to their respective leaders.

WPaxos (WAN Paxos) paper (TPDS journal version) can be found in https://ieeexplore.ieee.org/abstract/document/8765834. BibTex:

@article{ailijiang2019wpaxos,
  title={WPaxos: Wide area network flexible consensus},
  author={Ailijiang, Ailidani and Charapko, Aleksey and Demirbas, Murat and Kosar, Tevfik},
  journal={IEEE Transactions on Parallel and Distributed Systems},
  volume={31},
  number={1},
  pages={211--223},
  year={2019},
  publisher={IEEE}
}

Algorithms:

• Classical multi-Paxos
• Flexible Paxos
• WPaxos
• EPaxos
• SDPaxos
• Atomic Storage (Majority Replication)
• Chain Replication
• KPaxos (Static partitioned Paxos)
• Dynamo Key-value Store
• WanKeeper
• Vertical Paxos

Features:

• Benchmarking
• Linerizability checker
• Fault injection

1. Install Go.
2. Use go get command or Download Paxi source code from GitHub page.

go get github.com/ailidani/paxi

3. Compile everything from paxi/bin folder.

cd github.com/ailidani/paxi/bin
./build.sh

After compile, Golang will generate 3 executable files under bin folder.

• server is one replica instance.
• client is a simple benchmark that generates read/write reqeust to servers.
• cmd is a command line tool to test Get/Set requests.

Each executable file expects some parameters which can be seen by -help flag, e.g. ./server -help.

1. Create the configuration file according to the example, then start server with -config FILE_PATH option, default to "config.json" when omit.

2. Start 9 servers with different ids in format of "ZONE_ID.NODE_ID".

./server -id 1.1 -algorithm=paxos &
./server -id 1.2 -algorithm=paxos &
./server -id 1.3 -algorithm=paxos &
./server -id 2.1 -algorithm=paxos &
./server -id 2.2 -algorithm=paxos &
./server -id 2.3 -algorithm=paxos &
./server -id 3.1 -algorithm=paxos &
./server -id 3.2 -algorithm=paxos &
./server -id 3.3 -algorithm=paxos &

3. Start benchmarking client that connects to server ID 1.1 and benchmark parameters specified in config.json.

./client -id 1.1 -config config.json

When flag id is absent, client will randomly select any server for each operation.

The algorithms can also be running in simulation mode, where all nodes are running in one process and transport layer is replaced by Go channels. Check simulation.sh script on how to run.

Replication algorithm in Paxi follows the message passing model, where several message types and their handle function are registered. We use Paxos as an example for our step-by-step tutorial.

1. Define messages, register with gob in init() function if using gob codec. As show in msg.go.

2. Define a Replica structure embeded with paxi.Node interface.

type Replica struct {
	paxi.Node
	*Paxos
}

Define handle function for each message type. For example, to handle client Request

func (r *Replica) handleRequest(m paxi.Request) {
	if *adaptive {
		if r.Paxos.IsLeader() || r.Paxos.Ballot() == 0 {
			r.Paxos.HandleRequest(m)
		} else {
			go r.Forward(r.Paxos.Leader(), m)
		}
	} else {
		r.Paxos.HandleRequest(m)
	}

}

3. Register the messages with their handle function using Node.Register(interface{}, func()) interface in Replica constructor.

Replica use Send(to ID, msg interface{}), Broadcast(msg interface{}) functions in Node.Socket to send messages.

For data-store related functions check db.go file.

For quorum types check quorum.go file.

Client uses a simple RESTful API to submit requests. GET method with URL "http://ip:port/key" will read the value of given key. POST method with URL "http://ip:port/key" and body as the value, will write the value to key.