Commit ff799c29 authored by Yuqi Zhang's avatar Yuqi Zhang
Browse files

update readme. add crdt-list.

parent b07310b6
# Conflict-Free Replicated Priority Queues Based on Redis
# Conflict-Free Replicated Data Types Based on Redis
[**Add-Win CRPQ**](document/add-win-crpq.pdf) and [**Remove-Win CRPQ**](https://arxiv.org/abs/1905.01403) implementations based on Redis (6.0.5, previously 4.0.8).
Several Conflict-Free Replicated Data Types (CRDTs) implemented based on Redis(6.0.5).
Things we do for such implementation:
* Enable Redis to replicate in P2P mode.
* Implement the CRDT framework.
* Implement Add-Win CRPQ and Remove-Win CRPQ.
* Implement specific CRDTs according to their algorithms.
For more detail of our implementation, please read the *Performance measurements* section of [the technical report](https://arxiv.org/abs/1905.01403).
The CRDTs currently implemented:
* Replicated Priority Queue (RPQ)
* [Add-Win RPQ](document/add-win-RPQ.pdf)
* [Remove-Win RPQ](document/rwf-tr.pdf)
* [RWF-RPQ](document/rwf-tr.pdf)
* List
* [Remove-Win List](document/rwf-tr.pdf)
* [RWF-List](document/rwf-tr.pdf)
For more details of our implementation, please read the linked article of the CRDTs above. Also you can read the *Performance measurements* section of [the technical report](https://arxiv.org/abs/1905.01403). Note that the **Remove-Win RPQ** in this technical report corresponds to the **RWF-RPQ** here.
## Build
Our modified Redis is in folder *redis-6.0.5*. Please build it in the default mode:
```
```bash
cd redis-6.0.5
make
sudo make install
......@@ -22,7 +32,7 @@ sudo make install
## Test
In the folder *experiment/redis_test* there are configuration files and scripts for testing our modified Redis and CRPQ implementations. By default, the test will start 5 Redis server instances on the local machine, listening on port 6379, 6380, 6381, 6382 and 6383. You may choose to start some of these server instances by using parameters when you run the scripts.
In the folder *experiment/redis_test* there are scripts for testing our modified Redis and RPQ implementations. By default, the test will start 5 Redis server instances on the local machine, listening on port 6379, 6380, 6381, 6382 and 6383. You may choose to start some of these server instances by using parameters when you run the scripts.
Here we focus on 5 scripts:
......@@ -36,46 +46,61 @@ Here we show an example to test our implementation using all the 5 Redis server
Firstly go to the *experiment/redis_test*, **start** the Redis server instances and **construct P2P replication** among them.
```
```bash
cd experiment/redis_test
./server.sh
./construct_replication.sh
```
Now you can start redis clients to connect to Redis servers and do CRPQ read/write operations. Here shows **start a client** to connect one Redis server.
Now you can start redis clients to connect to Redis servers and do RPQ read/write operations. Here shows **start a client** to connect one Redis server.
```
```bash
./client.sh <server_port>
```
When you've finished testing, you may **close** the Redis server instances.
```
```bash
./shutdown.sh
```
Finally, you can **remove** the Redis database files (.rdb files) and log files (.log files).
```
```bash
./clean.sh
```
To further redo the experiment of our work our please check [here](experiment/Readme.md).
## CRPQ operations
## CRDT Operations
Different implementations of the same type of CRDT offer the same operations for users. We use **[type][op]** to name our CRDT operations. The operations name contains of an implementation type prefix and an operation type suffix. For example, the name of operation of RWF-RPQ to add one element is **rwfzadd**, where **rwf** is the implementation type prefix and **zadd** is the operation type suffix.
### RPQ operations
The **[type]** prefix of different RPQs are:
* Add-Win RPQ : **o**
* Remove-Win RPQ : **r**
* RWF-RPQ : **rwf**
The operations of RPQs are:
* **[type]zadd Q E V** : Add a new element *E* into the priority queue *Q* with initial value *V*.
* **[type]zincby Q E I** : Add the increment *I* to the value of element *E* in the priority queue *Q*.
* **[type]zrem Q E** : Remove element *E* from the priority queue *Q*.
* **[type]zscore Q E** : Read the value of element *E* from the priority queue *Q*.
* **[type]zmax Q** : Read the element with the largest value in the priority queue *Q*. Returns the element and its value.
### List operations
There are 5 operations of **Add-Win CRPQ** implemented in our modified Redis:
The **[type]** prefix of different Lists are:
* **ozadd Q E V** : Add a new element *E* into the priority queue *Q* with initial value *V*.
* **ozincby Q E I** : Add the increment *I* to the value of element *E* in the priority queue *Q*.
* **ozrem Q E** : Remove element *E* from the priority queue *Q*.
* **ozscore Q E** : Read the value of element *E* from the priority queue *Q*.
* **ozmax Q** : Read the element with the largest value in the priority queue *Q*. Returns the element and its value.
* Remove-Win List : **r**
* RWF-List : **rwf**
There are 5 operations of **Remove-Win CRPQ** implemented in our modified Redis:
The operations of Lists are:
* **rzadd Q E V** : Add a new element *E* into the priority queue *Q* with initial value *V*.
* **rzincby Q E I** : Add the increment *I* to the value of element *E* in the priority queue *Q*.
* **rzrem Q E** : Remove element *E* from the priority queue *Q*.
* **rzscore Q E** : Read the value of element *E* from the priority queue *Q*.
* **rzmax Q** : Read the element with the largest value in the priority queue *Q*. Returns the element and its value.
* **[type]linsert L prev id content font size color property** : Add a new element *id* after *prev* into the list *L*, with the content *content*, and the initial properties *font* *size* *color* and *property*(bitmap that encodes bold, italic and underline). If *prev* is "null", it means insert the element at the beginning of the list. If *prev* is "readd", it means to re-add the element that is previously added and then removed, and restore its position.
* **[type]lupdate L id type value** : Update the *type* property with *value* for element *id* in list *L*.
* **[type]lrem L id** : Remove element *id* from the list *L*.
# CRPQ Experiments
# CRDT Experiments
Here are the tests and the experiments we do on our CPRQ implementations. There are 3 folders here:
Here are the tests and the experiments we do on our CRDT implementations. There are 3 folders here:
* *redis_test* : The testing bash scripts and server configuration files. Also used in the experiment.
* **.sh* : Bash scripts for starting and testing on local machine.
* *test.py* : Local test we do to test the correctness of our CRPQ implementations.
* *connection.py* : Construct the server part of experiment framework, close it and clean it when the experiment is finished. It use ssh to control the server VMs to start their Redis instances, construct replication among them, and add networking delay between them. When the experiment is finished, close all the server and remove their .rdb and .log files.
* *bench* : The experiment code. Constructs the experiment, generates and runs operations, get logs and results.
* *constants.h* : The constants of our experiment.
* *generator.h* : The generator of CRPQ operations. Including write operations with deliberate concurrent conflicts.
* *test.py* : Local test we do to test the correctness of our CRDT implementations.
* *connection.py* : Used for experiments using virtual machines. Construct the server part of experiment framework, close it and clean it when the experiment is finished. It use ssh to control the server VMs to start their Redis instances, construct replication among them, and add networking delay between them. When the experiment is finished, close all the server and remove their .rdb and .log files.
* *bench* : The experiment code. Constructs the experiment, generates and runs operations, get logs and results.
* *exp_env.h* : Construct the experiment environment. Start servers, construct replication, setup delays. Shutdown and cleanup when this round finishes.
* *exp_runner.h* : Run the exp. Start client threads, calls the generator and periodically read from server to log.
* *exp_setting.h* : The setting parameters of this round of experiment.
* *queue_log.h*, *queue_log.cpp* : Log the execution of the write operations and the results of the read operations during the experiment. Write the results to the *result* folder at the end of the experiment.
* *main.cpp* : The entry of the experiment. It calls *redis_test/connection.py* to construct the experiment environment. Then starts multiple Redis client threads for each Redis server. It uses the generator to generate workload to run at the client threads. Finally it lets the log to write the results.
* *main.cpp* : The entry of the experiment.
* *util.h* : Some utilities: random generator, redis client, redis reply. And the base classes of the CRDT experiments:
* cmd : the CRDT command
* generator : cmd generation logic
* rdt_exp : experiment settings and experiment instance generation
* rdt_log : log the CRDT execution and read result, write result to *result* folder
* *rpq* and *list* : Actual RPQ and List experiment folder. They extend the base classes in *util.h*, and implement the specific CRDT experiment logics.
* *result* : The results and the data analysis.
* *result.py* : Compute the statistics of the result data and draw result figures.
* *result.zip* : The result data and figures of our experiment packed in a zip file.
* **.py* : The python scripts to compute the statistics of the result data and draw result figures.
## Experiment Framework
<div align=center><img src="exp-crop.png" width="65%" height="65%"></div>
We use VMware to run 4 virtual machines. VM 1-3 run Ubuntu server 16.04.5 LTS. VM4 runs Ubuntu desktop 16.04.5 LTS. Each of VM 1-3 is deployed with Redis server instances and simulates a data center. VM4 runs all the clients.
We run all server nodes and client nodes on a workstation. Logically we divide the Redis servers into 3 data centers. Each data center has 1-5 instances of Redis.
We achieve this by designating different localhost IPs for different kind of connections (e.g. 127.0.0.1 for client-server connection, 127.0.0.2 for between data center connection, and 127.0.0.3 for within data center connection).
The ips of VM 1-3 are *192.168.188.135*, *192.168.188.136*, *192.168.188.137*. You can see the ip settings in the beginning of *bench/main.cpp* and *redis_test/connection.py*.
The entry of the experiment is *bench/main.cpp*. It will do 30 rounds of experiment for each experiment settings. Each round it will call *redis_test/connection.py* to start new Redis server instances in all the server VMs and set the networking delays. The results will be written in the *results* folder in different subfolders according to different experiment settings and rounds.
The entry of the experiment is *bench/main.cpp*. It will do the pattern comparison experiment first, and then 16 rounds of environment related experiment for each environment settings.
The results will be written in the *results* folder in different subfolders according to different CRDTs, experiment settings and rounds.
## Perform the Experiment
To perform our CRPQ experiments, setup the experiment as shown above. You need to modify the ip settings at the start of *bench/main.cpp* and *redis_test/connection.py* if you want to use other ips for servers. Then start the experiment:
To perform our CRDT experiments, follow the instructions below. First you need to [compile the CRDT-Redis](../Readme.md#build). You can modify the IP settings at *bench/exp_env.h* if you want to use other IPs for servers. Then start the experiment:
cd bench
make run
```bash
cd bench
make run
```
After it finished you will get the data in the *result* folder.
You may then draw figures by (python 3.6 required):
You need to enter the sudo password for using tc to control message delay.
cd ../result
python result.py
After it finishes you will get the data in the *result* folder.
You may then draw figures by (python 3.6 required):
Note that there may be some bugged results that are apparently invalid. You may find them by using the according find function in *result/result.py*, and then fix them by use the according fix functions in *bench/main.cpp* accordingly.
```bash
cd ../result
python result.py
```
For more detail of our implementation, please read the *Performance measurements* section of [the technical report](https://arxiv.org/abs/1905.01403).
For more details of our implementation and experiment, please read the *Performance measurements* section of [the technical report](https://arxiv.org/abs/1905.01403) and [the article](document/add-win-RPQ.pdf).
experiment/exp-crop.png

13.9 KB | W: | H:

experiment/exp-crop.png

91.6 KB | W: | H:

experiment/exp-crop.png
experiment/exp-crop.png
experiment/exp-crop.png
experiment/exp-crop.png
  • 2-up
  • Swipe
  • Onion skin
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment