mirror of
https://github.com/OpenHFT/Chronicle-Map.git
synced 2021-09-13 01:26:12 +00:00
TeamCity Automatic Merge from master into feature/java11
This commit is contained in:
Team City
@@ -1,6 +1,7 @@
|
||||
### How to update
|
||||
# This is copied from OpenHFT/.gitignore
|
||||
# update the original and run OpenHFT/update_gitignore.sh
|
||||
data/
|
||||
|
||||
### Compiled class file
|
||||
*.class
|
||||
|
||||
+57
-112
@@ -18,26 +18,15 @@ In addition, you will be fully supported by our technical experts.
|
||||
|
||||
For more information on Chronicle Map (Enterprise Edition), please contact mailto:sales@chronicle.software[sales@chronicle.software].
|
||||
|
||||
To ensure that Chronicle Map's multi-master replication works under extreme circumstances, we have
|
||||
built a stress test to ensure this using the principles introduced by Netflix with their "chaos monkey".
|
||||
For more information on the stress test see <<CM_Replication_Stress_Test.adoc#,CM_Replication_Stress_Test>>.
|
||||
|
||||
= Replication
|
||||
|
||||
Chronicle Map Replication is a multi-master replicated data store. Chronicle Map supports both TCP and UDP replication
|
||||
Chronicle Map Replication is a multi-master replicated data store. Chronicle Map supports replication over TCP/IP
|
||||
|
||||
image::http://openhft.net/wp-content/uploads/2014/07/Chronicle-Map-TCP-Replication_simple_02.jpg[TCP/IP Replication]
|
||||
|
||||
== TCP / UDP Background
|
||||
== TCP Background
|
||||
TCP/IP is a reliable protocol. This means that, unless you have a network failure or hardware outage, the data is guaranteed to arrive. TCP/IP provides point-to-point connectivity. For example, if the message was sent to 100 hosts, the message would have to be sent 100 times.
|
||||
|
||||
With UDP, the message is only sent once. This is ideal if you have a large number of hosts, and you wish to broadcast the same data to each of them. However, one of the big drawbacks with UDP is that it is not a reliable protocol. This means, if the UDP message is broadcast onto the network, the hosts are not guaranteed to receive it; so they can miss data.
|
||||
|
||||
Some solutions attempt to build resilience into UDP, but arguably, this is in effect reinventing TCP/IP.
|
||||
|
||||
== How to setup UDP Replication
|
||||
On a good quality wired LAN, when using UDP, you will rarely miss messages. Nevertheless, this is a risk that we suggest you do not take. We suggest that whenever you use UDP replication, you use it in conjunction with throttled TCP replication. Therefore, if a host misses a message over UDP, it will pick it up later via TCP/IP.
|
||||
|
||||
== TCP/IP Throttling
|
||||
We are careful not to swamp your network with too much TCP/IP traffic. We do this by providing a throttled version of TCP replication. This works because Chronicle Map only broadcasts the latest update of each entry.
|
||||
|
||||
@@ -114,79 +103,69 @@ map = ChronicleMapBuilder
|
||||
|
||||
== Configuration
|
||||
|
||||
Configuration of map nodes is done either, programmatically, through configuration files, or by a combination of both.
|
||||
Configuration of map nodes is done either, by creating configuration programmatically, or through YAML configuration files.
|
||||
|
||||
=== Programmatically
|
||||
The example below shows how to set up two map instances that use TCP to replicate entries between them:
|
||||
|
||||
[source, java]
|
||||
....
|
||||
|
||||
// set up some socket aliases
|
||||
TCPRegistry.createServerSocketChannelFor("host.port1", "host.port2");
|
||||
|
||||
// create replicated maps
|
||||
final ReplicatedMap<Long, Long> replicaOne = createMap(1, "host.port1");
|
||||
final ReplicatedMap<Long, Long> replicaTwo = createMap(2, "host.port2");
|
||||
|
||||
// start replication event loops
|
||||
replicaOne.start();
|
||||
replicaTwo.start();
|
||||
|
||||
// register maps for replication events
|
||||
replicaOne.beginReplicationFromPeer("host.port2", (byte) 2);
|
||||
replicaTwo.beginReplicationFromPeer("host.port1", (byte) 1);
|
||||
|
||||
// allow replication to establish
|
||||
Jvm.pause(250L);
|
||||
|
||||
// access the underlying maps
|
||||
final Map<Long, Long> mapOne = replicaOne.getMap();
|
||||
final Map<Long, Long> mapTwo = replicaTwo.getMap();
|
||||
|
||||
// enter some data into mapOne
|
||||
for (long i = 0; i < 100; i++) {
|
||||
mapOne.put(i, i * 2);
|
||||
}
|
||||
|
||||
// enter some different data into mapTwo
|
||||
for (long i = 100; i < 200; i++) {
|
||||
mapTwo.put(i, i * 7);
|
||||
}
|
||||
|
||||
// wait for replication between nodes (conservative)
|
||||
Jvm.pause(1000L);
|
||||
|
||||
System.out.printf("mapOne has %d entries, expected 200%n", mapOne.size());
|
||||
System.out.printf("mapTwo has %d entries, expected 200%n", mapTwo.size());
|
||||
....
|
||||
|
||||
=== Configuration file
|
||||
The following example uses a basic `yaml` configuration file to define defaults for the map:
|
||||
The following example uses a basic `yaml` configuration file to define clustered replication for the map named `fx`:
|
||||
|
||||
[source, yaml]
|
||||
....
|
||||
!ReplicatedMapCfg {
|
||||
hostId: 1, # host id must be unique within the cluster
|
||||
listenAddressHostPort: "localhost:9301", # address on which to listen for inbound replication requests
|
||||
entries: 10000, # entries hint suggesting the initial size of the map
|
||||
keyClass: !type long, # class of the key type
|
||||
valueClass: !type long, # class of the value type
|
||||
exampleKey: 17, # an example key (used to size the map)
|
||||
exampleValue: 37, # an example value (used to size the map)
|
||||
name: testLongToLongMap, # a name for the map (used to determine filename)
|
||||
locator: "csp://localhost:9301/map/test", # a locator, used for discovery
|
||||
mapFileDataDirectory: target/mapData, # directory where map file should be stored
|
||||
mapLogDirectory: target/mapReplicationLogs, # directory where replication events should be recorded
|
||||
keysAreConstantSize: true, # determines how keys are serialised
|
||||
valuesAreConstantSize: true # determines how values are serialised
|
||||
!MapReplicationCfg {
|
||||
cluster: {
|
||||
host1: {
|
||||
hostId: 1,
|
||||
connectUri: hostport1,
|
||||
},
|
||||
host2: {
|
||||
hostId: 2,
|
||||
connectUri: hostport2,
|
||||
},
|
||||
host3: {
|
||||
hostId: 3,
|
||||
connectUri: hostport3,
|
||||
}
|
||||
},
|
||||
maps: {
|
||||
fx: {
|
||||
entries: 10000,
|
||||
keyClass: !type String,
|
||||
valueClass: !type software.chronicle.enterprise.map.ValueObject,
|
||||
averageKeySize: 64,
|
||||
averageValueSize: 128,
|
||||
mapFileDataDirectory: data/$hostId/,
|
||||
mapLogDirectory: logs/$hostId/,
|
||||
enableReplicationLogging: true
|
||||
},
|
||||
}
|
||||
}
|
||||
....
|
||||
|
||||
After putting 100 entries into each distinct map, the example demonstrates that the entries have been replicated so that each map
|
||||
now contains 200 entries.
|
||||
And below is an example using this configuration file to start up cluster and insert entries in different maps, verifying that
|
||||
all maps are eventually in sync:
|
||||
|
||||
This example is available in the repository, https://github.com/ChronicleEnterprise/Chronicle-Map-Enterprise/blob/master/src/test/java/software/chronicle/enterprise/map/example/MultiMasterMapReplicationExampleMain.java[MultiMasterMapReplicationExampleMain.java]
|
||||
[source, java]
|
||||
....
|
||||
try (ReplicatedMap clusterOnHost1 = createCluster(CLUSTER_YAML, 1);
|
||||
ReplicatedMap clusterOnHost3 = createCluster(CLUSTER_YAML, 3);
|
||||
ReplicatedMap clusterOnHost2 = createCluster(CLUSTER_YAML, 2)) {
|
||||
|
||||
final ChronicleMap<String, ValueObject> mapOnHost1 = clusterOnHost1.getReplicatedMap("fx");
|
||||
final ChronicleMap<String, ValueObject> mapOnHost2 = clusterOnHost2.getReplicatedMap("fx");
|
||||
final ChronicleMap<String, ValueObject> mapOnHost3 = clusterOnHost3.getReplicatedMap("fx");
|
||||
|
||||
mapOnHost1.put("USD/GBP", new ValueObject("BATS", System.currentTimeMillis(), 0.767957));
|
||||
mapOnHost2.put("GBP/USD", new ValueObject("BATS", System.currentTimeMillis(), 1.30216));
|
||||
mapOnHost3.put("EUR/USD", new ValueObject("LXN", System.currentTimeMillis(), 1.16337));
|
||||
|
||||
Jvm.pause(500L);
|
||||
|
||||
printMap("one", mapOnHost1);
|
||||
printMap("two", mapOnHost2);
|
||||
printMap("three", mapOnHost3);
|
||||
}
|
||||
....
|
||||
|
||||
|
||||
This example is available in the repository https://github.com/ChronicleEnterprise/Chronicle-Map-Enterprise-Demo/blob/master/Example1/src/main/java/software/chronicle/enterprise/map/ThreeWayMapReplicationExampleMain.java[here]
|
||||
|
||||
== Replication event logging
|
||||
|
||||
@@ -200,44 +179,10 @@ The example below shows the message flow for a map with a single remote peer rec
|
||||
....
|
||||
.. header omitted
|
||||
|
||||
targetHostId: 2 # message destination
|
||||
registerForReplication: {
|
||||
receiverHostId: 1, # local peer id
|
||||
keyClassName: java.lang.Long, # key type
|
||||
valueClassName: java.lang.Long, # value type
|
||||
sendAllEventsAfter: 0 # bootstrap timestamp
|
||||
}
|
||||
|
||||
..
|
||||
|
||||
== corresponding registration message from remote peer
|
||||
targetHostId: 1
|
||||
registerForReplication: {
|
||||
receiverHostId: 2,
|
||||
keyClassName: java.lang.Long,
|
||||
valueClassName: java.lang.Long,
|
||||
sendAllEventsAfter: 0
|
||||
}
|
||||
|
||||
..
|
||||
|
||||
== accept replication connection
|
||||
targetHostId: 2
|
||||
acceptingReplicationRequest: {
|
||||
acceptingHostId: 1
|
||||
}
|
||||
|
||||
..
|
||||
|
||||
== encoded replication update sent to remote peer
|
||||
targetHostId: 2
|
||||
replicatedEntry: !!binary AYChq5LqwKXqFAFOEwAAAAAAAAD/////////fw==
|
||||
....
|
||||
|
||||
This output can be generated by running the example program, https://github.com/ChronicleEnterprise/Chronicle-Map-Enterprise/blob/master/src/test/java/software/chronicle/enterprise/map/example/ReplicationEventLoggingExampleMain.java[ReplicationEventLoggingExampleMain.java]
|
||||
|
||||
'''
|
||||
For the replication stress test, see <<CM_Replication_Stress_Test.adoc#,CM_Replication_Stress_Test>>.
|
||||
|
||||
'''
|
||||
<<CM_Features.adoc#,Back to Features>>
|
||||
|
||||
@@ -1,284 +0,0 @@
|
||||
= Replication Stress test
|
||||
:toc: macro
|
||||
:toclevels: 1
|
||||
:css-signature: demo
|
||||
:toc-placement: macro
|
||||
:icons: font
|
||||
|
||||
toc::[]
|
||||
|
||||
The following stress test is provided for Chronicle Map Enterprise Edition. The test can be configured to run across multiple machines, with varying read/write workloads.
|
||||
|
||||
NOTE: This document assumes that you have access to the restricted https://github.com/ChronicleEnterprise/Chronicle-Map-Enterprise-Demo[Chronicle-Map-Enterprise-Demo] repository.
|
||||
|
||||
== Single master, multiple slaves
|
||||
|
||||
To execute a test, open the the `Example4` folder, and then execute the following command from a `bash` shell:
|
||||
|
||||
[source,bash]
|
||||
....
|
||||
$ ./run_stress_test.sh -c src/main/resources/singleMasterStressTest.yaml
|
||||
....
|
||||
|
||||
This command runs a test, using the specified configuration, which models a single master
|
||||
node working with three slave nodes. All nodes run within the same JVM for simplicity.
|
||||
|
||||
The test reports throughput metrics in terms of reads, writes, and network traffic. For example:
|
||||
|
||||
[source,bash]
|
||||
....
|
||||
Write stress-test-1: 237 inserts/sec, 702 updates/sec (total 940/sec)
|
||||
Read stress-test-2: 2175 reads/sec, 0.00% were null
|
||||
Read stress-test-3: 2177 reads/sec, 0.00% were null
|
||||
Read stress-test-4: 2176 reads/sec, 0.00% were null
|
||||
Node 3 received 1061 events/sec, 151KB/sec from remote 1
|
||||
Node 2 received 1062 events/sec, 151KB/sec from remote 1
|
||||
....
|
||||
|
||||
Every minute, the test pauses the writer, and checks that all maps within the cluster
|
||||
contain the same data; that is, they all have the same `keySet`, and the value for each key is
|
||||
identical throughout the cluster. For example:
|
||||
|
||||
[source,bash]
|
||||
....
|
||||
Starting map checker due to time of 11:37:30.035
|
||||
Checking maps for equality
|
||||
All maps contained the same data
|
||||
....
|
||||
|
||||
== Running the test in multiple JVMs
|
||||
|
||||
To spread the workload over multiple JVMs, use the `-h` parameter, which specifies the nodes that should be started in a particular JVM. The `singleMasterStressTest.yaml` configuration file contains 4 nodes (1 master, 3 slaves). The example below spreads these nodes across two processes.
|
||||
|
||||
. On one terminal, run the following command:
|
||||
|
||||
[source,bash]
|
||||
....
|
||||
$ ./run_stress_test.sh -c src/main/resources/singleMasterStressTest.yaml -h 1 -h 2
|
||||
....
|
||||
|
||||
+
|
||||
then wait for the message notifying that the test is starting:
|
||||
|
||||
[source,bash]
|
||||
....
|
||||
Built stress test artifacts, running test..
|
||||
....
|
||||
|
||||
. On another terminal, run the following command:
|
||||
|
||||
[source,bash]
|
||||
....
|
||||
$ ./run_stress_test.sh -c src/main/resources/singleMasterStressTest.yaml -h 3 -h 4
|
||||
....
|
||||
|
||||
+
|
||||
|
||||
In both terminals, throughput and map-equality checking will be reported.
|
||||
|
||||
== Multiple masters, multiple slaves
|
||||
|
||||
To test a multi-master scenario, where several different nodes are written to, execute the following command:
|
||||
|
||||
[source,bash]
|
||||
....
|
||||
./run_stress_test.sh -c src/main/resources/multipleMasterStressTest.yaml
|
||||
....
|
||||
|
||||
== Test configuration
|
||||
|
||||
Stress test scenarios are configured using the supplied `yaml` file. Using the examples
|
||||
included in this repository as a template, it is possible to configure custom scenarios
|
||||
to model real-world workloads.
|
||||
|
||||
The configuration is comprised of two components:
|
||||
|
||||
1. Test parameters
|
||||
2. Cluster configuration
|
||||
|
||||
Test parameters are described below:
|
||||
|
||||
[source,yaml]
|
||||
....
|
||||
!TestAndClusterConfig {
|
||||
testConfig: !StressTestConfig {
|
||||
numberOfSlaveNodes: 3, # map instances that are read from
|
||||
numberOfMasterNodes: 1, # map instances that are written to
|
||||
updatesPerSecond: 1000, # writes per instance per second
|
||||
queriesPerSecond: 2500, # reads per instance per second
|
||||
insertToUpdateRatio: 0.25, # ratio of remove/insert vs update operations
|
||||
keySpaceSize: 5000, # size of the key space (uniform random integers)
|
||||
valueSizeInBytes: 128, # payload size of value object
|
||||
testLengthMinutes: 2 # duration of a test
|
||||
},
|
||||
....
|
||||
|
||||
== Running tests across multiple physical machines
|
||||
|
||||
To demonstrate how to run tests across different machines, the following details are defined:
|
||||
|
||||
Hosts within the cluster:
|
||||
|
||||
[frame="topbot",options="header,footer"]
|
||||
|=======================================
|
||||
| host | host_id | IP | role
|
||||
| A | 1 |10.0.2.1 | master
|
||||
| B | 2 |10.0.2.14 | master
|
||||
| C | 3 |10.0.2.30 | slave
|
||||
| D | 4 |10.0.2.22 | slave
|
||||
|=======================================
|
||||
|
||||
NOTE: Substitute your own settings as appropriate.
|
||||
|
||||
**TCP port used for replication**; this port will need to be open on all hosts:
|
||||
|
||||
`13000`
|
||||
|
||||
**Configuration file name**:
|
||||
|
||||
`multiHost.yaml`
|
||||
|
||||
The cluster configuration in `multiHost.yaml` would be:
|
||||
|
||||
[source,yaml]
|
||||
....
|
||||
clusterConfig: !Clusters {
|
||||
myCluster: {
|
||||
config: !ReplicatedMapCfg {
|
||||
entries: 10000,
|
||||
keyClass: !type long,
|
||||
exampleKey: 17,
|
||||
name: stress-test-map,
|
||||
keysAreConstantSize: true,
|
||||
valuesAreConstantSize: false,
|
||||
keyDataAccess: !LongDataAccess {
|
||||
},
|
||||
keyReader: !LongMarshaller {
|
||||
},
|
||||
},
|
||||
topology: {
|
||||
context: !MapClusterContext {
|
||||
wireType: TEXT,
|
||||
},
|
||||
hostA: {
|
||||
hostId: 1,
|
||||
connectUri: "10.0.2.1:13000"
|
||||
},
|
||||
hostB: {
|
||||
hostId: 2,
|
||||
connectUri: "10.0.2.14:13000"
|
||||
},
|
||||
hostC: {
|
||||
hostId: 3,
|
||||
connectUri: "10.0.2.30:13000"
|
||||
},
|
||||
hostD: {
|
||||
hostId: 4,
|
||||
connectUri: "10.0.2.22:13000"
|
||||
}
|
||||
},
|
||||
monitoring: {
|
||||
replicationEventListener: !LoggingReplicationEventListener
|
||||
}
|
||||
}
|
||||
....
|
||||
|
||||
=== Packaging the test
|
||||
|
||||
To run the test on remote machines, the stress test can be packaged up using the following command:
|
||||
|
||||
[source,bash]
|
||||
....
|
||||
$ ./package_stress_test.sh
|
||||
....
|
||||
|
||||
This command generates a `tar` archive in the current directory, containing everything required to run a stress test.
|
||||
|
||||
Copy the resulting `tar` file to the remote server, unpack using `tar xf`, and then run the following command:
|
||||
|
||||
[source,bash]
|
||||
....
|
||||
./run_stress_test.sh -c [config-file]
|
||||
....
|
||||
|
||||
To run the example multi-machine configuration described above, the following commands would be run on each host:
|
||||
|
||||
[frame="topbot",options="header,footer"]
|
||||
|=======================================
|
||||
| host | host_id | IP | command
|
||||
| **A** | `1` |`10.0.2.1` | `./run_stress_test.sh -c multiHost.yaml -h 1`
|
||||
| **B** | `2` |`10.0.2.14` | `./run_stress_test.sh -c multiHost.yaml -h 2`
|
||||
| **C** | `3` |`10.0.2.30` | `./run_stress_test.sh -c multiHost.yaml -h 3`
|
||||
| **D** | `4` |`10.0.2.22` | `./run_stress_test.sh -c multiHost.yaml -h 4`
|
||||
|=======================================
|
||||
|
||||
== Destructive testing
|
||||
|
||||
There are two ways of recovering from node failure:
|
||||
|
||||
1. Delete previous map data, and rejoin the cluster.
|
||||
|
||||
2. Recover previous map data, and rejoin the cluster.
|
||||
|
||||
If the total dataset contains a large proportion of infrequently updated records, then it is preferable to use option 2.
|
||||
|
||||
If the entire key-set is updated frequently, then there is little difference between the two methods.
|
||||
|
||||
Both scenarios can be simulated using the stress test.
|
||||
|
||||
=== Delete existing data and rejoin (Option 1)
|
||||
|
||||
. Start one or more slave processes:
|
||||
|
||||
[source,bash]
|
||||
....
|
||||
$ ./run_stress_test.sh -c src/main/resources/singleMasterStressTest.yaml -h 3
|
||||
....
|
||||
|
||||
. Start one or more master nodes:
|
||||
|
||||
[source,bash]
|
||||
....
|
||||
$ ./run_stress_test.sh -c src/main/resources/singleMasterStressTest.yaml -h 1
|
||||
....
|
||||
|
||||
. When replication has been established, kill the slave process; for example, by using `Ctrl+c`.
|
||||
|
||||
. Restart the slave process (see step 1).
|
||||
By default, the stress test code deletes existing map data files for the specified hosts
|
||||
on startup.
|
||||
|
||||
[source,bash]
|
||||
....
|
||||
$ ./run_stress_test.sh -c src/main/resources/singleMasterStressTest.yaml -h 3
|
||||
....
|
||||
|
||||
+
|
||||
There should be a noticable difference in throughput after the restart, as
|
||||
the slave map receives the entire data-set from the master:
|
||||
|
||||
[source,bash]
|
||||
....
|
||||
# usual operation
|
||||
Node 3 received 1153 events/sec, 153KB/sec from remote 1
|
||||
|
||||
# slave process killed and restarted
|
||||
|
||||
# higher throughput as slave catches up
|
||||
Node 3 received 1646 events/sec, 234KB/sec from remote 1
|
||||
....
|
||||
|
||||
=== Recover existing data and rejoin (Option 2)
|
||||
|
||||
The stress test can be configured, using the `-r` parameter, to recover from existing data on startup.
|
||||
|
||||
Perform the same procedure as in Option 1, step 4, but this time when restarting the slave client use `-r`:
|
||||
|
||||
[source,bash]
|
||||
....
|
||||
$ ./run_stress_test.sh -c src/main/resources/singleMasterStressTest.yaml -h 3 -r
|
||||
....
|
||||
|
||||
This ensures that the existing map is recovered from disk. This may save network traffic, depending on how much data has been updated in the master map during downtime.
|
||||
|
||||
<<CM_Replication.adoc#,Back to Replication>>
|
||||
Reference in New Issue
Block a user