diff --git a/.gitignore b/.gitignore index bc0ad52f1..a40e8a1f9 100644 --- a/.gitignore +++ b/.gitignore @@ -1,6 +1,7 @@ ### How to update # This is copied from OpenHFT/.gitignore # update the original and run OpenHFT/update_gitignore.sh +data/ ### Compiled class file *.class diff --git a/docs/CM_Replication.adoc b/docs/CM_Replication.adoc index ae4ce11a6..f96fbc350 100755 --- a/docs/CM_Replication.adoc +++ b/docs/CM_Replication.adoc @@ -18,26 +18,15 @@ In addition, you will be fully supported by our technical experts. For more information on Chronicle Map (Enterprise Edition), please contact mailto:sales@chronicle.software[sales@chronicle.software]. -To ensure that Chronicle Map's multi-master replication works under extreme circumstances, we have -built a stress test to ensure this using the principles introduced by Netflix with their "chaos monkey". -For more information on the stress test see <>. - = Replication -Chronicle Map Replication is a multi-master replicated data store. Chronicle Map supports both TCP and UDP replication +Chronicle Map Replication is a multi-master replicated data store. Chronicle Map supports replication over TCP/IP image::http://openhft.net/wp-content/uploads/2014/07/Chronicle-Map-TCP-Replication_simple_02.jpg[TCP/IP Replication] -== TCP / UDP Background +== TCP Background TCP/IP is a reliable protocol. This means that, unless you have a network failure or hardware outage, the data is guaranteed to arrive. TCP/IP provides point-to-point connectivity. For example, if the message was sent to 100 hosts, the message would have to be sent 100 times. -With UDP, the message is only sent once. This is ideal if you have a large number of hosts, and you wish to broadcast the same data to each of them. However, one of the big drawbacks with UDP is that it is not a reliable protocol. This means, if the UDP message is broadcast onto the network, the hosts are not guaranteed to receive it; so they can miss data. - -Some solutions attempt to build resilience into UDP, but arguably, this is in effect reinventing TCP/IP. - -== How to setup UDP Replication -On a good quality wired LAN, when using UDP, you will rarely miss messages. Nevertheless, this is a risk that we suggest you do not take. We suggest that whenever you use UDP replication, you use it in conjunction with throttled TCP replication. Therefore, if a host misses a message over UDP, it will pick it up later via TCP/IP. - == TCP/IP Throttling We are careful not to swamp your network with too much TCP/IP traffic. We do this by providing a throttled version of TCP replication. This works because Chronicle Map only broadcasts the latest update of each entry. @@ -114,79 +103,69 @@ map = ChronicleMapBuilder == Configuration -Configuration of map nodes is done either, programmatically, through configuration files, or by a combination of both. +Configuration of map nodes is done either, by creating configuration programmatically, or through YAML configuration files. -=== Programmatically -The example below shows how to set up two map instances that use TCP to replicate entries between them: - -[source, java] -.... - -// set up some socket aliases -TCPRegistry.createServerSocketChannelFor("host.port1", "host.port2"); - -// create replicated maps -final ReplicatedMap replicaOne = createMap(1, "host.port1"); -final ReplicatedMap replicaTwo = createMap(2, "host.port2"); - -// start replication event loops -replicaOne.start(); -replicaTwo.start(); - -// register maps for replication events -replicaOne.beginReplicationFromPeer("host.port2", (byte) 2); -replicaTwo.beginReplicationFromPeer("host.port1", (byte) 1); - -// allow replication to establish -Jvm.pause(250L); - -// access the underlying maps -final Map mapOne = replicaOne.getMap(); -final Map mapTwo = replicaTwo.getMap(); - -// enter some data into mapOne -for (long i = 0; i < 100; i++) { - mapOne.put(i, i * 2); -} - -// enter some different data into mapTwo -for (long i = 100; i < 200; i++) { - mapTwo.put(i, i * 7); -} - -// wait for replication between nodes (conservative) -Jvm.pause(1000L); - -System.out.printf("mapOne has %d entries, expected 200%n", mapOne.size()); -System.out.printf("mapTwo has %d entries, expected 200%n", mapTwo.size()); -.... - -=== Configuration file -The following example uses a basic `yaml` configuration file to define defaults for the map: +The following example uses a basic `yaml` configuration file to define clustered replication for the map named `fx`: [source, yaml] .... -!ReplicatedMapCfg { - hostId: 1, # host id must be unique within the cluster - listenAddressHostPort: "localhost:9301", # address on which to listen for inbound replication requests - entries: 10000, # entries hint suggesting the initial size of the map - keyClass: !type long, # class of the key type - valueClass: !type long, # class of the value type - exampleKey: 17, # an example key (used to size the map) - exampleValue: 37, # an example value (used to size the map) - name: testLongToLongMap, # a name for the map (used to determine filename) - locator: "csp://localhost:9301/map/test", # a locator, used for discovery - mapFileDataDirectory: target/mapData, # directory where map file should be stored - mapLogDirectory: target/mapReplicationLogs, # directory where replication events should be recorded - keysAreConstantSize: true, # determines how keys are serialised - valuesAreConstantSize: true # determines how values are serialised +!MapReplicationCfg { + cluster: { + host1: { + hostId: 1, + connectUri: hostport1, + }, + host2: { + hostId: 2, + connectUri: hostport2, + }, + host3: { + hostId: 3, + connectUri: hostport3, + } + }, + maps: { + fx: { + entries: 10000, + keyClass: !type String, + valueClass: !type software.chronicle.enterprise.map.ValueObject, + averageKeySize: 64, + averageValueSize: 128, + mapFileDataDirectory: data/$hostId/, + mapLogDirectory: logs/$hostId/, + enableReplicationLogging: true + }, + } } .... -After putting 100 entries into each distinct map, the example demonstrates that the entries have been replicated so that each map -now contains 200 entries. +And below is an example using this configuration file to start up cluster and insert entries in different maps, verifying that +all maps are eventually in sync: -This example is available in the repository, https://github.com/ChronicleEnterprise/Chronicle-Map-Enterprise/blob/master/src/test/java/software/chronicle/enterprise/map/example/MultiMasterMapReplicationExampleMain.java[MultiMasterMapReplicationExampleMain.java] +[source, java] +.... +try (ReplicatedMap clusterOnHost1 = createCluster(CLUSTER_YAML, 1); + ReplicatedMap clusterOnHost3 = createCluster(CLUSTER_YAML, 3); + ReplicatedMap clusterOnHost2 = createCluster(CLUSTER_YAML, 2)) { + + final ChronicleMap mapOnHost1 = clusterOnHost1.getReplicatedMap("fx"); + final ChronicleMap mapOnHost2 = clusterOnHost2.getReplicatedMap("fx"); + final ChronicleMap mapOnHost3 = clusterOnHost3.getReplicatedMap("fx"); + + mapOnHost1.put("USD/GBP", new ValueObject("BATS", System.currentTimeMillis(), 0.767957)); + mapOnHost2.put("GBP/USD", new ValueObject("BATS", System.currentTimeMillis(), 1.30216)); + mapOnHost3.put("EUR/USD", new ValueObject("LXN", System.currentTimeMillis(), 1.16337)); + + Jvm.pause(500L); + + printMap("one", mapOnHost1); + printMap("two", mapOnHost2); + printMap("three", mapOnHost3); +} +.... + + +This example is available in the repository https://github.com/ChronicleEnterprise/Chronicle-Map-Enterprise-Demo/blob/master/Example1/src/main/java/software/chronicle/enterprise/map/ThreeWayMapReplicationExampleMain.java[here] == Replication event logging @@ -200,44 +179,10 @@ The example below shows the message flow for a map with a single remote peer rec .... .. header omitted -targetHostId: 2 # message destination -registerForReplication: { - receiverHostId: 1, # local peer id - keyClassName: java.lang.Long, # key type - valueClassName: java.lang.Long, # value type - sendAllEventsAfter: 0 # bootstrap timestamp -} - -.. - -== corresponding registration message from remote peer -targetHostId: 1 -registerForReplication: { - receiverHostId: 2, - keyClassName: java.lang.Long, - valueClassName: java.lang.Long, - sendAllEventsAfter: 0 -} - -.. - -== accept replication connection -targetHostId: 2 -acceptingReplicationRequest: { - acceptingHostId: 1 -} - -.. - == encoded replication update sent to remote peer targetHostId: 2 replicatedEntry: !!binary AYChq5LqwKXqFAFOEwAAAAAAAAD/////////fw== .... -This output can be generated by running the example program, https://github.com/ChronicleEnterprise/Chronicle-Map-Enterprise/blob/master/src/test/java/software/chronicle/enterprise/map/example/ReplicationEventLoggingExampleMain.java[ReplicationEventLoggingExampleMain.java] - -''' -For the replication stress test, see <>. - ''' <> diff --git a/docs/CM_Replication_Stress_Test.adoc b/docs/CM_Replication_Stress_Test.adoc deleted file mode 100755 index 0c4159907..000000000 --- a/docs/CM_Replication_Stress_Test.adoc +++ /dev/null @@ -1,284 +0,0 @@ -= Replication Stress test -:toc: macro -:toclevels: 1 -:css-signature: demo -:toc-placement: macro -:icons: font - -toc::[] - -The following stress test is provided for Chronicle Map Enterprise Edition. The test can be configured to run across multiple machines, with varying read/write workloads. - -NOTE: This document assumes that you have access to the restricted https://github.com/ChronicleEnterprise/Chronicle-Map-Enterprise-Demo[Chronicle-Map-Enterprise-Demo] repository. - -== Single master, multiple slaves - -To execute a test, open the the `Example4` folder, and then execute the following command from a `bash` shell: - -[source,bash] -.... -$ ./run_stress_test.sh -c src/main/resources/singleMasterStressTest.yaml -.... - -This command runs a test, using the specified configuration, which models a single master -node working with three slave nodes. All nodes run within the same JVM for simplicity. - -The test reports throughput metrics in terms of reads, writes, and network traffic. For example: - -[source,bash] -.... -Write stress-test-1: 237 inserts/sec, 702 updates/sec (total 940/sec) -Read stress-test-2: 2175 reads/sec, 0.00% were null -Read stress-test-3: 2177 reads/sec, 0.00% were null -Read stress-test-4: 2176 reads/sec, 0.00% were null -Node 3 received 1061 events/sec, 151KB/sec from remote 1 -Node 2 received 1062 events/sec, 151KB/sec from remote 1 -.... - -Every minute, the test pauses the writer, and checks that all maps within the cluster -contain the same data; that is, they all have the same `keySet`, and the value for each key is -identical throughout the cluster. For example: - -[source,bash] -.... -Starting map checker due to time of 11:37:30.035 -Checking maps for equality -All maps contained the same data -.... - -== Running the test in multiple JVMs - -To spread the workload over multiple JVMs, use the `-h` parameter, which specifies the nodes that should be started in a particular JVM. The `singleMasterStressTest.yaml` configuration file contains 4 nodes (1 master, 3 slaves). The example below spreads these nodes across two processes. - -. On one terminal, run the following command: - - [source,bash] - .... - $ ./run_stress_test.sh -c src/main/resources/singleMasterStressTest.yaml -h 1 -h 2 - .... - -+ -then wait for the message notifying that the test is starting: - - [source,bash] - .... - Built stress test artifacts, running test.. - .... - -. On another terminal, run the following command: - - [source,bash] - .... - $ ./run_stress_test.sh -c src/main/resources/singleMasterStressTest.yaml -h 3 -h 4 - .... - -+ - -In both terminals, throughput and map-equality checking will be reported. - -== Multiple masters, multiple slaves - -To test a multi-master scenario, where several different nodes are written to, execute the following command: - -[source,bash] -.... -./run_stress_test.sh -c src/main/resources/multipleMasterStressTest.yaml -.... - -== Test configuration - -Stress test scenarios are configured using the supplied `yaml` file. Using the examples -included in this repository as a template, it is possible to configure custom scenarios -to model real-world workloads. - -The configuration is comprised of two components: - -1. Test parameters -2. Cluster configuration - -Test parameters are described below: - -[source,yaml] -.... -!TestAndClusterConfig { - testConfig: !StressTestConfig { - numberOfSlaveNodes: 3, # map instances that are read from - numberOfMasterNodes: 1, # map instances that are written to - updatesPerSecond: 1000, # writes per instance per second - queriesPerSecond: 2500, # reads per instance per second - insertToUpdateRatio: 0.25, # ratio of remove/insert vs update operations - keySpaceSize: 5000, # size of the key space (uniform random integers) - valueSizeInBytes: 128, # payload size of value object - testLengthMinutes: 2 # duration of a test - }, -.... - -== Running tests across multiple physical machines - -To demonstrate how to run tests across different machines, the following details are defined: - -Hosts within the cluster: - -[frame="topbot",options="header,footer"] -|======================================= -| host | host_id | IP | role -| A | 1 |10.0.2.1 | master -| B | 2 |10.0.2.14 | master -| C | 3 |10.0.2.30 | slave -| D | 4 |10.0.2.22 | slave -|======================================= - -NOTE: Substitute your own settings as appropriate. - -**TCP port used for replication**; this port will need to be open on all hosts: - -`13000` - -**Configuration file name**: - -`multiHost.yaml` - -The cluster configuration in `multiHost.yaml` would be: - -[source,yaml] -.... - clusterConfig: !Clusters { - myCluster: { - config: !ReplicatedMapCfg { - entries: 10000, - keyClass: !type long, - exampleKey: 17, - name: stress-test-map, - keysAreConstantSize: true, - valuesAreConstantSize: false, - keyDataAccess: !LongDataAccess { - }, - keyReader: !LongMarshaller { - }, - }, - topology: { - context: !MapClusterContext { - wireType: TEXT, - }, - hostA: { - hostId: 1, - connectUri: "10.0.2.1:13000" - }, - hostB: { - hostId: 2, - connectUri: "10.0.2.14:13000" - }, - hostC: { - hostId: 3, - connectUri: "10.0.2.30:13000" - }, - hostD: { - hostId: 4, - connectUri: "10.0.2.22:13000" - } - }, - monitoring: { - replicationEventListener: !LoggingReplicationEventListener - } - } -.... - -=== Packaging the test - -To run the test on remote machines, the stress test can be packaged up using the following command: - -[source,bash] -.... -$ ./package_stress_test.sh -.... - -This command generates a `tar` archive in the current directory, containing everything required to run a stress test. - -Copy the resulting `tar` file to the remote server, unpack using `tar xf`, and then run the following command: - -[source,bash] -.... -./run_stress_test.sh -c [config-file] -.... - -To run the example multi-machine configuration described above, the following commands would be run on each host: - -[frame="topbot",options="header,footer"] -|======================================= -| host | host_id | IP | command -| **A** | `1` |`10.0.2.1` | `./run_stress_test.sh -c multiHost.yaml -h 1` -| **B** | `2` |`10.0.2.14` | `./run_stress_test.sh -c multiHost.yaml -h 2` -| **C** | `3` |`10.0.2.30` | `./run_stress_test.sh -c multiHost.yaml -h 3` -| **D** | `4` |`10.0.2.22` | `./run_stress_test.sh -c multiHost.yaml -h 4` -|======================================= - -== Destructive testing - -There are two ways of recovering from node failure: - -1. Delete previous map data, and rejoin the cluster. - -2. Recover previous map data, and rejoin the cluster. - -If the total dataset contains a large proportion of infrequently updated records, then it is preferable to use option 2. - -If the entire key-set is updated frequently, then there is little difference between the two methods. - -Both scenarios can be simulated using the stress test. - -=== Delete existing data and rejoin (Option 1) - -. Start one or more slave processes: - - [source,bash] - .... - $ ./run_stress_test.sh -c src/main/resources/singleMasterStressTest.yaml -h 3 - .... - -. Start one or more master nodes: - - [source,bash] - .... - $ ./run_stress_test.sh -c src/main/resources/singleMasterStressTest.yaml -h 1 - .... - -. When replication has been established, kill the slave process; for example, by using `Ctrl+c`. - -. Restart the slave process (see step 1). -By default, the stress test code deletes existing map data files for the specified hosts -on startup. - - [source,bash] - .... - $ ./run_stress_test.sh -c src/main/resources/singleMasterStressTest.yaml -h 3 - .... - -+ -There should be a noticable difference in throughput after the restart, as -the slave map receives the entire data-set from the master: - - [source,bash] - .... - # usual operation - Node 3 received 1153 events/sec, 153KB/sec from remote 1 - - # slave process killed and restarted - - # higher throughput as slave catches up - Node 3 received 1646 events/sec, 234KB/sec from remote 1 - .... - -=== Recover existing data and rejoin (Option 2) - -The stress test can be configured, using the `-r` parameter, to recover from existing data on startup. - -Perform the same procedure as in Option 1, step 4, but this time when restarting the slave client use `-r`: - -[source,bash] -.... -$ ./run_stress_test.sh -c src/main/resources/singleMasterStressTest.yaml -h 3 -r -.... - -This ensures that the existing map is recovered from disk. This may save network traffic, depending on how much data has been updated in the master map during downtime. - -<> \ No newline at end of file