diff --git a/ReadMe.adoc b/ReadMe.adoc index 07c07dfb1..0892e2afb 100755 --- a/ReadMe.adoc +++ b/ReadMe.adoc @@ -6,6 +6,13 @@ Chronicle Map can overflow seamlessly to disk, and can be distributed across the image::docs\images\CM_Overview.jpg[] +== Prerequisites + +To run Chronicle Map, please ensure you have first installed: + +* Java (1.8.0_91 or later) +* Apache Maven (3.2.3 or later) + === Use cases Chronicle Map is used in production around the world for: @@ -58,13 +65,22 @@ The Chronicle Map documentation comprises: .Documentation |=== |**Document**|**Purpose** -|<>|Features description. -|<>|Replication explanation. -|<>|Tutorial. -|<>|Frequently asked questions. -|<>|Downloading the software. -|<>|Updates from Chronicle Map version 2. -|<>|Compatibility and Versioning description. +|<> +|Features description. +|<> +|Tutorial. +|<> +|Frequently asked questions. +|<> +|Downloading the software. +|<> +|Updates from Chronicle Map version 2. +|<> +|Replication explanation. +|<> +|Example Replication stress test. +|<> +|Compatibility and Versioning description. |=== Linked documentation is contained in the `docs` folder. \ No newline at end of file diff --git a/docs/CM_Replication.adoc b/docs/CM_Replication.adoc index 4cf9746c9..1ba5df02a 100755 --- a/docs/CM_Replication.adoc +++ b/docs/CM_Replication.adoc @@ -1,4 +1,4 @@ -= Chronicle Map Replication += Chronicle Map Replication (Enterprise Edition feature) Neil Clifford :toc: macro :toclevels: 1 @@ -8,8 +8,10 @@ Neil Clifford toc::[] -Chronicle Map supports both TCP and UDP replication in the closed-source Chronicle Map Enterprise. -For more information on this please contact mailto:sales@chronicle.software. + +Chronicle Map Replication is a multi-master replicated data store. + +Chronicle Map supports both TCP and UDP replication image::http://openhft.net/wp-content/uploads/2014/07/Chronicle-Map-TCP-Replication_simple_02.jpg[TCP/IP Replication] @@ -92,3 +94,135 @@ map = ChronicleMapBuilder .replication(identifier, tcpConfig) .create(); ``` + +== Configuration + +Configuration of map nodes is done either programmatically, through configuration files, or a combination of both. + +The example below shows how to set up two map instances that use TCP to replicate entries between them: + +[source, java] +.... + +// set up some socket aliases +TCPRegistry.createServerSocketChannelFor("host.port1", "host.port2"); + +// create replicated maps +final ReplicatedMap replicaOne = createMap(1, "host.port1"); +final ReplicatedMap replicaTwo = createMap(2, "host.port2"); + +// start replication event loops +replicaOne.start(); +replicaTwo.start(); + +// register maps for replication events +replicaOne.beginReplicationFromPeer("host.port2", (byte) 2); +replicaTwo.beginReplicationFromPeer("host.port1", (byte) 1); + +// allow replication to establish +Jvm.pause(250L); + +// access the underlying maps +final Map mapOne = replicaOne.getMap(); +final Map mapTwo = replicaTwo.getMap(); + +// enter some data into mapOne +for (long i = 0; i < 100; i++) { + mapOne.put(i, i * 2); +} + +// enter some different data into mapTwo +for (long i = 100; i < 200; i++) { + mapTwo.put(i, i * 7); +} + +// wait for replication between nodes (conservative) +Jvm.pause(1000L); + +System.out.printf("mapOne has %d entries, expected 200%n", mapOne.size()); +System.out.printf("mapTwo has %d entries, expected 200%n", mapTwo.size()); +.... + +The following example uses a basic `yaml` configuration file to define defaults for the map: + +[source, yaml] +.... +!ReplicatedMapCfg { + +hostId: 1, # host id must be unique within the cluster +listenAddressHostPort: "localhost:9301", # address on which to listen for inbound replication requests +entries: 10000, # entries hint suggesting the initial size of the map +keyClass: !type long, # class of the key type +valueClass: !type long, # class of the value type +exampleKey: 17, # an example key (used to size the map) +exampleValue: 37, # an example value (used to size the map) +name: testLongToLongMap, # a name for the map (used to determine filename) +locator: "csp://localhost:9301/map/test", # a locator, used for discovery +mapFileDataDirectory: target/mapData, # directory where map file should be stored +mapLogDirectory: target/mapReplicationLogs, # directory where replication events should be recorded +keysAreConstantSize: true, # determines how keys are serialised +valuesAreConstantSize: true # determines how values are serialised + +} +.... + +After putting 100 entries into each distinct map, the example demonstrates that the entries have been replicated so that each map +now contains 200 entries. + +This example is available in the repository, link::https://github.com/ChronicleEnterprise/Chronicle-Map-Enterprise/blob/master/src/test/java/software/chronicle/enterprise/map/example/MultiMasterMapReplicationMain.java[MultiMasterMapReplicationMain.java] + + +== Replication event logging + +Chronicle Map Enterprise can be configured to log all replication events to a Chronicle Queue for auditing purposes. + +Currently, a map can be configured to log all *outgoing* events that it sends to remote peers. + +The example below shows the message flow for a map with a single remote peer receiving replication events: + +[source, java] +.... +.. header omitted + +targetHostId: 2 # message destination +registerForReplication: { + receiverHostId: 1, # local peer id + keyClassName: java.lang.Long, # key type + valueClassName: java.lang.Long, # value type + sendAllEventsAfter: 0 # bootstrap timestamp +} + +.. + +# corresponding registration message from remote peer +targetHostId: 1 +registerForReplication: { + receiverHostId: 2, + keyClassName: java.lang.Long, + valueClassName: java.lang.Long, + sendAllEventsAfter: 0 +} + +.. + +# accept replication connection +targetHostId: 2 +acceptingReplicationRequest: { + acceptingHostId: 1 +} + +.. + +# encoded replication update sent to remote peer +targetHostId: 2 +replicatedEntry: !!binary AYChq5LqwKXqFAFOEwAAAAAAAAD/////////fw== +.... + +This output can be generated by running the example program, +link::https://github.com/ChronicleEnterprise/Chronicle-Map-Enterprise/blob/master/src/test/java/software/chronicle/enterprise/map/example/ReplicationEventLoggingExampleMain.java[ReplicationEventLoggingExampleMain.java] + +''' +For an example replication stress test, see <<../CM_Replication_Stress_Test.adoc#,CM_Replication_Stress_Test.adoc>>. + +''' +<<../Features.adoc#,Back to Features>> diff --git a/docs/CM_Replication_Stress_Test.adoc b/docs/CM_Replication_Stress_Test.adoc new file mode 100755 index 000000000..0e77cbfaa --- /dev/null +++ b/docs/CM_Replication_Stress_Test.adoc @@ -0,0 +1,286 @@ += Replication Stress test +:toc: macro +:toclevels: 1 +:css-signature: demo +:toc-placement: macro +:icons: font + +toc::[] + +The following stress test is provided for Chronicle Map Enterprise Edition. The test can be configured to run across multiple machines, with varying read/write workloads. + +NOTE: This document assumes that you have access to the restricted link::https://github.com/ChronicleEnterprise/Chronicle-Map-Enterprise-Demo[Chronicle-Map-Enterprise-Demo] repository. + +== Single master, multiple slaves + +To execute a test, open the the `Example4` directory, then execute the following command from a `bash` shell: + +[source,bash] +.... +$ ./run_stress_test.sh -c src/main/resources/singleMasterStressTest.yaml +.... + +This command will run a test using the specified configuration, which models a single master +node along with three slave nodes. All nodes run within the same JVM for simplicity. + +The test will report throughput metrics in terms of reads, writes, and network traffic. For example: + +[source,bash] +.... +Write stress-test-1: 237 inserts/sec, 702 updates/sec (total 940/sec) +Read stress-test-2: 2175 reads/sec, 0.00% were null +Read stress-test-3: 2177 reads/sec, 0.00% were null +Read stress-test-4: 2176 reads/sec, 0.00% were null +Node 3 received 1061 events/sec, 151KB/sec from remote 1 +Node 2 received 1062 events/sec, 151KB/sec from remote 1 +.... + +Every minute, the test will pause the writer and check that all maps within the cluster +contain the same data; that is, they all have the same `keySet`, and the value for each key is +identical throughout the cluster. For example: + +[source,bash] +.... +Starting map checker due to time of 11:37:30.035 +Checking maps for equality +All maps contained the same data +.... + + +== Running the test in multiple JVMs + +To spread the workload over multiple JVMs, use the `-h` parameter, which specifies the nodes that should be started in a particular JVM. The `singleMasterStressTest.yaml` configuration file contains 4 nodes (1 master, 3 slaves). The example below spreads these nodes across two processes. + +In one terminal, run the following command: + +[source,bash] +.... +$ ./run_stress_test.sh -c src/main/resources/singleMasterStressTest.yaml -h 1 -h 2 +.... + +then wait for the message notifying that the test is starting: + +[source,bash] +.... +Built stress test artifacts, running test.. +.... + +Now in another terminal, run the following command: + +[source,bash] +.... +$ ./run_stress_test.sh -c src/main/resources/singleMasterStressTest.yaml -h 3 -h 4 +.... + +In both terminals, throughput and map-equality checking will be reported. + +== Multiple masters, multiple slaves + +To test a multi-master scenario, where several different nodes are written to, execute the following command: + +[source,bash] +.... +./run_stress_test.sh -c src/main/resources/multipleMasterStressTest.yaml +.... + +== Test configuration + +Stress test scenarios are configured using the supplied `yaml` file. Using the examples +included in this repository as a template, it is possible to configure custom scenarios +to model real-world workloads. + +The configuration is comprised of two components: + +1. Test parameters +2. Cluster configuration + +Test parameters are described below: + +[source,yaml] +.... +!TestAndClusterConfig { + testConfig: !StressTestConfig { + numberOfSlaveNodes: 3, # map instances that are read from + numberOfMasterNodes: 1, # map instances that are written to + updatesPerSecond: 1000, # writes per instance per second + queriesPerSecond: 2500, # reads per instance per second + insertToUpdateRatio: 0.25, # ratio of remove/insert vs update operations + keySpaceSize: 5000, # size of the key space (uniform random integers) + valueSizeInBytes: 128, # payload size of value object + testLengthMinutes: 2 # duration of a test + }, +.... + +== Running tests across multiple physical machines + +To demonstrate how to run tests across different machines, the following details are defined: +Hosts within the cluster: + +[frame="topbot",options="header,footer"] +|======================================= +| host | host_id | IP | role +| A | 1 |10.0.2.1 | master +| B | 2 |10.0.2.14 | master +| C | 3 |10.0.2.30 | slave +| D | 4 |10.0.2.22 | slave +|======================================= + +NOTE: Substitute your own settings as appropriate. + + +TCP port used for replication; this port will need to be open on all hosts: + +`13000` + +Configuration file name: + +`multiHost.yaml` + + +The cluster configuration in `multiHost.yaml` would be: + + +[source,yaml] +.... + clusterConfig: !Clusters { + myCluster: { + config: !ReplicatedMapCfg { + entries: 10000, + keyClass: !type long, + exampleKey: 17, + name: stress-test-map, + keysAreConstantSize: true, + valuesAreConstantSize: false, + keyDataAccess: !LongDataAccess { + }, + keyReader: !LongMarshaller { + }, + }, + topology: { + context: !MapClusterContext { + wireType: TEXT, + }, + hostA: { + hostId: 1, + connectUri: "10.0.2.1:13000" + }, + hostB: { + hostId: 2, + connectUri: "10.0.2.14:13000" + }, + hostC: { + hostId: 3, + connectUri: "10.0.2.30:13000" + }, + hostD: { + hostId: 4, + connectUri: "10.0.2.22:13000" + } + }, + monitoring: { + replicationEventListener: !LoggingReplicationEventListener + } + } +.... + + +=== Packaging the test + +To run the test on remote machines, the stress test can be packaged up using the following command: + +[source,bash] +.... +$ ./package_stress_test.sh +.... + +This command will generate a `tar` archive in the current directory, containing everything required to run a stress test. + +Copy the resulting `tar` file to the remote server, unpack using `tar xf`, and then run the following command: + +[source,bash] +.... +./run_stress_test.sh -c [config-file] +.... + +To run the example multi-machine configuration described above, the following commands would be run on each host: + +[frame="topbot",options="header,footer"] +|======================================= +| host | host_id | IP | command +| A | 1 |10.0.2.1 | ./run_stress_test.sh -c multiHost.yaml -h 1 +| B | 2 |10.0.2.14 | ./run_stress_test.sh -c multiHost.yaml -h 2 +| C | 3 |10.0.2.30 | ./run_stress_test.sh -c multiHost.yaml -h 3 +| D | 4 |10.0.2.22 | ./run_stress_test.sh -c multiHost.yaml -h 4 +|======================================= + + +== Destructive testing + +There are two ways of recovering from node failure: + +1. Wipe previous map data, and rejoin the cluster +2. Recover from previous map data, and rejoin the cluster + +If the total dataset contains a large proportion of infrequently updated records, then it is preferable to use option 2. + +If the entire key-set is updated frequently, then there is little difference between the two methods. + +Both scenarios can be simulated using the stress test. + +=== Delete existing data and rejoin (Option 1) + +. Start one or more slave processes: + + [source,bash] + .... + $ ./run_stress_test.sh -c src/main/resources/singleMasterStressTest.yaml -h 3 + .... + +. Start one or more master nodes: + + [source,bash] + .... + $ ./run_stress_test.sh -c src/main/resources/singleMasterStressTest.yaml -h 1 + .... + +. When replication has been established, kill the slave process, (for example by using `Ctrl+c`). + +. Restart the slave process (see step 1). +By default, the stress test code deletes existing map data files for the specified hosts +on startup. + + [source,bash] + .... + $ ./run_stress_test.sh -c src/main/resources/singleMasterStressTest.yaml -h 3 + .... + ++ +There should be a noticable difference in throughput after the restart, as +the slave map gets pushed the entire data-set from the master: + + [source,bash] + .... + # usual operation + Node 3 received 1153 events/sec, 153KB/sec from remote 1 + + # slave process killed and restarted + + # higher throughput as slave catches up + Node 3 received 1646 events/sec, 234KB/sec from remote 1 + .... + + +=== Recover existing data and rejoin (Option 2) + +The stress test can be configured, using the `-r` parameter, to recover from existing data on startup. + +Perform the same procedure as in Option 1, step 4, but this time when restarting the slave client use `-r`: + +[source,bash] +.... +$ ./run_stress_test.sh -c src/main/resources/singleMasterStressTest.yaml -h 3 -r +.... + +This will ensure that the existing map is recovered from disk, possibly saving network traffic, depending on how much data has been updated in the master map during downtime. + +<<../Replication.adoc#,Back to Replication>> \ No newline at end of file