mirror of
https://github.com/OpenHFT/Chronicle-Map.git
synced 2021-09-13 01:26:12 +00:00
Merge branch 'doco_refresh' of C:\Users\Neil\IdeaProjects\Chronicle-Map with conflicts.
This commit is contained in:
Neil Clifford
+23
-7
@@ -6,6 +6,13 @@ Chronicle Map can overflow seamlessly to disk, and can be distributed across the
|
||||
|
||||
image::docs\images\CM_Overview.jpg[]
|
||||
|
||||
== Prerequisites
|
||||
|
||||
To run Chronicle Map, please ensure you have first installed:
|
||||
|
||||
* Java (1.8.0_91 or later)
|
||||
* Apache Maven (3.2.3 or later)
|
||||
|
||||
=== Use cases
|
||||
Chronicle Map is used in production around the
|
||||
world for:
|
||||
@@ -58,13 +65,22 @@ The Chronicle Map documentation comprises:
|
||||
.Documentation
|
||||
|===
|
||||
|**Document**|**Purpose**
|
||||
|<<docs/CM_Features.adoc#,CM_Features.adoc>>|Features description.
|
||||
|<<docs/CM_Replication.adoc#,CM_Replication.adoc>>|Replication explanation.
|
||||
|<<docs/CM_Tutorial.adoc#,CM_Tutorial.adoc>>|Tutorial.
|
||||
|<<docs/CM_FAQs.adoc#,CM_FAQs.adoc>>|Frequently asked questions.
|
||||
|<<docs/CM_Download.adoc#,CM_Download.adoc>>|Downloading the software.
|
||||
|<<docs/CM_Updates.adoc#,CM_Updates.adoc>>|Updates from Chronicle Map version 2.
|
||||
|<<docs/CM_Compatibility_and_Versioning.adoc#,CM_Compatibility_and_Versioning>>|Compatibility and Versioning description.
|
||||
|<<docs/CM_Features.adoc#,CM_Features.adoc>>
|
||||
|Features description.
|
||||
|<<docs/CM_Tutorial.adoc#,CM_Tutorial.adoc>>
|
||||
|Tutorial.
|
||||
|<<docs/CM_FAQs.adoc#,CM_FAQs.adoc>>
|
||||
|Frequently asked questions.
|
||||
|<<docs/CM_Download.adoc#,CM_Download.adoc>>
|
||||
|Downloading the software.
|
||||
|<<docs/CM_Updates.adoc#,CM_Updates.adoc>>
|
||||
|Updates from Chronicle Map version 2.
|
||||
|<<docs/CM_Replication.adoc#,CM_Replication.adoc>>
|
||||
|Replication explanation.
|
||||
|<<docs/CM_Replication_Stress_Test.adoc#,CM_Replication_Stress_Test.adoc>>
|
||||
|Example Replication stress test.
|
||||
|<<docs/CM_Compatibility_and_Versioning.adoc#,CM_Compatibility_and_Versioning>>
|
||||
|Compatibility and Versioning description.
|
||||
|===
|
||||
|
||||
Linked documentation is contained in the `docs` folder.
|
||||
+137
-3
@@ -1,4 +1,4 @@
|
||||
= Chronicle Map Replication
|
||||
= Chronicle Map Replication (Enterprise Edition feature)
|
||||
Neil Clifford
|
||||
:toc: macro
|
||||
:toclevels: 1
|
||||
@@ -8,8 +8,10 @@ Neil Clifford
|
||||
|
||||
toc::[]
|
||||
|
||||
Chronicle Map supports both TCP and UDP replication in the closed-source Chronicle Map Enterprise.
|
||||
For more information on this please contact mailto:sales@chronicle.software.
|
||||
|
||||
Chronicle Map Replication is a multi-master replicated data store.
|
||||
|
||||
Chronicle Map supports both TCP and UDP replication
|
||||
|
||||
image::http://openhft.net/wp-content/uploads/2014/07/Chronicle-Map-TCP-Replication_simple_02.jpg[TCP/IP Replication]
|
||||
|
||||
@@ -92,3 +94,135 @@ map = ChronicleMapBuilder
|
||||
.replication(identifier, tcpConfig)
|
||||
.create();
|
||||
```
|
||||
|
||||
== Configuration
|
||||
|
||||
Configuration of map nodes is done either programmatically, through configuration files, or a combination of both.
|
||||
|
||||
The example below shows how to set up two map instances that use TCP to replicate entries between them:
|
||||
|
||||
[source, java]
|
||||
....
|
||||
|
||||
// set up some socket aliases
|
||||
TCPRegistry.createServerSocketChannelFor("host.port1", "host.port2");
|
||||
|
||||
// create replicated maps
|
||||
final ReplicatedMap<Long, Long> replicaOne = createMap(1, "host.port1");
|
||||
final ReplicatedMap<Long, Long> replicaTwo = createMap(2, "host.port2");
|
||||
|
||||
// start replication event loops
|
||||
replicaOne.start();
|
||||
replicaTwo.start();
|
||||
|
||||
// register maps for replication events
|
||||
replicaOne.beginReplicationFromPeer("host.port2", (byte) 2);
|
||||
replicaTwo.beginReplicationFromPeer("host.port1", (byte) 1);
|
||||
|
||||
// allow replication to establish
|
||||
Jvm.pause(250L);
|
||||
|
||||
// access the underlying maps
|
||||
final Map<Long, Long> mapOne = replicaOne.getMap();
|
||||
final Map<Long, Long> mapTwo = replicaTwo.getMap();
|
||||
|
||||
// enter some data into mapOne
|
||||
for (long i = 0; i < 100; i++) {
|
||||
mapOne.put(i, i * 2);
|
||||
}
|
||||
|
||||
// enter some different data into mapTwo
|
||||
for (long i = 100; i < 200; i++) {
|
||||
mapTwo.put(i, i * 7);
|
||||
}
|
||||
|
||||
// wait for replication between nodes (conservative)
|
||||
Jvm.pause(1000L);
|
||||
|
||||
System.out.printf("mapOne has %d entries, expected 200%n", mapOne.size());
|
||||
System.out.printf("mapTwo has %d entries, expected 200%n", mapTwo.size());
|
||||
....
|
||||
|
||||
The following example uses a basic `yaml` configuration file to define defaults for the map:
|
||||
|
||||
[source, yaml]
|
||||
....
|
||||
!ReplicatedMapCfg {
|
||||
|
||||
hostId: 1, # host id must be unique within the cluster
|
||||
listenAddressHostPort: "localhost:9301", # address on which to listen for inbound replication requests
|
||||
entries: 10000, # entries hint suggesting the initial size of the map
|
||||
keyClass: !type long, # class of the key type
|
||||
valueClass: !type long, # class of the value type
|
||||
exampleKey: 17, # an example key (used to size the map)
|
||||
exampleValue: 37, # an example value (used to size the map)
|
||||
name: testLongToLongMap, # a name for the map (used to determine filename)
|
||||
locator: "csp://localhost:9301/map/test", # a locator, used for discovery
|
||||
mapFileDataDirectory: target/mapData, # directory where map file should be stored
|
||||
mapLogDirectory: target/mapReplicationLogs, # directory where replication events should be recorded
|
||||
keysAreConstantSize: true, # determines how keys are serialised
|
||||
valuesAreConstantSize: true # determines how values are serialised
|
||||
|
||||
}
|
||||
....
|
||||
|
||||
After putting 100 entries into each distinct map, the example demonstrates that the entries have been replicated so that each map
|
||||
now contains 200 entries.
|
||||
|
||||
This example is available in the repository, link::https://github.com/ChronicleEnterprise/Chronicle-Map-Enterprise/blob/master/src/test/java/software/chronicle/enterprise/map/example/MultiMasterMapReplicationMain.java[MultiMasterMapReplicationMain.java]
|
||||
|
||||
|
||||
== Replication event logging
|
||||
|
||||
Chronicle Map Enterprise can be configured to log all replication events to a Chronicle Queue for auditing purposes.
|
||||
|
||||
Currently, a map can be configured to log all *outgoing* events that it sends to remote peers.
|
||||
|
||||
The example below shows the message flow for a map with a single remote peer receiving replication events:
|
||||
|
||||
[source, java]
|
||||
....
|
||||
.. header omitted
|
||||
|
||||
targetHostId: 2 # message destination
|
||||
registerForReplication: {
|
||||
receiverHostId: 1, # local peer id
|
||||
keyClassName: java.lang.Long, # key type
|
||||
valueClassName: java.lang.Long, # value type
|
||||
sendAllEventsAfter: 0 # bootstrap timestamp
|
||||
}
|
||||
|
||||
..
|
||||
|
||||
# corresponding registration message from remote peer
|
||||
targetHostId: 1
|
||||
registerForReplication: {
|
||||
receiverHostId: 2,
|
||||
keyClassName: java.lang.Long,
|
||||
valueClassName: java.lang.Long,
|
||||
sendAllEventsAfter: 0
|
||||
}
|
||||
|
||||
..
|
||||
|
||||
# accept replication connection
|
||||
targetHostId: 2
|
||||
acceptingReplicationRequest: {
|
||||
acceptingHostId: 1
|
||||
}
|
||||
|
||||
..
|
||||
|
||||
# encoded replication update sent to remote peer
|
||||
targetHostId: 2
|
||||
replicatedEntry: !!binary AYChq5LqwKXqFAFOEwAAAAAAAAD/////////fw==
|
||||
....
|
||||
|
||||
This output can be generated by running the example program,
|
||||
link::https://github.com/ChronicleEnterprise/Chronicle-Map-Enterprise/blob/master/src/test/java/software/chronicle/enterprise/map/example/ReplicationEventLoggingExampleMain.java[ReplicationEventLoggingExampleMain.java]
|
||||
|
||||
'''
|
||||
For an example replication stress test, see <<../CM_Replication_Stress_Test.adoc#,CM_Replication_Stress_Test.adoc>>.
|
||||
|
||||
'''
|
||||
<<../Features.adoc#,Back to Features>>
|
||||
|
||||
Executable
+286
@@ -0,0 +1,286 @@
|
||||
= Replication Stress test
|
||||
:toc: macro
|
||||
:toclevels: 1
|
||||
:css-signature: demo
|
||||
:toc-placement: macro
|
||||
:icons: font
|
||||
|
||||
toc::[]
|
||||
|
||||
The following stress test is provided for Chronicle Map Enterprise Edition. The test can be configured to run across multiple machines, with varying read/write workloads.
|
||||
|
||||
NOTE: This document assumes that you have access to the restricted link::https://github.com/ChronicleEnterprise/Chronicle-Map-Enterprise-Demo[Chronicle-Map-Enterprise-Demo] repository.
|
||||
|
||||
== Single master, multiple slaves
|
||||
|
||||
To execute a test, open the the `Example4` directory, then execute the following command from a `bash` shell:
|
||||
|
||||
[source,bash]
|
||||
....
|
||||
$ ./run_stress_test.sh -c src/main/resources/singleMasterStressTest.yaml
|
||||
....
|
||||
|
||||
This command will run a test using the specified configuration, which models a single master
|
||||
node along with three slave nodes. All nodes run within the same JVM for simplicity.
|
||||
|
||||
The test will report throughput metrics in terms of reads, writes, and network traffic. For example:
|
||||
|
||||
[source,bash]
|
||||
....
|
||||
Write stress-test-1: 237 inserts/sec, 702 updates/sec (total 940/sec)
|
||||
Read stress-test-2: 2175 reads/sec, 0.00% were null
|
||||
Read stress-test-3: 2177 reads/sec, 0.00% were null
|
||||
Read stress-test-4: 2176 reads/sec, 0.00% were null
|
||||
Node 3 received 1061 events/sec, 151KB/sec from remote 1
|
||||
Node 2 received 1062 events/sec, 151KB/sec from remote 1
|
||||
....
|
||||
|
||||
Every minute, the test will pause the writer and check that all maps within the cluster
|
||||
contain the same data; that is, they all have the same `keySet`, and the value for each key is
|
||||
identical throughout the cluster. For example:
|
||||
|
||||
[source,bash]
|
||||
....
|
||||
Starting map checker due to time of 11:37:30.035
|
||||
Checking maps for equality
|
||||
All maps contained the same data
|
||||
....
|
||||
|
||||
|
||||
== Running the test in multiple JVMs
|
||||
|
||||
To spread the workload over multiple JVMs, use the `-h` parameter, which specifies the nodes that should be started in a particular JVM. The `singleMasterStressTest.yaml` configuration file contains 4 nodes (1 master, 3 slaves). The example below spreads these nodes across two processes.
|
||||
|
||||
In one terminal, run the following command:
|
||||
|
||||
[source,bash]
|
||||
....
|
||||
$ ./run_stress_test.sh -c src/main/resources/singleMasterStressTest.yaml -h 1 -h 2
|
||||
....
|
||||
|
||||
then wait for the message notifying that the test is starting:
|
||||
|
||||
[source,bash]
|
||||
....
|
||||
Built stress test artifacts, running test..
|
||||
....
|
||||
|
||||
Now in another terminal, run the following command:
|
||||
|
||||
[source,bash]
|
||||
....
|
||||
$ ./run_stress_test.sh -c src/main/resources/singleMasterStressTest.yaml -h 3 -h 4
|
||||
....
|
||||
|
||||
In both terminals, throughput and map-equality checking will be reported.
|
||||
|
||||
== Multiple masters, multiple slaves
|
||||
|
||||
To test a multi-master scenario, where several different nodes are written to, execute the following command:
|
||||
|
||||
[source,bash]
|
||||
....
|
||||
./run_stress_test.sh -c src/main/resources/multipleMasterStressTest.yaml
|
||||
....
|
||||
|
||||
== Test configuration
|
||||
|
||||
Stress test scenarios are configured using the supplied `yaml` file. Using the examples
|
||||
included in this repository as a template, it is possible to configure custom scenarios
|
||||
to model real-world workloads.
|
||||
|
||||
The configuration is comprised of two components:
|
||||
|
||||
1. Test parameters
|
||||
2. Cluster configuration
|
||||
|
||||
Test parameters are described below:
|
||||
|
||||
[source,yaml]
|
||||
....
|
||||
!TestAndClusterConfig {
|
||||
testConfig: !StressTestConfig {
|
||||
numberOfSlaveNodes: 3, # map instances that are read from
|
||||
numberOfMasterNodes: 1, # map instances that are written to
|
||||
updatesPerSecond: 1000, # writes per instance per second
|
||||
queriesPerSecond: 2500, # reads per instance per second
|
||||
insertToUpdateRatio: 0.25, # ratio of remove/insert vs update operations
|
||||
keySpaceSize: 5000, # size of the key space (uniform random integers)
|
||||
valueSizeInBytes: 128, # payload size of value object
|
||||
testLengthMinutes: 2 # duration of a test
|
||||
},
|
||||
....
|
||||
|
||||
== Running tests across multiple physical machines
|
||||
|
||||
To demonstrate how to run tests across different machines, the following details are defined:
|
||||
Hosts within the cluster:
|
||||
|
||||
[frame="topbot",options="header,footer"]
|
||||
|=======================================
|
||||
| host | host_id | IP | role
|
||||
| A | 1 |10.0.2.1 | master
|
||||
| B | 2 |10.0.2.14 | master
|
||||
| C | 3 |10.0.2.30 | slave
|
||||
| D | 4 |10.0.2.22 | slave
|
||||
|=======================================
|
||||
|
||||
NOTE: Substitute your own settings as appropriate.
|
||||
|
||||
|
||||
TCP port used for replication; this port will need to be open on all hosts:
|
||||
|
||||
`13000`
|
||||
|
||||
Configuration file name:
|
||||
|
||||
`multiHost.yaml`
|
||||
|
||||
|
||||
The cluster configuration in `multiHost.yaml` would be:
|
||||
|
||||
|
||||
[source,yaml]
|
||||
....
|
||||
clusterConfig: !Clusters {
|
||||
myCluster: {
|
||||
config: !ReplicatedMapCfg {
|
||||
entries: 10000,
|
||||
keyClass: !type long,
|
||||
exampleKey: 17,
|
||||
name: stress-test-map,
|
||||
keysAreConstantSize: true,
|
||||
valuesAreConstantSize: false,
|
||||
keyDataAccess: !LongDataAccess {
|
||||
},
|
||||
keyReader: !LongMarshaller {
|
||||
},
|
||||
},
|
||||
topology: {
|
||||
context: !MapClusterContext {
|
||||
wireType: TEXT,
|
||||
},
|
||||
hostA: {
|
||||
hostId: 1,
|
||||
connectUri: "10.0.2.1:13000"
|
||||
},
|
||||
hostB: {
|
||||
hostId: 2,
|
||||
connectUri: "10.0.2.14:13000"
|
||||
},
|
||||
hostC: {
|
||||
hostId: 3,
|
||||
connectUri: "10.0.2.30:13000"
|
||||
},
|
||||
hostD: {
|
||||
hostId: 4,
|
||||
connectUri: "10.0.2.22:13000"
|
||||
}
|
||||
},
|
||||
monitoring: {
|
||||
replicationEventListener: !LoggingReplicationEventListener
|
||||
}
|
||||
}
|
||||
....
|
||||
|
||||
|
||||
=== Packaging the test
|
||||
|
||||
To run the test on remote machines, the stress test can be packaged up using the following command:
|
||||
|
||||
[source,bash]
|
||||
....
|
||||
$ ./package_stress_test.sh
|
||||
....
|
||||
|
||||
This command will generate a `tar` archive in the current directory, containing everything required to run a stress test.
|
||||
|
||||
Copy the resulting `tar` file to the remote server, unpack using `tar xf`, and then run the following command:
|
||||
|
||||
[source,bash]
|
||||
....
|
||||
./run_stress_test.sh -c [config-file]
|
||||
....
|
||||
|
||||
To run the example multi-machine configuration described above, the following commands would be run on each host:
|
||||
|
||||
[frame="topbot",options="header,footer"]
|
||||
|=======================================
|
||||
| host | host_id | IP | command
|
||||
| A | 1 |10.0.2.1 | ./run_stress_test.sh -c multiHost.yaml -h 1
|
||||
| B | 2 |10.0.2.14 | ./run_stress_test.sh -c multiHost.yaml -h 2
|
||||
| C | 3 |10.0.2.30 | ./run_stress_test.sh -c multiHost.yaml -h 3
|
||||
| D | 4 |10.0.2.22 | ./run_stress_test.sh -c multiHost.yaml -h 4
|
||||
|=======================================
|
||||
|
||||
|
||||
== Destructive testing
|
||||
|
||||
There are two ways of recovering from node failure:
|
||||
|
||||
1. Wipe previous map data, and rejoin the cluster
|
||||
2. Recover from previous map data, and rejoin the cluster
|
||||
|
||||
If the total dataset contains a large proportion of infrequently updated records, then it is preferable to use option 2.
|
||||
|
||||
If the entire key-set is updated frequently, then there is little difference between the two methods.
|
||||
|
||||
Both scenarios can be simulated using the stress test.
|
||||
|
||||
=== Delete existing data and rejoin (Option 1)
|
||||
|
||||
. Start one or more slave processes:
|
||||
|
||||
[source,bash]
|
||||
....
|
||||
$ ./run_stress_test.sh -c src/main/resources/singleMasterStressTest.yaml -h 3
|
||||
....
|
||||
|
||||
. Start one or more master nodes:
|
||||
|
||||
[source,bash]
|
||||
....
|
||||
$ ./run_stress_test.sh -c src/main/resources/singleMasterStressTest.yaml -h 1
|
||||
....
|
||||
|
||||
. When replication has been established, kill the slave process, (for example by using `Ctrl+c`).
|
||||
|
||||
. Restart the slave process (see step 1).
|
||||
By default, the stress test code deletes existing map data files for the specified hosts
|
||||
on startup.
|
||||
|
||||
[source,bash]
|
||||
....
|
||||
$ ./run_stress_test.sh -c src/main/resources/singleMasterStressTest.yaml -h 3
|
||||
....
|
||||
|
||||
+
|
||||
There should be a noticable difference in throughput after the restart, as
|
||||
the slave map gets pushed the entire data-set from the master:
|
||||
|
||||
[source,bash]
|
||||
....
|
||||
# usual operation
|
||||
Node 3 received 1153 events/sec, 153KB/sec from remote 1
|
||||
|
||||
# slave process killed and restarted
|
||||
|
||||
# higher throughput as slave catches up
|
||||
Node 3 received 1646 events/sec, 234KB/sec from remote 1
|
||||
....
|
||||
|
||||
|
||||
=== Recover existing data and rejoin (Option 2)
|
||||
|
||||
The stress test can be configured, using the `-r` parameter, to recover from existing data on startup.
|
||||
|
||||
Perform the same procedure as in Option 1, step 4, but this time when restarting the slave client use `-r`:
|
||||
|
||||
[source,bash]
|
||||
....
|
||||
$ ./run_stress_test.sh -c src/main/resources/singleMasterStressTest.yaml -h 3 -r
|
||||
....
|
||||
|
||||
This will ensure that the existing map is recovered from disk, possibly saving network traffic, depending on how much data has been updated in the master map during downtime.
|
||||
|
||||
<<../Replication.adoc#,Back to Replication>>
|
||||
Reference in New Issue
Block a user