mirror of
https://github.com/OpenHFT/Chronicle-Map.git
synced 2021-09-13 01:26:12 +00:00
284 lines
8.5 KiB
Plaintext
Executable File
284 lines
8.5 KiB
Plaintext
Executable File
= Replication Stress test
|
|
:toc: macro
|
|
:toclevels: 1
|
|
:css-signature: demo
|
|
:toc-placement: macro
|
|
:icons: font
|
|
|
|
toc::[]
|
|
|
|
The following stress test is provided for Chronicle Map Enterprise Edition. The test can be configured to run across multiple machines, with varying read/write workloads.
|
|
|
|
NOTE: This document assumes that you have access to the restricted https://github.com/ChronicleEnterprise/Chronicle-Map-Enterprise-Demo[Chronicle-Map-Enterprise-Demo] repository.
|
|
|
|
== Single master, multiple slaves
|
|
|
|
To execute a test, open the the `Example4` folder, and then execute the following command from a `bash` shell:
|
|
|
|
[source,bash]
|
|
....
|
|
$ ./run_stress_test.sh -c src/main/resources/singleMasterStressTest.yaml
|
|
....
|
|
|
|
This command runs a test, using the specified configuration, which models a single master
|
|
node working with three slave nodes. All nodes run within the same JVM for simplicity.
|
|
|
|
The test reports throughput metrics in terms of reads, writes, and network traffic. For example:
|
|
|
|
[source,bash]
|
|
....
|
|
Write stress-test-1: 237 inserts/sec, 702 updates/sec (total 940/sec)
|
|
Read stress-test-2: 2175 reads/sec, 0.00% were null
|
|
Read stress-test-3: 2177 reads/sec, 0.00% were null
|
|
Read stress-test-4: 2176 reads/sec, 0.00% were null
|
|
Node 3 received 1061 events/sec, 151KB/sec from remote 1
|
|
Node 2 received 1062 events/sec, 151KB/sec from remote 1
|
|
....
|
|
|
|
Every minute, the test pauses the writer, and checks that all maps within the cluster
|
|
contain the same data; that is, they all have the same `keySet`, and the value for each key is
|
|
identical throughout the cluster. For example:
|
|
|
|
[source,bash]
|
|
....
|
|
Starting map checker due to time of 11:37:30.035
|
|
Checking maps for equality
|
|
All maps contained the same data
|
|
....
|
|
|
|
== Running the test in multiple JVMs
|
|
|
|
To spread the workload over multiple JVMs, use the `-h` parameter, which specifies the nodes that should be started in a particular JVM. The `singleMasterStressTest.yaml` configuration file contains 4 nodes (1 master, 3 slaves). The example below spreads these nodes across two processes.
|
|
|
|
. On one terminal, run the following command:
|
|
|
|
[source,bash]
|
|
....
|
|
$ ./run_stress_test.sh -c src/main/resources/singleMasterStressTest.yaml -h 1 -h 2
|
|
....
|
|
|
|
+
|
|
then wait for the message notifying that the test is starting:
|
|
|
|
[source,bash]
|
|
....
|
|
Built stress test artifacts, running test..
|
|
....
|
|
|
|
. On another terminal, run the following command:
|
|
|
|
[source,bash]
|
|
....
|
|
$ ./run_stress_test.sh -c src/main/resources/singleMasterStressTest.yaml -h 3 -h 4
|
|
....
|
|
|
|
+
|
|
|
|
In both terminals, throughput and map-equality checking will be reported.
|
|
|
|
== Multiple masters, multiple slaves
|
|
|
|
To test a multi-master scenario, where several different nodes are written to, execute the following command:
|
|
|
|
[source,bash]
|
|
....
|
|
./run_stress_test.sh -c src/main/resources/multipleMasterStressTest.yaml
|
|
....
|
|
|
|
== Test configuration
|
|
|
|
Stress test scenarios are configured using the supplied `yaml` file. Using the examples
|
|
included in this repository as a template, it is possible to configure custom scenarios
|
|
to model real-world workloads.
|
|
|
|
The configuration is comprised of two components:
|
|
|
|
1. Test parameters
|
|
2. Cluster configuration
|
|
|
|
Test parameters are described below:
|
|
|
|
[source,yaml]
|
|
....
|
|
!TestAndClusterConfig {
|
|
testConfig: !StressTestConfig {
|
|
numberOfSlaveNodes: 3, # map instances that are read from
|
|
numberOfMasterNodes: 1, # map instances that are written to
|
|
updatesPerSecond: 1000, # writes per instance per second
|
|
queriesPerSecond: 2500, # reads per instance per second
|
|
insertToUpdateRatio: 0.25, # ratio of remove/insert vs update operations
|
|
keySpaceSize: 5000, # size of the key space (uniform random integers)
|
|
valueSizeInBytes: 128, # payload size of value object
|
|
testLengthMinutes: 2 # duration of a test
|
|
},
|
|
....
|
|
|
|
== Running tests across multiple physical machines
|
|
|
|
To demonstrate how to run tests across different machines, the following details are defined:
|
|
|
|
Hosts within the cluster:
|
|
|
|
[frame="topbot",options="header,footer"]
|
|
|=======================================
|
|
| host | host_id | IP | role
|
|
| A | 1 |10.0.2.1 | master
|
|
| B | 2 |10.0.2.14 | master
|
|
| C | 3 |10.0.2.30 | slave
|
|
| D | 4 |10.0.2.22 | slave
|
|
|=======================================
|
|
|
|
NOTE: Substitute your own settings as appropriate.
|
|
|
|
**TCP port used for replication**; this port will need to be open on all hosts:
|
|
|
|
`13000`
|
|
|
|
**Configuration file name**:
|
|
|
|
`multiHost.yaml`
|
|
|
|
The cluster configuration in `multiHost.yaml` would be:
|
|
|
|
[source,yaml]
|
|
....
|
|
clusterConfig: !Clusters {
|
|
myCluster: {
|
|
config: !ReplicatedMapCfg {
|
|
entries: 10000,
|
|
keyClass: !type long,
|
|
exampleKey: 17,
|
|
name: stress-test-map,
|
|
keysAreConstantSize: true,
|
|
valuesAreConstantSize: false,
|
|
keyDataAccess: !LongDataAccess {
|
|
},
|
|
keyReader: !LongMarshaller {
|
|
},
|
|
},
|
|
topology: {
|
|
context: !MapClusterContext {
|
|
wireType: TEXT,
|
|
},
|
|
hostA: {
|
|
hostId: 1,
|
|
connectUri: "10.0.2.1:13000"
|
|
},
|
|
hostB: {
|
|
hostId: 2,
|
|
connectUri: "10.0.2.14:13000"
|
|
},
|
|
hostC: {
|
|
hostId: 3,
|
|
connectUri: "10.0.2.30:13000"
|
|
},
|
|
hostD: {
|
|
hostId: 4,
|
|
connectUri: "10.0.2.22:13000"
|
|
}
|
|
},
|
|
monitoring: {
|
|
replicationEventListener: !LoggingReplicationEventListener
|
|
}
|
|
}
|
|
....
|
|
|
|
=== Packaging the test
|
|
|
|
To run the test on remote machines, the stress test can be packaged up using the following command:
|
|
|
|
[source,bash]
|
|
....
|
|
$ ./package_stress_test.sh
|
|
....
|
|
|
|
This command generates a `tar` archive in the current directory, containing everything required to run a stress test.
|
|
|
|
Copy the resulting `tar` file to the remote server, unpack using `tar xf`, and then run the following command:
|
|
|
|
[source,bash]
|
|
....
|
|
./run_stress_test.sh -c [config-file]
|
|
....
|
|
|
|
To run the example multi-machine configuration described above, the following commands would be run on each host:
|
|
|
|
[frame="topbot",options="header,footer"]
|
|
|=======================================
|
|
| host | host_id | IP | command
|
|
| **A** | `1` |`10.0.2.1` | `./run_stress_test.sh -c multiHost.yaml -h 1`
|
|
| **B** | `2` |`10.0.2.14` | `./run_stress_test.sh -c multiHost.yaml -h 2`
|
|
| **C** | `3` |`10.0.2.30` | `./run_stress_test.sh -c multiHost.yaml -h 3`
|
|
| **D** | `4` |`10.0.2.22` | `./run_stress_test.sh -c multiHost.yaml -h 4`
|
|
|=======================================
|
|
|
|
== Destructive testing
|
|
|
|
There are two ways of recovering from node failure:
|
|
|
|
1. Delete previous map data, and rejoin the cluster.
|
|
|
|
2. Recover previous map data, and rejoin the cluster.
|
|
|
|
If the total dataset contains a large proportion of infrequently updated records, then it is preferable to use option 2.
|
|
|
|
If the entire key-set is updated frequently, then there is little difference between the two methods.
|
|
|
|
Both scenarios can be simulated using the stress test.
|
|
|
|
=== Delete existing data and rejoin (Option 1)
|
|
|
|
. Start one or more slave processes:
|
|
|
|
[source,bash]
|
|
....
|
|
$ ./run_stress_test.sh -c src/main/resources/singleMasterStressTest.yaml -h 3
|
|
....
|
|
|
|
. Start one or more master nodes:
|
|
|
|
[source,bash]
|
|
....
|
|
$ ./run_stress_test.sh -c src/main/resources/singleMasterStressTest.yaml -h 1
|
|
....
|
|
|
|
. When replication has been established, kill the slave process; for example, by using `Ctrl+c`.
|
|
|
|
. Restart the slave process (see step 1).
|
|
By default, the stress test code deletes existing map data files for the specified hosts
|
|
on startup.
|
|
|
|
[source,bash]
|
|
....
|
|
$ ./run_stress_test.sh -c src/main/resources/singleMasterStressTest.yaml -h 3
|
|
....
|
|
|
|
+
|
|
There should be a noticable difference in throughput after the restart, as
|
|
the slave map receives the entire data-set from the master:
|
|
|
|
[source,bash]
|
|
....
|
|
# usual operation
|
|
Node 3 received 1153 events/sec, 153KB/sec from remote 1
|
|
|
|
# slave process killed and restarted
|
|
|
|
# higher throughput as slave catches up
|
|
Node 3 received 1646 events/sec, 234KB/sec from remote 1
|
|
....
|
|
|
|
=== Recover existing data and rejoin (Option 2)
|
|
|
|
The stress test can be configured, using the `-r` parameter, to recover from existing data on startup.
|
|
|
|
Perform the same procedure as in Option 1, step 4, but this time when restarting the slave client use `-r`:
|
|
|
|
[source,bash]
|
|
....
|
|
$ ./run_stress_test.sh -c src/main/resources/singleMasterStressTest.yaml -h 3 -r
|
|
....
|
|
|
|
This ensures that the existing map is recovered from disk. This may save network traffic, depending on how much data has been updated in the master map during downtime.
|
|
|
|
<<CM_Replication.adoc#,Back to Replication>> |