Alawn/Chronicle-Map
1
0
Fork 0
mirror of https://github.com/OpenHFT/Chronicle-Map.git synced 2021-09-13 01:26:12 +00:00
Code Releases Activity

Initial re-work of documentation

Browse Source
This commit is contained in:
Neil Clifford
2017-10-04 16:45:58 +01:00
parent 8fe5ca5e7d
commit abff04de0d
1 changed files with 93 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/CM_Replication.adoc
Executable
+93
View File
@@ -0,0 +1,93 @@
= Chronicle Map Replication
Neil Clifford
:toc: macro
:toclevels: 1
:css-signature: demo
:toc-placement: macro
:icons: font
toc::[]
Chronicle Map supports both TCP and UDP replication
image::http://openhft.net/wp-content/uploads/2014/07/Chronicle-Map-TCP-Replication_simple_02.jpg[TCP/IP Replication]
== TCP / UDP Background.
TCP/IP is a reliable protocol. This means that, unless you have a network failure or hardware outage, the data is guaranteed to arrive. TCP/IP provides point-to-point connectivity. For example, if the message was sent to 100 hosts, the message would have to be sent 100 times.
With UDP, the message is only sent once. This is ideal if you have a large number of hosts and you wish to broadcast the same data to each of them. However, one of the big drawbacks with UDP is that it's not a reliable protocol. This means, if the UDP message is broadcast onto the network, the hosts are not guaranteed to receive it; so they can miss data.
Some solutions attempt to build resilience into UDP, but arguably, this is in effect reinventing TCP/IP.
== How to setup UDP Replication
On a good quality wired LAN, when using UDP, you will rarely miss messages. Nevertheless this is a risk that we suggest you don't take. We suggest that whenever you use UDP replication you use it in conjunction with throttled TCP replication. Therefore if a host misses a message over UDP, it will pick it up later via TCP/IP.
== TCP/IP Throttling
We are careful not to swamp your network with too much TCP/IP traffic. We do this by providing a throttled version of TCP replication. This works because Chronicle Map only broadcasts the latest update of each entry.
== How Chronicle Map Replication works
Chronicle Map provides multi-master hash-map replication. This means that each remote map mirrors its changes over to another remote map. Neither map is considered the master store of data. Each map uses timestamps to reconcile changes.
We refer to an instance of a remote map as a **node**. Each node can be connected to up to 128 other nodes.
The data that is stored locally in each node becomes eventually consistent. So changes made to one node, for example by calling `put()`, will be replicated over to the other node.
To achieve a high level of performance and throughput, the call to `put()` will not block,
With `ConcurrentHashMap`, it is typical to check the return code of some methods to obtain the old value; for example, `remove()`.
Due to the loose coupling and lock-free nature of this multi-master implementation, this return value is only the old value on the node's local data store. In other
words, the nodes are only concurrent locally. It is worth understanding that another node, performing exactly the same operation, may return a different value. However, reconciliation will ensure that all the maps
will become eventually consistent.
== Reconciliation
If two or more nodes receive a change to their maps for the same key, but different values, say by a user of the maps, calling the `put(key,value)`, then, initially each node will update its local store, and each local store will hold a different value.
The aim of multi-master replication is
to provide eventual consistency across the nodes. So, with multi-master replication, whenever a node is changed it will notify the other nodes of its change; we refer to this notification as an event.
The event will hold a timestamp indicating the time that the change occurred. It will also hold the state transition; in this case it was a `put` with a key and value.
Eventual consistency is achieved by looking at the timestamp from the remote node. If, for a given key, the remote node's timestamp is newer than the local node's timestamp, then the event from the remote node will be applied to the local node; otherwise, the event will be ignored.
Since none of the nodes is a primary, each node holds information about the other nodes. For this node its own identifier is referred to as its 'localIdentifier'. The identifiers of other nodes are the 'remoteIdentifiers'.
On an update, or insert of a key/value, this node pushes the information about the change to the remote nodes. The nodes use non-blocking Java NIO I/O, and all replication is done on a single thread.
However, there is an edge case. If two nodes update their map at the same time with different values, we have to deterministically resolve which update wins. This is because eventual
consistency mandates that both nodes should end up holding the same data locally.
Although it is rare that two remote
nodes receive an update to their maps at exactly the same time, for the same key, we have to handle this edge case. We cannot therefore rely on timestamps alone to reconcile
the updates. Typically, the update with the newest timestamp should win, but in this example both timestamps are the same, and the decision made to one node should be identical to the decision made to the other. This dilemma is resolved by using a node identifier. The node identifier is a unique
'byte' value that is assigned to each node. When the timestamps are the same, the remote node with the smaller identifier will be preferred.
== Multiple Processes on the same server with Replication
On a single server, if you have a number of Java processes, and then within each Java process you create an instance of a Chronicle Map which binds to the same underline 'file', they exchange data via shared memory, rather than by TCP or UDP replication.
If an instance of Chronicle Map, which is not performing TCP Replication, is updated, then this update can be picked up by another instance of Chronicle Map. This other Chronicle Map instance could be TCP replicated. In such an example, the TCP replicated Chronicle Map instance would then push the update to the remote nodes.
Likewise, if the TCP replicated Chronicle Map instance received an update from a remote node, then this update would be immediately available to all the instances of Chronicle Map on the server.
== Identifier for Replication
If you are only replicating your Chronicle Map instances on the same server, then you do not have to set up TCP and UDP replication. You also do not have to set the identifiers; as the identifiers are only used for the resolution of conflicts amongst remote servers.
If however, you wish to replicate data between two or more servers, then all of the Chronicle Map instances, including those not actively participating in TCP or UDP replication, must have their identifiers set.
The identifier must be unique to each server. Each ChronicleMap on the same server must have
the same identifier. The reason that all Chronicle Map instances must have the identifier set, is because
the memory is laid out slightly differently when using replication, so even if a map is not actively performing TCP or UDP replication itself, if it wishes to replicate with one that is, it must have its memory laid out in the same way to be compatible.
If the identifiers are not set up uniquely, then the updates will be ignored. For example,
a Chronicle Map instance that is set up with the identifier equal to '1', will ignore all events which contain the remote identifier of '1'. In other words, Chronicle Map replication ignores updates which have originated from itself. This is to avoid the circularity of events.
When setting up the identifier you can use values from `1` to `127`.
The identifier is setup on the builder as follows:
```java
TcpTransportAndNetworkConfig tcpConfig = ...
map = ChronicleMapBuilder
.of(Integer.class, CharSequence.class)
.replication(identifier, tcpConfig)
.create();
```