mirror of
https://github.com/OpenHFT/Chronicle-Map.git
synced 2021-09-13 01:26:12 +00:00
Replace term Chronicle Map instance with Chronicle Map [data|key-value] store in specification, to reduce confusion
This commit is contained in:
@@ -28,8 +28,8 @@ applications. Notably trading, financial market applications.
|
||||
https://en.wikipedia.org/wiki/Conflict-free_replicated_data_type) strategy.
|
||||
|
||||
**Unique features**
|
||||
- Multiple processes could access a Chronicle Map instance concurrently. At the same time,
|
||||
the instance is *in-process for each of the accessing processes*. (Out-of-process approach to IPC
|
||||
- Multiple processes could access a Chronicle Map concurrently. At the same time,
|
||||
the data store is *in-process for each of the accessing processes*. (Out-of-process approach to IPC
|
||||
is simply incompatible with Chronicle Map's median latency target of < 1 μs.)
|
||||
|
||||
- Replication *without logs*, with constant footprint cost, guarantees progress even if the network
|
||||
@@ -70,7 +70,7 @@ queries*.
|
||||
e. g. [Chronicle Enterprise](http://chronicle.software/products/chronicle-enterprise/).
|
||||
|
||||
**What is the Chronicle Map's data structure?** In one sentence and simplified, a Chronicle Map
|
||||
instance is a big chunk of shared memory (optionally mapped to disk), split into independent
|
||||
data store is a big chunk of shared memory (optionally mapped to disk), split into independent
|
||||
segments, each segment has an independent memory allocation for storing the entries, a hash table
|
||||
for search, and a lock in shared memory (implemented via CAS loops) for managing concurrent access.
|
||||
Read [the Chronicle Map data store design overview](spec/2-design-overview.md) for more.
|
||||
@@ -159,7 +159,7 @@ Chronicle Map doesn't support
|
||||
Functional changes in Chronicle Map 3:
|
||||
|
||||
- Chronicle Map 3 has [formal data store specification](spec), that verbalizes [the guarantees
|
||||
which the data store provides](spec/1-design-goals.md#guarantees-2) and gives a way to verify those
|
||||
which the data store provides](spec/1-design-goals.md#guarantees-1) and gives a way to verify those
|
||||
guarantees.
|
||||
- Added support for multi-key queries.
|
||||
- "Listeners" mechanism fully reworked, see the [Behaviour Customization](#behaviour-customization)
|
||||
@@ -169,11 +169,11 @@ Functional changes in Chronicle Map 3:
|
||||
- "Stateless clients" functionality (i. e. remote calls) is moved to [Chronicle Engine](
|
||||
https://github.com/OpenHFT/Chronicle-Engine).
|
||||
- Replication is done via [Chronicle Engine](https://github.com/OpenHFT/Chronicle-Engine).
|
||||
- Chronicle Map 2 has hard creation-time limit on the number of entries storable in the Chronicle
|
||||
Map instance. If the size exceeds this limit, an exception is thrown. In Chronicle Map 3, this
|
||||
limitation is removed, though the number of entries still has to be configured on the Chronicle Map
|
||||
instance creation, exceeding this configured limit is possible, but discouraged. See the
|
||||
[Number of entries configuration](#number-of-entries-configuration) section.
|
||||
- Chronicle Map 2 has hard creation-time limit on the number of entries storable in a Chronicle
|
||||
Map. If the size exceeds this limit, an exception is thrown. In Chronicle Map 3, this limitation
|
||||
is removed, though the number of entries still has to be configured on the Chronicle Map creation,
|
||||
exceeding this configured limit is possible, but discouraged. See the [Number of entries
|
||||
configuration](#number-of-entries-configuration) section.
|
||||
- Chronicle Map 3 supports [entry checksums](#entry-checksums) that allows to detect data
|
||||
corruption, hence brings additional safety.
|
||||
- Chronicle Map 3 allows to [recover](#recovery) after failures and corruptions.
|
||||
|
||||
@@ -23,12 +23,12 @@ the following rules, they might be changed in any way, or removed.
|
||||
Semantics and actual effects of API calls might change between Chronicle Map versions with
|
||||
[different "minor" number](versioning.md), e. g. default serialization forms of key or value objects
|
||||
of some types could change, "high level" configurations in `ChronicleMapBuilder` could produce
|
||||
Chronicle Map instances with different properties, etc. Such changes should be documented in release
|
||||
notes.
|
||||
Chronicle Maps with different "low level" configurations, etc. Such changes should be documented in
|
||||
release notes.
|
||||
|
||||
### Persistence File Binary Form
|
||||
|
||||
A persisted Chronicle Map instance (a binary file), produced by a production release of the library,
|
||||
A persisted Chronicle Map store (a binary file), produced by a production release of the library,
|
||||
should be accessible and updatable by any subsequent production release of the library in 3.x
|
||||
branch, unless the older version has bugs that lead to data corruption.
|
||||
|
||||
@@ -51,7 +51,7 @@ Though this might lead to failures, but not to data corruption on either of the
|
||||
|
||||
The Chronicle Map Data Store Specification could be updated without changing the revision while it
|
||||
remains backward compatible, i. e. the implementation of the newer version of the specification
|
||||
could access and update any persisted Chronicle Map instance, produced by an older version.
|
||||
could access and update any persisted Chronicle Map store, produced by an older version.
|
||||
In addition, potential *concurrent* accessing implementations of older versions ("within" the same
|
||||
revision) shouldn't lose correctness and corrupt the data.
|
||||
|
||||
|
||||
@@ -11,10 +11,10 @@
|
||||
- Chronicle Map supports fully-featured concurrent access from multiple processes, mapping the same
|
||||
file.
|
||||
- Chronicle Map supports *isolated* multi-key operations, either on a single key-value domain
|
||||
(a single Chronicle Map instance) or several ones. But multi-key updates to persisted Chronicle
|
||||
Map(s) are *not* atomic (basically, *isolated* and *atomic* terms used in the sense of
|
||||
[ACID](https://en.wikipedia.org/wiki/ACID) here; see the detailed definition below
|
||||
in [Guarantees](#guarantees-1) section).
|
||||
(a single Chronicle Map store) or several ones. But multi-key updates to persisted Chronicle Map(s)
|
||||
are *not* atomic (basically, *isolated* and *atomic* terms used in the sense of [ACID](
|
||||
https://en.wikipedia.org/wiki/ACID) here; see the detailed definition below in
|
||||
[Guarantees](#guarantees-1) section).
|
||||
- Chronicle Map runs only on little-endian architectures.
|
||||
- Chronicle Map is *not* distributed: the whole data store resides the memory of a single machine
|
||||
(a single file, if persisted).
|
||||
@@ -89,25 +89,25 @@ https://en.wikipedia.org/wiki/File_locking).
|
||||
If the above assumptions are met, Chronicle Map aims to satisfy the following guarantees:
|
||||
|
||||
- Single-key and multi-key accesses and updates to the Chronicle Map (multi-key updates could span
|
||||
several Chronicle Map instances) are concurrently isolated, i. e. accesses involving a certain key
|
||||
several Chronicle Map stores) are concurrently isolated, i. e. accesses involving a certain key
|
||||
are totally ordered across accessing threads and processes. All updates made to the entry during
|
||||
the previous update are visible during the subsequent accesses. After a multi-key update,
|
||||
a subsequent multi-key query involving a subset of the updated keys (and possibly some more keys)
|
||||
observes all updates made to all the entries corresponding to the keys from that subset.
|
||||
- Under any conditions (concurrent access, inter-process access) and using any type of access
|
||||
(querying the entry by the key or iteration), only entries that were ever stored in the Chronicle
|
||||
Map instance are observed. Reading corrupted or half written values by some keys, or observing some
|
||||
Map store are observed. Reading corrupted or half written values by some keys, or observing some
|
||||
keys that were never stored during the iteration is impossible.
|
||||
|
||||
If the Chronicle Map instance is persisted to the file and the operating system fails to flush all
|
||||
If the Chronicle Map store is persisted to the file and the operating system fails to flush all
|
||||
the dirty memory to the disk due to a power-off or any other failure, when the file is mapped to
|
||||
the memory and accessed again, it is required to perform a *special recovery procedure* on the
|
||||
Chronicle Map first, which identifies and purges corrupted entries from the Chronicle Map.
|
||||
Therefore, *some entries updated shortly before the failure could be lost.*
|
||||
|
||||
> The aforementioned *special recovery procedure* is `ChronicleHashBuilder.recoverPersistedTo()`
|
||||
> or `ChronicleHashInstanceBuilder.recover()` in the reference Java implementation. See
|
||||
> [Recovery](../README.md#recovery) section in the Chronicle Map tutorial for more information.
|
||||
> in the reference Java implementation. See [Recovery](../README.md#recovery) section in the
|
||||
> Chronicle Map tutorial for more information.
|
||||
|
||||
## Goals
|
||||
|
||||
|
||||
@@ -19,7 +19,7 @@ performing a query to a Chronicle Map:
|
||||
For multi-key queries,
|
||||
|
||||
1. Identify all the segments in which the involved keys should be held.
|
||||
2. Acquire the locks of all the involved segments. Within each involved Chronicle Map instance,
|
||||
2. Acquire the locks of all the involved segments. Within each involved Chronicle Map store,
|
||||
acquire the segment locks in the order of their segments.
|
||||
|
||||
> Acquiring segment locks always in the same order is needed to avoid dead-locks, e. g. when the
|
||||
@@ -76,7 +76,7 @@ an operation system, Chronicle Map doesn't implement queues of threads, waiting
|
||||
and conditional wake-ups. Threads acquire locks in a spin loop, yielding and/or sleeping after a
|
||||
certain threshold numbers of unsuccessful attempts.
|
||||
|
||||
> Chronicle Map implementations specialized for single-process access to instances may implement
|
||||
> Chronicle Map implementations specialized for single-process access to stores may implement
|
||||
> some kind of thread queues, wake-ups and cooperative scheduling on the runtime level; certain lock
|
||||
> acquisition strategy is out of scope of the Chronicle Map data store specification.
|
||||
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
# 3. Chronicle Map Data Store Memory Layout
|
||||
|
||||
A newly created Chronicle Map instance (or just mapped from an already existing persistence file)
|
||||
takes one big continuous block of memory. Its structure, from lower addresses to higher:
|
||||
A newly created Chronicle Map store (or just mapped from an already existing persistence file) takes
|
||||
one big continuous block of memory. Its structure, from lower addresses to higher:
|
||||
|
||||
1. [The self-bootstrapping header](#self-bootstrapping-header).
|
||||
2. The alignment to the next cache line boundary (to the next multiple of 64) by addresses.
|
||||
@@ -24,7 +24,7 @@ takes one big continuous block of memory. Its structure, from lower addresses to
|
||||
> it spans almost always reside in TLB cache and always need to be flushed to the disk.
|
||||
|
||||
> The alignment (more precisely, the offset to the next area, the segment headers area) is stored
|
||||
> in the global mutable state, rather then computed each time a mapped Chronicle Map instance is
|
||||
> in the global mutable state, rather then computed each time a mapped Chronicle Map store is
|
||||
> accessed, because other accessing file systems, operation systems and Chronicle Map
|
||||
> implementations may use a different page size for memory mapping.
|
||||
|
||||
@@ -49,7 +49,7 @@ of this size-prefixed blob, computed by [xxHash](https://github.com/Cyan4973/xxH
|
||||
|
||||
The self-bootstrapping header itself (i. e. the "message" of the size-prefixed blob) is encoded in
|
||||
Text Wire format. Once created, this header is never changed. It contains all configurations,
|
||||
immutable for Chronicle Map during the instance lifetime: number of segments, various sizes,
|
||||
immutable for Chronicle Map during the data store lifetime: number of segments, various sizes,
|
||||
offsets, etc. See the specification of the fields on [Map Header Fields](3_1-header-fields.md) page.
|
||||
|
||||
## Global mutable state
|
||||
@@ -82,7 +82,7 @@ The global mutable state is 33 bytes long.
|
||||
4. Bytes 16..20 - the number of used extra segment tiers. An unsigned 40-bit value, stored in
|
||||
the little-endian order.
|
||||
5. Bytes 21..24 - the offset of the segment headers area from the beginning of the memory of this
|
||||
Chronicle Map instance. An unsigned 32-bit value, stored in the little-endian ordered. This field
|
||||
Chronicle Map store. An unsigned 32-bit value, stored in the little-endian ordered. This field
|
||||
determines the size of [the 4th area of the general Chronicle Map structure](
|
||||
#segment-headers-alignment).
|
||||
6. Bytes 25..32 - the Chronicle Map data store size, the offset to the end of the [main segments
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# 3.1. Chronicle Map Data Store Header Fields
|
||||
|
||||
Header of a persisted Chronicle Map instance is written once, when the instance and the file are
|
||||
Header of a persisted Chronicle Map store is written once, when the instance and the file are
|
||||
created, and never changed after that. It is stored in [Text Wire Format
|
||||
](https://github.com/OpenHFT/RFC/blob/master/Wire-Format-API/Text/Text-Wire-Format-API-0.1.md)).
|
||||
The top-level structure name is `!net.openhft.chronicle.map.VanillaChronicleMap`. It contains the
|
||||
|
||||
@@ -7,7 +7,7 @@
|
||||
|
||||
## Chronicle Map creation
|
||||
|
||||
This section describes the process of creation of a new Chronicle Map instance (or opening the
|
||||
This section describes the process of creation of a new Chronicle Map data store (or opening the
|
||||
existing Map, persisted to some file).
|
||||
|
||||
1. If the Chronicle Map should be persisted to some file, and this file does not exist, this file
|
||||
|
||||
+1
-1
@@ -21,7 +21,7 @@ This is currently work-in-progress.
|
||||
|
||||
- To provide the ability to write a Chronicle Map implementation in a non-JVM language e.g. C, C++,
|
||||
C#, or Go. As Chronicle Map is designed for concurrent inter-process access, performance-critical
|
||||
operations on a Chronicle Map instance could be done from a C++ process, while a concurrent process
|
||||
operations on a Chronicle Map store could be done from a C++ process, while a concurrent process
|
||||
running the Java implementation is doing some duty tasks such as backup to a rational database.
|
||||
|
||||
- This specification is the source of truth to reason about the Chronicle Map data store behaviour
|
||||
|
||||
Reference in New Issue
Block a user