diff --git a/spec/0-purposes.md b/spec/0-purposes.md deleted file mode 100644 index 76e16f541..000000000 --- a/spec/0-purposes.md +++ /dev/null @@ -1,23 +0,0 @@ -# Rational behind The Chronicle Map Data Store Specification - - - To provide the ability to write a small, drastically simplified Chronicle Map implementation in Java - for a specific use-case. It could hard-code many aspects which the reference implementation has - to handle as abstractions which are harder for the JVM to execute efficiently. Another way of improving - performance is by weakening guarantees implied by the specification and the reference implementation. - - - To provide the ability to write a Chronicle Map implementation in a non-JVM language e.g. C++, C#. - As Chronicle Map is designed for concurrent inter-process access, performance-critical operations - on a Chronicle Map instance could be done from a C++ process, while a concurrent Java process - is doing some duty tasks such as backup to a rational database. - - - Chronicle Map specification is the source of truth to reason about the Chronicle Map data - structure behaviour and correctness. If the reference implementation doesn't follow the - specification, it is considered as an implementation bug. - - - To provide the ability for Chronicle Map adopters, database developers and researchers to study - Chronicle Map design and possibly draw their own conclusions about it's behaviour, guarantees and - efficiency. The reference implementation is quite complex so it's hard to understand how it - works by just reading the code, especially for people who don't know - Java. - - - To keep the rationalization of the design decisions together in one place. diff --git a/spec/1-design-goals.md b/spec/1-design-goals.md index 5fa3fec56..e4a4634bd 100644 --- a/spec/1-design-goals.md +++ b/spec/1-design-goals.md @@ -1,4 +1,4 @@ -# Chronicle Map Data Store Design Goals, Assumptions and Guarantees +# 1. Chronicle Map Data Store Design Goals, Assumptions and Guarantees - Chronicle Map is an in-memory, highly concurrent key-value store. Generally keys and values are arbitrary sequences of bytes. diff --git a/spec/2-design-overview.md b/spec/2-design-overview.md index 740399fec..fb78cc909 100644 --- a/spec/2-design-overview.md +++ b/spec/2-design-overview.md @@ -1,4 +1,4 @@ -# Chronicle Map Data Store Design Overview +# 2. Chronicle Map Data Store Design Overview ## Logic @@ -82,7 +82,8 @@ certain threshold numbers of unsuccessful attempts. ## The segment data structure -Under normal conditions, a segment is represented by a single *tier*. +Under normal conditions, a segment is represented by a single *tier*. (*Tier* and *segment tier* are +interchangeable terms in this specification.) ### Segment tier diff --git a/spec/3-memory-layout.md b/spec/3-memory-layout.md index 5d7abc615..fd026a5ee 100644 --- a/spec/3-memory-layout.md +++ b/spec/3-memory-layout.md @@ -1,35 +1,312 @@ -# Chronicle Map Data Store Memory Layout and Initialization +# 3. Chronicle Map Data Store Memory Layout A newly created Chronicle Map instance (or just mapped from an already existing persistence file) takes one big continuous block of memory. Its structure, from lower addresses to higher: - 1. [Self-bootstrapping header](#self-bootstrapping-header) - 2. [Global mutable state](#global-mutable-state) - 3. Alignment to the next page boundary by addresses (page size of the current memory mapping), + 1. [The self-bootstrapping header](#self-bootstrapping-header). + 2. The alignment to the next cache line boundary (to the next multiple of 64) by addresses. + + > The purpose of this alignment is to make the *global mutable state* to always fit a single cache + > line. + + The self-bootstrapping header and the alignment are not present, when the Chronicle Map is not + persisted to disk. + + 3. [The global mutable state](#global-mutable-state). + 4. + The alignment to the next page boundary by addresses (page size of the current memory mapping), if a new Chronicle Map is created, if an existing one (i. e. persisted) is loaded, the alignment - is specified in the global mutable state (specifically this field of global mutable state is + is read from the global mutable state (specifically this field of the global mutable state is actually immutable, once written). > The purpose of this alignment is to minimize the number of pages spanned by the following > *segment headers area*. The segment headers area is frequently accessed and updated, so the pages > it spans almost always reside in TLB cache and always need to be flushed to the disk. - > The alignment is persisted because other accessing processes, operation systems and - > implementations of Chronicle Map may use different page size for memory mapping. + > The alignment (more precisely, the offset to the next area, the segment headers area) is stored + > in the global mutable state, rather then computed each time a mapped Chronicle Map instance is + > accessed, because other accessing file systems, operation systems and Chronicle Map + > implementations may use a different page size for memory mapping. - 4. [Segment headers area](#segment-headers-area) - 5. [Main segments area](#main-segments-area) - 6. Zero to several [extra tier bulks](#extra-tier-bulks). + 5. [The segment headers area](#segment-headers-area). + 6. [The main segments area](#main-segments-area). + 7. Zero or several [extra tier bulks](#extra-tier-bulks). -## Self-Bootstrapping Header +## Self-bootstrapping header -It's structure and initialization order is described in [Self Bootstrapping Data 1.0]( +It's structure and initialization order are described in the [Self Bootstrapping Data]( https://github.com/OpenHFT/RFC/blob/master/Self-Bootstrapping-Data/Self-Bootstraping-Data-0.1.md) -specification. It could be in Binary Wire or Text Wire formats (see details in the Self -Bootstrapping Data specification). Once created, this header is never changed. It contains all -configurations, immutable for Chronicle Map during the instance "lifetime": number of segments, -various sizes, offsets, etc. See the specification of the fields on [Map Header -Fields](3_1-map-header-fields.md) page. +specification. It is encoded in Binary Wire format. Once created, this header is never changed. It +contains all configurations, immutable for Chronicle Map during the instance lifetime: number of +segments, various sizes, offsets, etc. See the specification of the fields on [Map Header +Fields](3_1-header-fields.md) page. -## Global Mutable State +## Global mutable state + +The global mutable state is 36 bytes long. + + 1. Bytes 0..7 - the global mutable state lock, a [lock structure](3_2-lock-structure.md). + + > The lock structure supports three levels of locking, while for the segment header only exclusive + > (the write lock) level is used. This is done for sharing the lock implementation for the + > global mutable state and segment headers (see below). + + 2. Bytes 8..11 - the number of allocated [extra tier bulks](#extra-tier-bulks). A non-negative + 32-bit value, stored in the little-endian order. + 3. Bytes 12..19 - the index of the first *free* segment tier. A non-negative 64-bit value, stored + in the little-endian order. Extra segment tiers are allocated in bulks, so there is usually a chain + of allocated, but unused yet segment tiers. This field points to the head of this chain, or has + value `0`, if there are no free segment tiers. + + ##### Tier index + *Tier index* is *1-counted* from the beginning of the main segments area, and counting continues in + the extra tier bulks, i. e. the first tier of the segment #0 has *tier index* 1, the first tier of + the segment #1 has tier index 2, ... the first tier of the last segment (it's index is + [`actualSegments`](3_1-header-fields.md#actualSegments) − 1) has tier index `actualSegments`, + the first tier of the first extra tier bulk has tier index `actualSegments` + 1, etc. Tier indexes + are 1-counted, because value 0 has some special meaning. + 4. Bytes 20..27 - the number of used extra segment tiers. A non-negative 64-bit value, stored in + the little-endian order. + 5. Bytes 28..35 - the offset of the segment headers area from the beginning of the memory of this + Chronicle Map instance. This field determines the size of [the 4th area of the general Chronicle + Map structure](#segment-headers-alignment). + +> The reference Java implementation: [`VanillaGlobalMutableState` +> ](../src/main/java/net/openhft/chronicle/hash/VanillaGlobalMutableState.java). + +## Segment headers area + +The offset to this area is stored in the 5th field of the [global mutable state +](#global-mutable-state). + +The size (in bytes) of a segment headers area is [`actualSegments`]( +3_1-header-fields.md#actualSegments) * [`segmentHeaderSize`](3_1-header-fields.md#segmentHeaderSize +). Each segment header starts at offsets from the start of the segment headers area, that are +multiples of `segmentHeaderSize`. Each segment header is 32 bytes long. `segmentHeaderSize` − +32 is the alignment between segment headers. + +> The purpose of the alignment between segment headers is reducing false sharing, when adjacent +> segment headers are accessed concurrently. + +### Segment header structure + + 1. Bytes 0..7 - the segment [lock structure](3_2-lock-structure.md). + 2. Bytes 8..11 - the number of entries, stored in the first tier of the segment. A 32-bit + *unsigned* value, stored in the little-endian order. + 3. Bytes 12..15 - the smallest index of a chunk in the entry space of the first tier of the + segment, that could possibly be free. A 32-bit *unsigned* value, stored in the little-endian order. + This field is used to optimize allocation of space for new entries, the search for a sufficient + range of continuous free chunks in the [free list](#free-list) is started from this index, rather + than 0. This field is updated on each entry allocation and deletion in the first tier of the + segment, if the smallest index of a free chunk is changed. When all chunks in the entry space are + allocated (i. e. the current tier becomes *full*), the value of this field is changed to + [`actualChunksPerSegmentTier`](3_1-header-fields.md#actualChunksPerSegmentTier) (an impossible + chunk index, which varies from 0 to `actualChunksPerSegmentTier` − 1). + 4. Bytes 16..31 - reserved for use by extensions. + +> The reference Java implementation: [`BigSegmentHeader` +> ](../src/main/java/net/openhft/chronicle/hash/impl/BigSegmentHeader.java) + +## Main segments area + +This area contains first tiers of the Chronicle Map's segments. + +A main segments area starts immediately after the end of a segment headers area without extra +offsets and alignments. + +The size of a main segments area is [`actualSegments`](3_1-header-fields.md#actualSegments) * +[`tierSize`](3_1-header-fields.md#tierSize). Segment tiers in a main segments area doesn't have +extra gaps in memory between each other. + +### Segment tier structure + +See also [segment tiers design overview](2-design-overview.md#segment-tier) for more explanations +about this structure. + +Segment tier is [`tierSize`](3_1-header-fields.md#tierSize) bytes long. + +The segment tier structure: + + 1. [Hash lookup](#hash-lookup) + 2. [Segment tier counters area](#segment-tier-counters-area) + 3. [Free list](#free-list) + 4. [Entry space](#entry-space) + +#### Hash lookup + +A hash lookup area starts at the same address as a segment tier, containing it. + +A hash lookup consists of [`tierHashLookupCapacity`](3_1-header-fields.md#tierHashLookupCapacity) +slots each of [`tierHashLookupSlotSize`](3_1-header-fields.md#tierHashLookupSlotSize) bytes. In each +slot a 32-bit or 64-bit (if the `tierHashLookupSlotSize` is 4 or 8, respectively) value is stored in +little-endian order. The slot value of 0 designates an empty slot. + +In [`tierHashLookupKeyBits`](3_1-header-fields.md#tierHashLookupKeyBits) lower bits of a slot value +a *hash lookup key* is stored. It is a part of a Chronicle Map's key hash code, extracted by the +[`hashSplitting`](3_1-header-fields.md#hashSplitting) algorithm. In addition, if the `hashSplitting` +extracts the part of the hash code of 0, a hash lookup key of all set `tierHashLookupKeyBits` bits +(i. e. `(1 << tierHashLookupKeyBits) - 1`) is used instead, to avoid the full hash lookup slot to +look like an empty slot, if the *hash lookup value* is also 0. + +In [`tierHashLookupValueBits`](3_1-header-fields.md#tierHashLookupValueBits) bits, following after +the lower key bits (i. e. bits from `tierHashLookupKeyBits`-th to `tierHashLookupKeyBits + +tierHashLookupValueBits - 1`-th, inclusive) of a slot value a *hash lookup value* is stored. It is +an index of the first chunk of the range of chunks (possibly only a single chunk) in the entry space +of this segment tier, in which a Chronicle Map's entry is stored. + +> The reference Java implementation: [`CompactOffHeapLinearHashTable` +> ](../src/main/java/net/openhft/chronicle/hash/impl/CompactOffHeapLinearHashTable.java) (the base +> class), [`IntCompactOffHeapLinearHashTable` +> ](../src/main/java/net/openhft/chronicle/hash/impl/IntCompactOffHeapLinearHashTable.java) (the +> subclass for hash lookups with 4-byte slots), [`LongCompactOffHeapLinearHashTable` +> ](../src/main/java/net/openhft/chronicle/hash/impl/LongCompactOffHeapLinearHashTable.java) (the +> subclass for hash lookups with 8-byte slots). + +#### Segment tier counters area + +A segment tier counters area starts with offset [`tierHashLookupOuterSize` +](3_1-header-fields.md#tierHashLookupOuterSize) from the beginning address of a containing segment +tier. + +The segment tier counters structure is 64 bytes long: + + 1. Bytes 0..7 - the [index](#tier-index) of the next segment tier, chained after this segment tier. + In tiers, which are first in their segments (i. e. they reside the [main segments area + ](#main-segments-area)) this index points to the second tier in the chain, this field value in the + second tier in the chain - to the third tier in the chain, and so on. + + It is a 64-bit value, stored in the little-endian order. If the value of this field is 0, this + means there is no chained segment tier in this segment yet after the current tier, in other words, + the current tier is the last in the chain for the current segment. + + When the tier is in the *free* state, i. e. allocated in the extra tier bulk, but not yet assigned + to some segment, the value of this field is the index of the next *free* tier. The index of the + first free tier is pointed by the 3rd field of the [global mutable state](#global-mutable-state). + + 2. Bytes 8..15 - the [index](#tier-index) of the previous segment tier, chained before this segment + tier. It is a 64-bit value, stored in the little-endian order. In tiers in the main segments area, + i. e. first tiers in segments' chains, this field has value 0, this means there is no previous + tiers in chains. + + This field together with the previous field form a doubly-linked list of chained tiers within each + segment. + + 3. Bytes 16..23 - for tiers from extra tier bulks (i. e. tiers that are not first in chains of + their segments), this field is the smallest index of a chunk in the entry space of this tier, + that could possibly be free. For tiers from the main segment area (i. e. first in chains of their + segments), this field is unused, the 3rd field of the [segment header structure + ](#segment-header-structure) is used instead. This field and the 3rd field in the segment header + structure have the same semantics, with the only difference that this field serves chained tiers + (tiers from extra tier bulks), while the 3rd field in the segment header structure serves first + tiers in chains of their segments. + + A 64-bit non-negative value, stored in the little-endian order. + + > This field is effectively "pulled" from the tier counters structure to the segment header + > structure for tiers in the main segment area, because if there is only one tier in the chain + > (that should be true for the majority of (or all) segments), the tier counters area is not + > touched, reducing the number of accessed cache lines and pollution of caches. + + 4. Bytes 24..27 - the index of the segment, to which this tier belongs. A 32-bit non-negative + value, stored in the little-endian order. + + 5. Bytes 28..31 - the 0-counted order of this tier in the chain of the segment. A 32-bit + non-negative value, stored in the little-endian order. In the tiers from the main segments area the + value of this field is 0 (because they are first in chains of their segments), in the tiers which + are second in chains of their segments the value of this field is 1, in third tiers - 2, and so on. + + 6. Bytes 32..35 - for tiers that from extra tier bulks (i. e. not first in chains of their segments + ), this field is the number of entries, stored in the tier. In tiers from the main segments area + this field is unused, the 2nd field of the [segment header structure](#segment-header-structure) is + used instead. Like the 3rd field of the tier counters structure, this field and the 2nd field in + the segment header structure have the same semantics, with the only difference that this field + serves chained tiers (tiers from extra tier bulks), while the 3rd field in the segment header + structure serves first tiers in chains of their segments. + + A 32-bit unsigned value, stored in the little-endian order. + + > Again, like the 3rd field of this structure, this field is "pulled" to the segment headers in + > order to avoid accessing tier counters area altogether, when there is only one tier in a segment + > chain. + + 7. Bytes 36..63 - reserved for use by extensions. + +> The reference Java implementation: [`TierCountersArea` +> ](../src/main/java/net/openhft/chronicle/hash/impl/TierCountersArea.java). + +#### Free list + +A free list starts with offset [`tierHashLookupOuterSize` +](3_1-header-fields.md#tierHashLookupOuterSize) + 64 from the beginning address of a containing +segment tier. + +This is a list of [`actualChunksPerSegmentTier`](3_1-header-fields.md#actualChunksPerSegmentTier) +bits, each corresponding to a chunk in the [entry space](#entry-space). If the chunk is allocated +for some entry, the corresponding bit is set to 1, otherwise it is clear. + +> The reference Java implementation: [`SingleThreadedFlatBitSetFrame` +> ](https://github.com/OpenHFT/Chronicle-Algorithms/blob/chronicle-algorithms-1.1.6/src/main/java/net/openhft/chronicle/algo/bitset/SingleThreadedFlatBitSetFrame.java). + +#### Entry space + +An entry space starts with offset [`tierHashLookupOuterSize` +](3_1-header-fields.md#tierHashLookupOuterSize) + 64 + [`tierFreeListOuterSize` +](3_1-header-fields.md#tierFreeListOuterSize) from the beginning address of a containing segment +tier. + +The entry space starts with an internal offset of [`tierEntrySpaceInnerOffset` +](3_1-header-fields.md#tierEntrySpaceInnerOffset) bytes. Then [`actualChunksPerSegmentTier` +](3_1-header-fields.md#actualChunksPerSegmentTier) *chunks* follow without gaps between each other, +each chunk of [`chunkSize`](3_1-header-fields.md#chunkSize) bytes. + +A chunk is the minimum allocation unit. A single or several continuous chunks are used to store the +Chronicle Map's entries. These ranges doesn't intersect, i. e. each chunk is used to store at most +one entry. + +##### Stored entry structure + +Within the given range of chunks, the entry is stored starting from the beginning of the first chunk +of the range. The stored entry structure is: + + 1. The length of the key (in bytes), stored using the [`keySizeMarshaller` + ](3_1-header-fields.md#keySizeMarshaller) algorithm. + 2. The key itself stored, a sequence of bytes of the length, determined by the previous field of + this structure. The key is stored using the [`keyDataAccess`](3_1-header-fields.md#keyDataAccess) + strategy. + 3. The length of the value (in bytes), stored using the [`valueSizeMarshaller` + ](3_1-header-fields.md#valueSizeMarshaller) algorithm. + 4. The gap (likely 0-long, though), imposed by the value [`alignment` + ](3_1-header-fields.md#alignment) strategy. + 5. The value itself stored, a sequence of bytes of the length, determined by the 3rd field of this + structure. The value is stored using the [`valueDataAccess`](3_1-header-fields.md#valueDataAccess) + strategy. + 6. (Optional field) If [`checksumEntries`](3_1-header-fields.md#checksumEntries) is `true`, there + is a 4-byte checksum stored after the value bytes. See also the [checksum algorithm + ](4-hashing-algorithms.md#checksum-algorithm.md). + +The memory between the end of the 6th field (or 5th, if checksums are not stored) and the end of the +last chunk, allocated for the entry, is unused. This is an internal fragmentation. + +> The reference Java implementation of the stored entry structure: primarily [`HashEntryStages` +> ](../src/main/java/net/openhft/chronicle/hash/impl/stage/entry/HashEntryStages.java) and +> [`MapEntryStages` +> ](../src/main/java/net/openhft/chronicle/map/impl/stage/entry/MapEntryStages.java) classes. + +## Extra tier bulks + +The size of an extra tier bulks area is [`tierBulkSizeInBytes` +](3_1-header-fields.md#tierBulkSizeInBytes), multiplied by the number of extra tier bulks, stored +in the 2nd field of the [global mutable state](#global-mutable-state) structure. Extra tier bulks +start immediately at the end of the main segments area (the end of the last segment tier in the main +segments area) and doesn't have gaps between each other. + +### Extra tier bulk + +Each extra tier bulk starts with metadata space of [`tierBulkInnerOffsetToTiers` +](3_1-header-fields.md#tierBulkInnerOffsetToTiers) (reserved for use by extensions), after that +[`tiersInBulk`](3_1-header-fields.md#tiersInBulk) segment tiers are located without gaps between +each other. + +See the [segment tier structure](#segment-tier-structure) for information about the detailed +structure of each particular tier in extra tier bulks. diff --git a/spec/3_1-map-header-fields.md b/spec/3_1-header-fields.md similarity index 93% rename from spec/3_1-map-header-fields.md rename to spec/3_1-header-fields.md index 97efcb2f4..f7c23bbd9 100644 --- a/spec/3_1-map-header-fields.md +++ b/spec/3_1-header-fields.md @@ -1,10 +1,10 @@ -# Chronicle Map Data Store Header Fields +# 3.1. Chronicle Map Data Store Header Fields Header of a persisted Chronicle Map instance is written once, when the instance and the file are -created, and never changed after that. Tt is stored in -[Binary Wire Format]( -https://github.com/OpenHFT/RFC/blob/master/Wire-Format-API/Binary/Binary-Wire-Format-API-0.1.md)) -with the following fields: +created, and never changed after that. It is stored in [Binary Wire Format +](https://github.com/OpenHFT/RFC/blob/master/Wire-Format-API/Binary/Binary-Wire-Format-API-0.1.md)). +The top-level structure name is `!net.openhft.chronicle.map.VanillaChronicleMap`. It contains the +following fields: ##### `dataFileVersion` @@ -141,10 +141,10 @@ The number of bits of the tier hash lookup's slots, used to store parts of the C hash codes, playing the role of key in hash lookup tables. The part of hash codes is extracted by the [`hashSplitting`](#hashSplitting) algorithm. A 32-bit value is [1, 63] range, e. g. `20`. -##### `tierHashLookupEntrySize` +##### `tierHashLookupSlotSize` The size of the tier hash lookup's slots, in bytes. A 32-bit value, `4` or `8`. Invariant: -`tierHashLookupKeyBits` + `tierHashLookupValueBits` = `tierHashLookupEntrySize` * 8. +`tierHashLookupKeyBits` + `tierHashLookupValueBits` = `tierHashLookupSlotSize` * 8. ##### `tierHashLookupCapacity` @@ -162,7 +162,7 @@ in a hash lookup. A positive 64-bit value, e. g. `1638`. ##### `tierHashLookupInnerSize` The size of hash lookups in bytes. A positive 64-bit value, equals to [`tierHashLookupInnerSize` -](#tierHashLookupInnerSize) * [`tierHashLookupEntrySize`](#tierHashLookupEntrySize). A positive +](#tierHashLookupInnerSize) * [`tierHashLookupSlotSize`](#tierHashLookupSlotSize). A positive 64-bit value, e. g. `8192`. ##### `tierHashLookupOuterSize` @@ -307,4 +307,10 @@ spaces. A positive, 32-bit, power of 2 value, e. g. `1`. ##### `worstAlignment` The most bytes could theoretically be wasted on the value [`alignment`](#alignment). A non-negative -32-bit value, e. g. `0`. \ No newline at end of file +32-bit value, e. g. `0`. + +> ## The reference Java implementation +> +> The described fields are defined in [`VanillaChronicleHash` +> ](../src/main/java/net/openhft/chronicle/hash/impl/VanillaChronicleHash.java) and +> [`VanillaChronicleMap`](../src/main/java/net/openhft/chronicle/map/VanillaChronicleMap.java). \ No newline at end of file diff --git a/spec/3_2-lock-structure.md b/spec/3_2-lock-structure.md index 0e069ecc4..3c9ab6d63 100644 --- a/spec/3_2-lock-structure.md +++ b/spec/3_2-lock-structure.md @@ -1,13 +1,12 @@ -# The Lock Structure +# 3.2. The Lock Structure and Locking Operations The lock structure is 8 bytes (64 bits) long. - - Count word - - Bits 0..29 - read lock count (little-endian) - - Bit 30 - update lock flag - - Bit 31 - write lock flag - - Wait word - - Bits 32..63 - wait count (little-endian) + 1. *Count word*: + 1. Bits 0..29 - read lock count (little-endian) + 2. Bit 30 - update lock flag + 3. Bit 31 - write lock flag + 2. *Wait word*: bits 32..63 - wait count (little-endian) ## The *try acquire read lock* procedure @@ -119,7 +118,7 @@ procedure. Implementations doesn't make locking attempts indefinitely [for the same reasons, as in case of acquiring read and update lock](#time-limited-read-or-update-lock-acquisition). -> The reference Java implementation: +> ## The reference Java implementation > > Attempt operations: [`VanillaReadWriteUpdateWithWaitsLockingStrategy`]( > https://github.com/OpenHFT/Chronicle-Algorithms/blob/chronicle-algorithms-1.1.6/src/main/java/net/openhft/chronicle/algo/locks/VanillaReadWriteUpdateWithWaitsLockingStrategy.java) diff --git a/spec/4-hashing-algorithms.md b/spec/4-hashing-algorithms.md new file mode 100644 index 000000000..ea16ea09a --- /dev/null +++ b/spec/4-hashing-algorithms.md @@ -0,0 +1,58 @@ +# The Chronicle Map Key Hashing and Checksum Algorithms + +Keys in Chronicle Map are arbitrary sequences of bytes. [xxHash +](https://github.com/Cyan4973/xxHash/) algorithm is applied to the key, to obtain the primary key +hash code. Then the [`hashSplitting`](3_1-header-fields.md#hashSplitting) algorithm is applied, to +determine the segment in which the key should be stored, and the part of the key hash code to be +stored in a segment tier's hash lookup. + +> The reference Java implementation: [XxHash_r39 +> ](https://github.com/OpenHFT/Chronicle-Algorithms/blob/chronicle-algorithms-1.1.6/src/main/java/net/openhft/chronicle/algo/hashing/XxHash_r39.java). +> Although the Java implementation class has `_r39` suffix, the xxHash algorithm is stable since r3 +> and [won't change in the future +> ](https://github.com/Cyan4973/xxHash/issues/34#issuecomment-169176338). A different version of +> the algorithm could have a different name. + +## Checksum algorithm + +### Primary checksum + +The primary checksum is a 64-bit value. + +If the 2nd field of the [stored entry structure +](3-memory-layout.md#stored-entry-structure) ends at the same address, as the 6th field of the same +structure starts, i. e. the value size is 0, and the size itself is stored using 0 bytes, and there +is no value alignment, the key hash code *is* the primary checksum. + +Otherwise, the [xxHash](https://github.com/Cyan4973/xxHash/) algorithm is applied to the memory +range between the end of the 2nd field of the stored entry structure and the end of the 5th field, +i. e. between the end of the stored key and the end of the stored value. The resulting hash value +is called *payload checksum*. + +> xxHash is used to compute the payload checksum instead of CRC32, because the Java implementation +> of CRC32 appears to be slower than xxHash while having the same (if not worse) quality, and the +> native CRC32 implementation, using specialized processor instructions, is available from Java only +> via JNI, having significant per-call costs, that eliminates any benefits. + +Then the following procedure is used to compute the primary checksum: +```java +long K2 = 0x9ae16a3b2f90404fL; +long mul = K2 + (keySize << 1); +long a = keyHashCode + K2; +long c = ((payloadChecksum >>> 37) | (payloadChecksum << 27)) * mul + a; +long d = (((a >>> 25) | (a << 39)) + payloadChecksum) * mul; +long a1 = (c ^ d) * mul ^ ((c ^ d) * mul >>> 47); +long primaryChecksum = ((d ^ a1) * mul ^ ((d ^ a1) * mul >>> 47)) * mul; +``` +where the `keySize` is the key sequence length in bytes, `keyHashCode` is the key hash code, and +the `payloadChecksum` is the payload checksum, computed on the previous step. The given procedure +code is in [Java language](https://docs.oracle.com/javase/specs/jls/se8/html/index.html). + +### Entry checksum + +Entry checksum is 4 bytes long, it is obtained by XOR-ing lowest and highest 32 bits of the [primary +checksum[#primary-checksum]. The entry checksum is stored in the 6th field of the [stored entry +structure](3-memory-layout.md#stored-entry-structure). + +> The reference Java implementation: [`HashEntryChecksumStrategy` +> ](../src/main/java/net/openhft/chronicle/hash/impl/stage/entry/HashEntryChecksumStrategy.java). \ No newline at end of file diff --git a/spec/README.md b/spec/README.md index 2af4b635c..841358bba 100644 --- a/spec/README.md +++ b/spec/README.md @@ -2,7 +2,33 @@ This is currently work-in-progress. - - [Rational behind this Specification](0-purposes.md) - - [Design Goals, Assumptions and Guarantees](1-design-goals.md) - - [Design overview](2-design-overview.md) - - [Memory layout](3-memory-layout.md) + 1. [Design goals, assumptions and guarantees](1-design-goals.md) + 2. [Design overview](2-design-overview.md) + 3. [Memory layout](3-memory-layout.md) + 1. [Header fields](3_1-header-fields.md) + 2. [Lock structure and locking operations](3_2-lock-structure.md) + 4. [Key hashing and checksum algorithms](4-hashing-algorithms.md) + +## Rational behind this Specification + + - To provide the ability to write a small, drastically simplified Chronicle Map implementation in + Java for a specific use-case. It could hard-code many aspects which the reference implementation + has to handle as abstractions which are harder for the JVM to execute efficiently. Another way of + improving performance is by weakening guarantees implied by the specification and the reference + implementation. + + - To provide the ability to write a Chronicle Map implementation in a non-JVM language e.g. C, C++, + C#, or Go. As Chronicle Map is designed for concurrent inter-process access, performance-critical + operations on a Chronicle Map instance could be done from a C++ process, while a concurrent process + running the Java implementation is doing some duty tasks such as backup to a rational database. + + - This specification is the source of truth to reason about the Chronicle Map data store behaviour + and correctness. If the reference implementation doesn't follow the specification, it is considered + as an implementation bug. + + - To provide the ability for Chronicle Map adopters, database developers and researchers to study + the Chronicle Map design and possibly draw their own conclusions about it's behaviour, guarantees + and efficiency. The reference implementation is quite complex so it's hard to understand how it + works by solely reading the code, especially for people who don't know Java. + + - To keep the rationalization of the design decisions together in one place. diff --git a/src/main/java/net/openhft/chronicle/hash/impl/CompactOffHeapLinearHashTable.java b/src/main/java/net/openhft/chronicle/hash/impl/CompactOffHeapLinearHashTable.java index 9e155082d..7d8339699 100644 --- a/src/main/java/net/openhft/chronicle/hash/impl/CompactOffHeapLinearHashTable.java +++ b/src/main/java/net/openhft/chronicle/hash/impl/CompactOffHeapLinearHashTable.java @@ -84,10 +84,10 @@ public abstract class CompactOffHeapLinearHashTable { private final long keyMask; private final long valueMask; - CompactOffHeapLinearHashTable(long capacity, int entrySize, int keyBits, int valueBits) { + CompactOffHeapLinearHashTable(long capacity, int slotSize, int keyBits, int valueBits) { this.capacityMask = capacity - 1L; - this.capacityMask2 = capacityMask * entrySize; + this.capacityMask2 = capacityMask * slotSize; this.keyBits = keyBits; this.keyMask = mask(keyBits); @@ -95,7 +95,7 @@ public abstract class CompactOffHeapLinearHashTable { } CompactOffHeapLinearHashTable(VanillaChronicleHash h) { - this(h.tierHashLookupCapacity, h.tierHashLookupEntrySize, h.tierHashLookupKeyBits, + this(h.tierHashLookupCapacity, h.tierHashLookupSlotSize, h.tierHashLookupKeyBits, h.tierHashLookupValueBits); } diff --git a/src/main/java/net/openhft/chronicle/hash/impl/VanillaChronicleHash.java b/src/main/java/net/openhft/chronicle/hash/impl/VanillaChronicleHash.java index c4407145b..50d597fd1 100644 --- a/src/main/java/net/openhft/chronicle/hash/impl/VanillaChronicleHash.java +++ b/src/main/java/net/openhft/chronicle/hash/impl/VanillaChronicleHash.java @@ -100,7 +100,7 @@ public abstract class VanillaChronicleHash 4) { + if (!privateAPI.aligned64BitMemoryOperationsAtomic() && tierHashLookupSlotSize > 4) { throw new IllegalStateException("aligned64BitMemoryOperationsAtomic() == false, " + - "but hash lookup slot is " + tierHashLookupEntrySize); + "but hash lookup slot is " + tierHashLookupSlotSize); } tierHashLookupCapacity = privateAPI.tierHashLookupCapacity(); maxEntriesPerHashLookup = (long) (tierHashLookupCapacity * MAX_LOAD_FACTOR); - tierHashLookupInnerSize = tierHashLookupCapacity * tierHashLookupEntrySize; + tierHashLookupInnerSize = tierHashLookupCapacity * tierHashLookupSlotSize; tierHashLookupOuterSize = CACHE_LINES.align(tierHashLookupInnerSize, BYTES); tierFreeListInnerSize = LONGS.align( @@ -248,7 +248,7 @@ public abstract class VanillaChronicleHash "tierHashLookupValueBits").int32(); tierHashLookupKeyBits = wireIn.read(() -> "tierHashLookupKeyBits").int32(); - tierHashLookupEntrySize = wireIn.read(() -> "tierHashLookupEntrySize").int32(); + tierHashLookupSlotSize = wireIn.read(() -> "tierHashLookupSlotSize").int32(); tierHashLookupCapacity = wireIn.read(() -> "tierHashLookupCapacity").int64(); maxEntriesPerHashLookup = wireIn.read(() -> "maxEntriesPerHashLookup").int64(); tierHashLookupInnerSize = wireIn.read(() -> "tierHashLookupInnerSize").int64(); @@ -292,7 +292,7 @@ public abstract class VanillaChronicleHash "tierHashLookupValueBits").int32(tierHashLookupValueBits); wireOut.write(() -> "tierHashLookupKeyBits").int32(tierHashLookupKeyBits); - wireOut.write(() -> "tierHashLookupEntrySize").int32(tierHashLookupEntrySize); + wireOut.write(() -> "tierHashLookupSlotSize").int32(tierHashLookupSlotSize); wireOut.write(() -> "tierHashLookupCapacity").int64(tierHashLookupCapacity); wireOut.write(() -> "maxEntriesPerHashLookup").int64(maxEntriesPerHashLookup); wireOut.write(() -> "tierHashLookupInnerSize").int64(tierHashLookupInnerSize); @@ -366,13 +366,13 @@ public abstract class VanillaChronicleHash(); - if (tierHashLookupEntrySize == 4) { + if (tierHashLookupSlotSize == 4) { hashLookup = new IntCompactOffHeapLinearHashTable(this); - } else if (tierHashLookupEntrySize == 8) { + } else if (tierHashLookupSlotSize == 8) { hashLookup = new LongCompactOffHeapLinearHashTable(this); } else { throw new AssertionError("hash lookup slot size could be 4 or 8, " + - tierHashLookupEntrySize + " observed"); + tierHashLookupSlotSize + " observed"); } }