diff --git "a/C:\\Users\\buddy\\dev\\shm\\OPERAND_CHRONICLE_MAP" "b/C:\\Users\\buddy\\dev\\shm\\OPERAND_CHRONICLE_MAP" new file mode 100644 index 000000000..191599e8e Binary files /dev/null and "b/C:\\Users\\buddy\\dev\\shm\\OPERAND_CHRONICLE_MAP" differ diff --git "a/C:\\Users\\buddy\\dev\\shm\\OPERAND_ChronicleStampedLock" "b/C:\\Users\\buddy\\dev\\shm\\OPERAND_ChronicleStampedLock" new file mode 100644 index 000000000..3132cb74a Binary files /dev/null and "b/C:\\Users\\buddy\\dev\\shm\\OPERAND_ChronicleStampedLock" differ diff --git "a/C:\\Users\\buddy\\dev\\shm\\OPERAND_ChronicleStampedLock=ReaderCount" "b/C:\\Users\\buddy\\dev\\shm\\OPERAND_ChronicleStampedLock=ReaderCount" new file mode 100644 index 000000000..d85ddf55e Binary files /dev/null and "b/C:\\Users\\buddy\\dev\\shm\\OPERAND_ChronicleStampedLock=ReaderCount" differ diff --git "a/C:\\Users\\buddy\\dev\\shm\\OPERAND_ChronicleStampedLock=WriterCount" "b/C:\\Users\\buddy\\dev\\shm\\OPERAND_ChronicleStampedLock=WriterCount" new file mode 100644 index 000000000..48886ec1a Binary files /dev/null and "b/C:\\Users\\buddy\\dev\\shm\\OPERAND_ChronicleStampedLock=WriterCount" differ diff --git a/spec/1-design-goals.md b/spec/1-design-goals.md index 293ec81db..eeebd13bd 100644 --- a/spec/1-design-goals.md +++ b/spec/1-design-goals.md @@ -4,7 +4,7 @@ arbitrary sequences of bytes. - Chronicle Map optionally persists to a *single file* via the memory-mapping facility, present in Windows and POSIX-compatible operating systems (`mmap`). The file shouldn't necessarily materialize - on disk. It's OK for Chronicle Map if the file resides a memory-mounted file system. The whole + on disk. It's OK for Chronicle Map if the file resides on a memory-mounted file system. The whole Chronicle Map state is contained in the file contents i. e. it is possible to move, copy or send the file to another machine, access it and observe exactly the same Chronicle Map state. Chronicle Map doesn't use the metadata of the file. @@ -16,7 +16,7 @@ https://en.wikipedia.org/wiki/ACID) here; see the detailed definition below in [Guarantees](#guarantees-1) section). - Chronicle Map runs only on little-endian architectures. - - Chronicle Map is *not* distributed: the whole data store resides the memory of a single machine + - Chronicle Map is *not* distributed: the whole data store resides in the memory of a single machine (a single file, if persisted). ## Guarantees @@ -35,7 +35,7 @@ Chronicle Map implementation. It might be an OS thread or a "greener" thread. #### CPU - - CPU supports atomic 64-bit compare-and-swap operations with aligned memory, i. e. if several + - The CPU supports atomic 64-bit compare-and-swap operations with aligned memory, i. e. if several threads, possibly belonging to different processes, try to perform compare-and-swap operation on the same 4-byte block, at most one thread will succeed and all the rest will fail. - Aligned 32-bit or 64-bit writes are atomic, that means either all 4(8) bytes are written to @@ -48,7 +48,7 @@ The two above points are true for CPUs with x86 and x86_64 architectures. - If some values Xi are written at addresses Ai, then 32- or 64-bit value Y is written at address B, there is a way to ensure some memory order. In particular, if a - concurrent thread reads value Y from address B, after that it will read Xii values from Ai addresses (but not the old values, stored at those addresses). > In terms of the Java Memory model, this assumption means "writes to Ai addresses diff --git a/spec/2-design-overview.md b/spec/2-design-overview.md index 097263695..ed7accc3f 100644 --- a/spec/2-design-overview.md +++ b/spec/2-design-overview.md @@ -161,7 +161,6 @@ after the previous tier.* All tiers, either first in their segments or chained,
  1. If the keys are identical, the entry for the queried key within the segment is found, exit the procedure.
    2. -
  2. If the keys are not identical (because of a full collision of parts of their hash codes for storing in hash lookup tables), continue the search in the current tier's hash lookup table (step #2), without resetting the hash lookup slot index.
    3. diff --git a/spec/3-memory-layout.md b/spec/3-memory-layout.md index cbaa7404b..2312eccaf 100644 --- a/spec/3-memory-layout.md +++ b/spec/3-memory-layout.md @@ -13,7 +13,7 @@ one big continuous block of memory. Its structure, from lower addresses to highe persisted to disk. 3. [The global mutable state](#global-mutable-state). - 4. + 4. [Segment header alignment](#segment-headers-alignment") The alignment to the next page boundary by addresses (page size of the current memory mapping), if a new Chronicle Map is created, if an existing one (i. e. persisted) is loaded, the alignment is read from the global mutable state (specifically this field of the global mutable state is @@ -21,7 +21,7 @@ one big continuous block of memory. Its structure, from lower addresses to highe > The purpose of this alignment is to minimize the number of pages spanned by the following > *segment headers area*. The segment headers area is frequently accessed and updated, so the pages - > it spans almost always reside in TLB cache and always need to be flushed to the disk. + > it spans almost always reside in the TLB cache and always need to be flushed to the disk. > The alignment (more precisely, the offset to the next area, the segment headers area) is stored > in the global mutable state, rather then computed each time a mapped Chronicle Map store is @@ -30,18 +30,18 @@ one big continuous block of memory. Its structure, from lower addresses to highe 5. [The segment headers area](#segment-headers-area). 6. [The main segments area](#main-segments-area). - 7. Zero or several [extra tier bulks](#extra-tier-bulks). + 7. Zero or more [extra tier bulks](#extra-tier-bulks). The process of initialization of the Chronicle Map's memory is described on [Initialization Order]( 5-initialization.md) page. ## Self-bootstrapping header -The concept is described in [Self Bootstrapping Data]( +The concept is described in the [Self Bootstrapping Data]( https://github.com/OpenHFT/RFC/blob/master/Self-Bootstrapping-Data/Self-Bootstraping-Data-0.1.md) specification. -The structure of this area is described in [Size Prefixed Blob]( +The structure of this area is described in the [Size Prefixed Blob]( https://github.com/OpenHFT/RFC/blob/master/Size-Prefixed-Blob/Size-Prefixed-Blob-0.1.md) specification. The first 8 bytes contains the hash value of bytes sequence from 9th byte to the end of this size-prefixed blob, computed by [xxHash](https://github.com/Cyan4973/xxHash/) algorithm @@ -63,12 +63,13 @@ The global mutable state is 33 bytes long. > global mutable state and segment headers (see below). 2. Bytes 8..10 - the number of allocated [extra tier bulks](#extra-tier-bulks). An unsigned 24-bit - value, stored in the little-endian order. + value, stored in little-endian order. See also the [extra tier bulk allocation](5-initialization.md#entra-tier-bulk-allocation) operation. + 3. Bytes 11..15 - the index of the first *free* segment tier. An unsigned 40-bit value, stored - in the little-endian order. Extra segment tiers are allocated in bulks, so there is usually a chain + in little-endian order. Extra segment tiers are allocated in bulks, so there is usually a chain of allocated, but unused yet segment tiers. This field points to the head of this chain, or has value `0`, if there are no free segment tiers. @@ -79,15 +80,18 @@ The global mutable state is 33 bytes long. [`actualSegments`](3_1-header-fields.md#actualsegments) − 1) has tier index `actualSegments`, the first tier of the first extra tier bulk has tier index `actualSegments` + 1, etc. Tier indexes are 1-counted, because value 0 has some special meaning. + 4. Bytes 16..20 - the number of used extra segment tiers. An unsigned 40-bit value, stored in - the little-endian order. + little-endian order. + 5. Bytes 21..24 - the offset of the segment headers area from the beginning of the memory of this - Chronicle Map store. An unsigned 32-bit value, stored in the little-endian ordered. This field + Chronicle Map store. An unsigned 32-bit value, stored in little-endian ordered. This field determines the size of [the 4th area of the general Chronicle Map structure]( #segment-headers-alignment). + 6. Bytes 25..32 - the Chronicle Map data store size, the offset to the end of the [main segments area](#main-segments area) or the last [extra tier bulk](#extra-tier-bulks). A non-negative 64-bit - value, stored in the little-endian order. + value, stored in little-endian order. > The reference Java implementation: [`VanillaGlobalMutableState` > ](../src/main/java/net/openhft/chronicle/hash/VanillaGlobalMutableState.java). @@ -97,7 +101,7 @@ The global mutable state is 33 bytes long. The offset to this area is stored in the 5th field of the [global mutable state ](#global-mutable-state). -The size (in bytes) of a segment headers area is [`actualSegments`]( +The size (in bytes) of a segment header's area is [`actualSegments`]( 3_1-header-fields.md#actualsegments) * [`segmentHeaderSize`](3_1-header-fields.md#segmentheadersize ). Each segment header starts at offsets from the start of the segment headers area, that are multiples of `segmentHeaderSize`. Each segment header is 32 bytes long. `segmentHeaderSize` − @@ -126,6 +130,7 @@ multiples of `segmentHeaderSize`. Each segment header is 32 bytes long. `segment It is a 64-bit value, stored in the little-endian order. If the value of this field is 0, this means there is no chained segment tier in this segment yet after the first tier, in other words, the first tier is the only one in the chain for the current segment. + 5. Bytes 24..31 - reserved for use by extensions. > The reference Java implementation: [`BigSegmentHeader` @@ -133,13 +138,13 @@ multiples of `segmentHeaderSize`. Each segment header is 32 bytes long. `segment ## Main segments area -This area contains first tiers of the Chronicle Map's segments. +This area contains the first tiers of the Chronicle Map's segments. A main segments area starts immediately after the end of a segment headers area without extra offsets and alignments. The size of a main segments area is [`actualSegments`](3_1-header-fields.md#actualsegments) * -[`tierSize`](3_1-header-fields.md#tiersize). Segment tiers in a main segments area doesn't have +[`tierSize`](3_1-header-fields.md#tiersize). Segment tiers in a main segments area don't have extra gaps in memory between each other. ### Segment tier structure @@ -147,7 +152,7 @@ extra gaps in memory between each other. See also [segment tiers design overview](2-design-overview.md#segment-tier) for more explanations about this structure. -Segment tier is [`tierSize`](3_1-header-fields.md#tiersize) bytes long. +A segment tier is [`tierSize`](3_1-header-fields.md#tiersize) bytes long. The segment tier structure: @@ -163,7 +168,7 @@ A hash lookup area starts at the same address as a segment tier, containing it. A hash lookup consists of [`tierHashLookupCapacity`](3_1-header-fields.md#tierhashlookupcapacity) slots each of [`tierHashLookupSlotSize`](3_1-header-fields.md#tierhashlookupslotsize) bytes. In each slot a 32-bit or 64-bit (if the `tierHashLookupSlotSize` is 4 or 8, respectively) value is stored in -little-endian order. The slot value of 0 designates an empty slot. +little-endian order. A slot value of 0 designates an empty slot. ##### Hash lookup key diff --git a/spec/3_2-lock-structure.md b/spec/3_2-lock-structure.md index 9d1730e93..83ed46d79 100644 --- a/spec/3_2-lock-structure.md +++ b/spec/3_2-lock-structure.md @@ -2,11 +2,13 @@ The lock structure is 8 bytes (64 bits) long. - 1. *Count word*: + *Count word*: 1. Bits 0..29 - read lock count (little-endian) 2. Bit 30 - update lock flag 3. Bit 31 - write lock flag - 2. *Wait word*: bits 32..63 - wait count (little-endian) + +*Wait word*: + 1. bits 32..63 - wait count (little-endian) ## Try acquire read lock @@ -107,6 +109,7 @@ procedure and call one depending on the context. + ## Release write lock, or write to update lock downgrade, or write to read lock downgrade Perform a CAS operation on the count word of the lock state, comparing 0x80000000 (i. e. a count @@ -114,7 +117,7 @@ word with the write lock flag set) and swapping with 0 (in case of releasing wri 0x40000000 (in case of write to update lock downgrade), or 1 (in case of write to read lock downgrade). The result of the CAS operation is the result of the procedure. -## Try upgrade update to write lock +## Try upgrade to write lock Perform a CAS operation on the count word of the lock state, comparing 0x40000000 (i. e. a count word with the update lock flag set, the write lock flag not set, the read lock count of zero) and @@ -140,6 +143,7 @@ procedure. + ## Time-limited write lock acquisition or update to write upgrade 1. Perform the corresponding *try acquire* procedure ([write lock](#try-acquire-write-lock) or diff --git a/spec/5-initialization.md b/spec/5-initialization.md index 2e613181e..8c46c66e1 100644 --- a/spec/5-initialization.md +++ b/spec/5-initialization.md @@ -17,7 +17,7 @@ existing Map, persisted to some file). [wait until the Chronicle Map is ready](#wait-until-chronicle-map-is-ready). Otherwise, acquire an exclusive file lock, and check the file size again. If the file size is non-zero now, release the file lock and [wait until the Chronicle Map is ready](#wait-until-chronicle-map-is-ready). If - the file is still empty, [write self bootstrapping header](#write-self-bootstrapping-header) to the + the file is still empty, [write the self bootstrapping header](#write-self-bootstrapping-header) to the beginning of the file, then release the file lock. > The purpose of acquiring file lock is to ensure that only one concurrent process (if any) @@ -49,7 +49,7 @@ existing Map, persisted to some file). out memory from the beginning of the tier to the start of the [entry space]( 3-memory-layout.md#entry-space), i. e. zero out this tier's hash lookup, segment tier counters area and free list. - 6. White the segment headers offset into the 5th field of the global mutable state. + 6. Write the segment headers offset into the 5th field of the global mutable state. 7. Write the offset to the end of the main segments area into the 6th field of the global mutable state. 8. If the Chronicle Map is persisted, ensure all data written to the file is flushed to the disk.