diff --git a/spec/1-design-goals.md b/spec/1-design-goals.md index 41d11cc88..acb0d85be 100644 --- a/spec/1-design-goals.md +++ b/spec/1-design-goals.md @@ -71,7 +71,8 @@ Chronicle Map *doesn't* assume anything specific about how memory if flushed to #### File locking -The operating system supports exclusive [file locking](https://en.wikipedia.org/wiki/File_locking). +The operating system supports *advisory* exclusive [file locking]( +https://en.wikipedia.org/wiki/File_locking). ### Guarantees diff --git a/spec/3-memory-layout.md b/spec/3-memory-layout.md index d4accf2b2..67e61b2ac 100644 --- a/spec/3-memory-layout.md +++ b/spec/3-memory-layout.md @@ -32,14 +32,25 @@ takes one big continuous block of memory. Its structure, from lower addresses to 6. [The main segments area](#main-segments-area). 7. Zero or several [extra tier bulks](#extra-tier-bulks). +The process of initialization of the Chronicle Map's memory is described on [Initialization Order]( +5-initialization.md) page. + ## Self-bootstrapping header -It's structure and initialization order are described in the [Self Bootstrapping Data]( +The concept is described in [Self Bootstrapping Data]( https://github.com/OpenHFT/RFC/blob/master/Self-Bootstrapping-Data/Self-Bootstraping-Data-0.1.md) -specification. It is encoded in Text Wire format. Once created, this header is never changed. It -contains all configurations, immutable for Chronicle Map during the instance lifetime: number of -segments, various sizes, offsets, etc. See the specification of the fields on [Map Header -Fields](3_1-header-fields.md) page. +specification. + +The structure of this area is described in [Size Prefixed Blob]( +https://github.com/OpenHFT/RFC/blob/master/Size-Prefixed-Blob/Size-Prefixed-Blob-0.1.md) +specification. The first 8 bytes contains the hash value of bytes sequence from 9th byte to the end +of this size-prefixed blob, computed by [xxHash](https://github.com/Cyan4973/xxHash/) algorithm +(XXH64 version). The "message" is marked as user data (i. e. the user/meta data bit is set to 0). + +The self-bootstrapping header itself (i. e. the "message" of the size-prefixed blob) is encoded in +Text Wire format. Once created, this header is never changed. It contains all configurations, +immutable for Chronicle Map during the instance lifetime: number of segments, various sizes, +offsets, etc. See the specification of the fields on [Map Header Fields](3_1-header-fields.md) page. ## Global mutable state @@ -53,6 +64,9 @@ The global mutable state is 36 bytes long. 2. Bytes 8..11 - the number of allocated [extra tier bulks](#extra-tier-bulks). A non-negative 32-bit value, stored in the little-endian order. + + See also the [extra tier bulk allocation](5-initialization.md#entra-tier-bulk-allocation) + operation. 3. Bytes 12..19 - the index of the first *free* segment tier. A non-negative 64-bit value, stored in the little-endian order. Extra segment tiers are allocated in bulks, so there is usually a chain of allocated, but unused yet segment tiers. This field points to the head of this chain, or has diff --git a/spec/3_2-lock-structure.md b/spec/3_2-lock-structure.md index 3c9ab6d63..eec64312b 100644 --- a/spec/3_2-lock-structure.md +++ b/spec/3_2-lock-structure.md @@ -8,7 +8,7 @@ The lock structure is 8 bytes (64 bits) long. 3. Bit 31 - write lock flag 2. *Wait word*: bits 32..63 - wait count (little-endian) -## The *try acquire read lock* procedure +## Try acquire read lock 1. Read the lock state. 2. If the write lock flag is set, or the wait count is non-zero, or the read lock count is @@ -16,16 +16,40 @@ The lock structure is 8 bytes (64 bits) long. 3. Make a copy of the count word, with the read lock count incremented. 4. Perform a *compare-and-swap* (CAS) operation on the count word, comparing with the state that was read and swapping with the updated, made on the previous step. If the CAS operation succeeds, - the procedure succeeds. If the operations fails, the procedure fails. + the procedure succeeds. If the CAS operation fails, the procedure fails. -## The *try acquire update lock* procedure +## Release read lock + + 1. Read the lock state. + 2. If the read lock count is 0, then the procedure fails. + 3. Make a copy of the count word, with the read lock count decremented. + 4. Perform a CAS operation on the count word, comparing with the state that was read on the 1st + step and swapping with the updated, made on the previous step. If the CAS operation succeeds, + the procedure succeeds. If the CAS operation fails, continue this procedure, starting from the 1st + step + +> There is formally an infinite loop in this procedure, but if implemented correctly (and only +> such implementations alter the lock structure) it should either fail or succeed in some finite +> number of iterations. + +## Try acquire update lock 1. Read the lock state. 2. If the update or write lock flag is set or the wait count is non-zero, then the procedure fails. 3. Make a copy of the count word, with the update lock flag set. 4. Perform a CAS operation on the count word, comparing with the state that was read and swapping with the updated, made on the previous step. If the CAS operation succeeds, the procedure succeeds. - If the operations fails, the procedure fails. + If the CAS operation fails, the procedure fails. + +## Release update lock + + 1. Read the lock state. + 2. If the update lock flag is not set, then the procedure fails. + 3. Make a copy of the count word, with the update lock flag unset. + 4. Perform a CAS operation on the count word, comparing with the state that was read on the 1st + step and swapping with the updated, made on the previous step. If the CAS operation succeeds, the + procedure succeeds. If the CAS operation fails, continue this procedure, starting form the 1st + step. ## Time-limited read or update lock acquisition @@ -40,13 +64,16 @@ implementation may vary for the read and update locks as well. > The reference Java implementation doesn't insert any resources yielding directives after failed > try acquire attempts, for both read and update lock acquisition. -Implementations doesn't try to acquire the read or update lock indefinitely, after some finite +Implementations must not try to acquire the read or update lock indefinitely, after some finite number of failed attempts or some finite time elapsed since the time-limited lock acquisition -procedure starts, the procedure fails. +procedure starts, the procedure must fail. > Indefinite lock acquisition is dead lock prone. -## The *try acquire write lock* procedure +> The reference Java implementation makes attempts to acquire the lock for 1 minute, then throws +> an exception. + +## Try acquire write lock 1. (Optional step) Read the count word of the lock state. 2. (Optional step) If the count word is non-zero, the procedure fails. @@ -70,7 +97,17 @@ procedure and call one depending on the context. > The reference Java implementation uses only the version of this procedure without the first two > steps. -## The *try upgrade update to write lock* procedure + + + +## Release write lock, or write to update lock downgrade, or write to read lock downgrade + +Perform a CAS operation on the count word of the lock state, comparing 0x80000000 (i. e. a count +word with the write lock flag set) and swapping with 0 (in case of releasing write lock), or +0x40000000 (in case of write to update lock downgrade), or 1 (in case of write to read lock +downgrade). The result of the CAS operation is the result of the procedure. + +## Try upgrade update to write lock Perform a CAS operation on the count word of the lock state, comparing 0x40000000 (i. e. a count word with the update lock flag set, the write lock flag not set, the read lock count of zero) and @@ -78,7 +115,7 @@ swapping with 0x80000000, i. e. a count word with the write lock flag set, the u set, and the read lock count of zero. The result of the CAS operation is the result of the procedure. -### The *register wait* procedure +### Register wait 1. Read the wait word of the lock structure. 2. If the wait count is 231 − 1, the procedure fails: *wait count overflow*. @@ -86,7 +123,7 @@ procedure. read with a wait word with the wait count incremented. If the CAS operation fails, begin the register wait procedure from the start. If the CAS operation succeeds, the procedure succeeds. -### The *deregister wait* procedure +### Deregister wait 1. Read the wait word of the lock structure. 2. If the wait count is 0, the procedure fails: *wait count underflow*. @@ -94,14 +131,15 @@ procedure. read with a wait word with the wait count decremented. If the CAS operation fails, begin the deregister wait procedure from the start. If the CAS operation succeeds, the procedure succeeds. + + ## Time-limited write lock acquisition or update to write upgrade - 1. Perform the corresponding *try acquire* procedure ([write lock]( - #the-try-acquire-write-lock-procedure) or [upgrade update to write lock]( - #the-try-upgrade-update-to-write-lock-procedure)). If the attempt succeeds, this procedure - succeeds. If the attempt fails, continue with the following steps. - 2. [Register a wait](#the-register-wait-procedure). If the register wait procedure fails, this - time-limited procedure fails. + 1. Perform the corresponding *try acquire* procedure ([write lock](#try-acquire-write-lock) or + [upgrade update to write lock](#try-upgrade-update-to-write-lock)). If the attempt succeeds, this + procedure succeeds. If the attempt fails, continue with the following steps. + 2. [Register a wait](#register-wait). If the register wait procedure fails, this time-limited + procedure fails. 3. Read the lock state. 4. If the count word of the lock state is not equal to 0 (in case of acquiring the write lock) or 0x40000000 (in case of upgrading update to write lock), go to the step 7. @@ -111,9 +149,9 @@ procedure. flag set). If the CAS operation succeeds, the time-limited acquisition succeeds. If the CAS operation fails, continue with the step 7. 7. Check if this time-limited procedure has run out of the time limit or the number of attempts. - If it is, the time-limited acquisition fails, [deregister a wait](#the-deregister-wait-procedure) - before exiting from this procedure (the result of the deregister wait procedure doesn't matter). - Otherwise, continue from the step 3 of this procedure. + If it is, the time-limited acquisition fails, [deregister a wait](#deregister-wait) before exiting + from this procedure (the result of the deregister wait procedure doesn't matter). Otherwise, + continue from the step 3 of this procedure. Implementations doesn't make locking attempts indefinitely [for the same reasons, as in case of acquiring read and update lock](#time-limited-read-or-update-lock-acquisition). @@ -121,7 +159,7 @@ acquiring read and update lock](#time-limited-read-or-update-lock-acquisition). > ## The reference Java implementation > > Attempt operations: [`VanillaReadWriteUpdateWithWaitsLockingStrategy`]( -> https://github.com/OpenHFT/Chronicle-Algorithms/blob/chronicle-algorithms-1.1.6/src/main/java/net/openhft/chronicle/algo/locks/VanillaReadWriteUpdateWithWaitsLockingStrategy.java) +> https://github.com/OpenHFT/Chronicle-Algorithms/blob/master/src/main/java/net/openhft/chronicle/algo/locks/VanillaReadWriteUpdateWithWaitsLockingStrategy.java) > > Time-limited operations: [`BigSegmentHeader`]( > ../src/main/java/net/openhft/chronicle/hash/impl/BigSegmentHeader.java) diff --git a/spec/4-hashing-algorithms.md b/spec/4-hashing-algorithms.md index 4572ff7d4..70d3f9a5f 100644 --- a/spec/4-hashing-algorithms.md +++ b/spec/4-hashing-algorithms.md @@ -1,16 +1,16 @@ -# The Chronicle Map Key Hashing and Checksum Algorithms +# 4. The Chronicle Map Key Hashing and Checksum Algorithms Keys in Chronicle Map are arbitrary sequences of bytes. [xxHash -](https://github.com/Cyan4973/xxHash/) algorithm is applied to the key, to obtain the primary key -hash code. Then the [`hashSplitting`](3_1-header-fields.md#hashsplitting) algorithm is applied, to -determine the segment in which the key should be stored, and the part of the key hash code to be -stored in a segment tier's hash lookup. +](https://github.com/Cyan4973/xxHash/) algorithm (XXH64 version) is applied to the key, to obtain +the primary key hash code. Then the [`hashSplitting`](3_1-header-fields.md#hashsplitting) algorithm +is applied, to determine the segment in which the key should be stored, and the part of the key hash +code to be stored in a segment tier's hash lookup. -> The reference Java implementation: [XxHash_r39 -> ](https://github.com/OpenHFT/Chronicle-Algorithms/blob/chronicle-algorithms-1.1.6/src/main/java/net/openhft/chronicle/algo/hashing/XxHash_r39.java). +> The reference Java implementation: [`XxHash_r39`]( +> https://github.com/OpenHFT/Chronicle-Algorithms/blob/chronicle-algorithms-1.1.6/src/main/java/net/openhft/chronicle/algo/hashing/XxHash_r39.java). > Although the Java implementation class has `_r39` suffix, the xxHash algorithm is stable since r3 -> and [won't change in the future -> ](https://github.com/Cyan4973/xxHash/issues/34#issuecomment-169176338). A different version of +> and [won't change in the future]( +> https://github.com/Cyan4973/xxHash/issues/34#issuecomment-169176338). A different version of > the algorithm could have a different name. ## Checksum algorithm @@ -24,10 +24,10 @@ If the 2nd field of the [stored entry structure structure starts, i. e. the value size is 0, and the size itself is stored using 0 bytes, and there is no value alignment, the key hash code *is* the primary checksum. -Otherwise, the [xxHash](https://github.com/Cyan4973/xxHash/) algorithm is applied to the memory -range between the end of the 2nd field of the stored entry structure and the end of the 5th field, -i. e. between the end of the stored key and the end of the stored value. The resulting hash value -is called *payload checksum*. +Otherwise, the [xxHash](https://github.com/Cyan4973/xxHash/) algorithm (XXH64 version) is applied to +the memory range between the end of the 2nd field of the stored entry structure and the end of the +5th field, i. e. between the end of the stored key and the end of the stored value. The resulting +hash value is called *payload checksum*. > xxHash is used to compute the payload checksum instead of CRC32, because the Java implementation > of CRC32 appears to be slower than xxHash while having the same (if not worse) quality, and the @@ -48,11 +48,18 @@ where the `keySize` is the key sequence length in bytes, `keyHashCode` is the ke the `payloadChecksum` is the payload checksum, computed on the previous step. The given procedure code is in [Java language](https://docs.oracle.com/javase/specs/jls/se8/html/index.html). +> The above procedure is actually the [CityHash](https://github.com/google/cityhash) and [FarmHash]( +> https://github.com/google/farmhash) algorithm for bytes sequences of lengths between 8 and 16 ( +> CityHash and FarmHash are equivalent for such input lengths), where `keySize` is the length of the +> input sequence, `keyHashCode` value is the starting 8 bytes of the sequence, `payloadChecksum` is +> the ending 8 bytes of the sequence. This procedure presumably has good hashing properties, however +> it was not checked for the changed semantics of the three parameters. + ### Entry checksum Entry checksum is 4 bytes long, it is obtained by XOR-ing lowest and highest 32 bits of the [primary -checksum[#primary-checksum]. The entry checksum is stored in the 6th field of the [stored entry +checksum](#primary-checksum). The entry checksum is stored in the 6th field of the [stored entry structure](3-memory-layout.md#stored-entry-structure). -> The reference Java implementation: [`HashEntryChecksumStrategy` -> ](../src/main/java/net/openhft/chronicle/hash/impl/stage/entry/HashEntryChecksumStrategy.java). \ No newline at end of file +> The reference Java implementation: [`HashEntryChecksumStrategy`]( +> ../src/main/java/net/openhft/chronicle/hash/impl/stage/entry/HashEntryChecksumStrategy.java). \ No newline at end of file diff --git a/spec/5-initialization.md b/spec/5-initialization.md new file mode 100644 index 000000000..120464c1e --- /dev/null +++ b/spec/5-initialization.md @@ -0,0 +1,150 @@ +# 5. Chronicle Map Data Store Initialization Operations + +> In the reference Java implementation the operations, described on this page, are defined in +> [`ChronicleMapBuilder`](../src/main/java/net/openhft/chronicle/map/ChronicleMapBuilder.java) and +> [`VanillaChronicleHash`]( +> ..\src\main\java\net\openhft\chronicle\hash\impl\VanillaChronicleHash.java) classes. + +## Chronicle Map creation + +This section describes the process of creation of a new Chronicle Map instance (or opening the +existing Map, persisted to some file). + + 1. If the Chronicle Map should be persisted to some file, and this file does not exist, this file + is created as empty, i. e. with size of 0 bytes. + + 2. If the Chronicle Map should be persisted to some file, and the size of the file is non-zero, + [wait until the Chronicle Map is ready](#wait-until-chronicle-map-is-ready). Otherwise, acquire + an exclusive file lock, and check the file size again. If the file size is non-zero now, release + the file lock and [wait until the Chronicle Map is ready](#wait-until-chronicle-map-is-ready). If + the file is still empty, [write self bootstrapping header](#write-self-bootstrapping-header) to the + beginning of the file, then release the file lock. + + > The purpose of acquiring file lock is to ensure that only one concurrent process (if any) + > initializes the Chronicle Map data store, others wait until they could open the already created + > Chronicle Map. This also excludes possibility for any issues with conflicting Chronicle Map + > configurations, attempted to be created from different processes. + + > This step follows [double-checked locking](https://en.wikipedia.org/wiki/Double-checked_locking) + > pattern to avoid file locking in most cases when an already existing Chronicle Map is opened. + + This step is serialized for the persistence file among threads within the current process, because + file locking mechanism in operating systems is per-process, not per-thread. I. e. concurrent + threads within the same process don't lock the same file at the same time. + + Concurrent threads perform any initialization operations (Chronicle Map creation and [extra tier + bulk allocation](#extra-tier-bulk-allocation)) with canonicalized file descriptor, that is closed + only after last Chronicle Map in-process representation structure, referencing this Chronicle Map + data store (hence this file descriptor) is disposed. This is required, because in Linux, when any + file descriptor referencing a file is closed, all file locks are released unconditionally. + + If on this step we were waiting until Chronicle Map is ready, then start using Chronicle Map + normally. If on this step we wrote self-bootstrapping header or the Chronicle Map is not persisted + to any file (i. e. purely in-memory), continue initialization with following steps: + + 3. Zero out the [global mutable state](3-memory-layout.md#global-mutable-state)'s space in the + file. + 4. Zero out the [segment headers area](3-memory-layout.md#segment-headers-area) in the file. + 5. For each segment tier in the [main segments area](3-memory-layout.md#main-segments-area), zero + out memory from the beginning of the tier to the start of the [entry space]( + 3-memory-layout.md#entry-space), i. e. zero out this tier's hash lookup, segment tier counters area + and free list. + 6. White the segment headers offset into the 5th field of the global mutable state. + 7. If the Chronicle Map is persisted, ensure all data written to the file is flushed to the disk. + For example on Linux, this could be done with `msync` and `fdatasync` calls, on Windows - with + `FlushFileBuffers` system call. + 8. If the Chronicle Map is persisted, write the readiness bit into the header, i. e. the highest + bit of the 32-bit word by offset 8 from the beginning of the file, read and written in the + little-endian order. Read the [Size Prefix Blob]( + https://github.com/OpenHFT/RFC/blob/master/Size-Prefixed-Blob/Size-Prefixed-Blob-0.1.md) + specification for details. This step is the event threads and processes [waiting until Chronicle + Map is ready](#wait-until-chronicle-map-is-ready) actually wait for. + +### Wait until Chronicle Map is ready + +Check until the readiness bit is set to 0. The readiness bit it the highest bit of 32-bit word by +offset 8 from the beginning of the Chronicle Map persistence file, read in the little-endian order. + +Optionally yield processor resources after failed checks, e. g. make the waiting thread sleep for +some time. The exact yielding operation (or if it is making the current process sleeping, the exact +sleep interval) is unspecified. + +> The reference Java implementations turns the waiting thread to sleep for 100 ms after failed +> attempts. + +Implementations must not wait until the Chronicle Map is ready indefinitely, after some finite +number of failed checks, or some finite time elapsed since the start, the wait procedure must fail. + +> Indefinite wait is dead lock prone, if the concurrent process or thread doing the initialization +> fails itself. + +> The reference Java implementation checks until the Chronicle Map is ready for approximately 1 +> minute, then throws an exception. + +### Write self bootstrapping header + + 1. Determine the self-bootstrapping header length (in bytes). + 2. Write the correct hash code (64-bit value) at the beginning 8 bytes of the file, in the + little-endian order. The "correct hash code" is the result of application of xxHash (XXH64 version) + algorithm to the bytes sequence, first 4 bytes of which is the self-bootstrapping header length, + determined on the step 1, encoded in the little-endian order, the rest bytes of the sequence + is the self-bootstrapping header itself. + 3. Write the length (determined on the step 1) to the size word (by offset 8 from the beginning of + the file), with the readiness bit set to 1. + 4. Write the self-bootstrapping header to the file itself, starting from offset 12 in the file. + +See also [memory layout of self-bootstrapping header](3-memory-layout.md#self-bootstrapping-header). + +## Extra tier bulk allocation + +This procedure is called only within [tier allocation](#tier-allocation) procedure, hence guarded +by [global mutable state](3-memory-layout.md#global-mutable-state) lock. + + 1. If the Chronicle Map is persisted to disk, extend the file's mapping from the current end of the + Chronicle Map data store (either the end of the [main segments area]( + 3-memory-layout.md#main-segments-area) or the end of the previous extra tier bulk) by the minimum + memory amount allowed for mapping (a multiple of the native page size), that covers space required + for the next extra tier bulk, i. e. [`tierBulkSizeInBytes`]( + 3_1-header-fields.md#tierbulksizeinbytes) bytes from the previous end of the Chronicle Map data + store. + 2. Prepare the extra tier bulk's metadata space, that starts at the beginning of the extra tier + bulk and spans [`tierBulkInnerOffsetToTiers`](3_1-header-fields.md#tierbulkinneroffsettotiers), + according to the way how metadata space is actually used. This is unspecified. + 3. For each tier in the extra tier bulk, zero out memory from the beginning of the tier to the + start of the [entry space](3-memory-layout.md#entry-space), i. e. zero out this tier's hash lookup, + segment tier counters area and free list. This step is equivalent to the 5th step of the [Chronicle + Map creation](#chronicle-map-creation) procedure. + 4. For each (but the last one) tier in the extra tier bulk, write [index of the next tier]( + 3-memory-layout.md#tier-index) into the 1st field of the [tier counters area]( + 3-memory-layout.md#segment-tier-counters-area). This step forms a singly-linked list of allocated, + but unused yet segment tiers. + 5. If the Chronicle Map is persisted, ensure that memory of the newly allocated extra tier bulk + is flushed to the disk (via `msync` call on Linux, `FlushFileBuffers` on Windows). + 6. Increment the value in the 2nd field (the number of allocated extra tier bulks) of the [global + mutable state](3-memory-layout.md#global-mutable-state) of this Chronicle Map. + 7. Write the [index](3-memory-layout.md#tier-index) of the first tier in this extra tier bulk into + the 3rd field (the index of the first *free* segment tier) of this Chronicle Map's global mutable + state. + +## Tier allocation + + 1. [Acquire write lock](3_2-lock-structure.md#time-limited-write-lock-acquisition) on the [global + mutable state](3-memory-layout.md#global-mutable-state) lock (the 1st field). If acquisition is + not successful, this operation fails. + 2. Check that the value in the 4th field (the number of used extra segment tiers) of the global + mutable state is less than the [`maxExtraTiers`](3_1-header-fields.md#maxextratiers) value, + otherwise the operation fails. + 3. Read the value in the 3rd field (the index of the first free segment tier) of the global mutable + state. If the value is not positive, [allocate an extra tier bulk](#extra-tier-bulk-allocation), + and then read the value in the 3rd field of the global mutable state again. If it not positive, + the operation fails. + 4. Read the value of the 1st field (index of the next linked tier) of the tier counter area in the + tier with index, read on the previous step. + 5. Increment the value in the 4th field (the number of used extra segment tiers) of the global + mutable state. + 6. Write the tier index, read on the 4th step of this operation, into the 3rd field (the index of + the first free segment tier) of the global mutable state. + 7. [Release write lock](3_2-lock-structure.md#release-write-lock) on the global mutable state lock. + If the release is not successful, this operation fails. + +The result of the procedure is the tier index, read on the 3rd step. \ No newline at end of file diff --git a/spec/README.md b/spec/README.md index 841358bba..7c72d00ca 100644 --- a/spec/README.md +++ b/spec/README.md @@ -8,6 +8,7 @@ This is currently work-in-progress. 1. [Header fields](3_1-header-fields.md) 2. [Lock structure and locking operations](3_2-lock-structure.md) 4. [Key hashing and checksum algorithms](4-hashing-algorithms.md) + 5. [Initialization operations](5-initialization.md) ## Rational behind this Specification