diff --git a/README.md b/README.md index 0c880d646..74cd01efc 100644 --- a/README.md +++ b/README.md @@ -34,8 +34,8 @@ applications. Notably trading, financial market applications. - Replication *without logs*, with constant footprint cost, guarantees progress even if the network doesn't sustain write rates. -Chronicle Map has two meanings: [the language-agnostic data structure](spec) and [the -implementation of this data structure for the JVM](src). Currently, this is the only implementation. +Chronicle Map has two meanings: [the language-agnostic data store](spec) and [the +implementation of this data store for the JVM](src). Currently, this is the only implementation. **From Java perspective,** Chronicle Map is a `ConcurrentMap` implementation which stores the entries *off-heap*, serializing/deserializing key and value objects to/from off-heap memory @@ -68,11 +68,11 @@ queries*. Chronicle Map project, there are projects *on top* of Chronicle Map which address these questions, e. g. [Chronicle Enterprise](http://chronicle.software/products/chronicle-enterprise/). -**What is the Chronicle Map data structure?** In one sentence and simplified, Chronicle Map instance -is a big chunk of shared memory (optionally mapped to disk), split into independent segments, each -segment has an independent memory allocation for storing the entries, a hash table for search, and a -lock in shared memory (implemented via CAS loops) for managing concurrent access. Read [the -Chronicle Map design overview](spec/2-design-overview.md) for more. +**What is the Chronicle Map's data structure?** In one sentence and simplified, a Chronicle Map +instance is a big chunk of shared memory (optionally mapped to disk), split into independent +segments, each segment has an independent memory allocation for storing the entries, a hash table +for search, and a lock in shared memory (implemented via CAS loops) for managing concurrent access. +Read [the Chronicle Map data store design overview](spec/2-design-overview.md) for more. ### Chronicle Map is *not* @@ -1058,7 +1058,7 @@ Combined, interception SPI interfaces and `ChronicleMap.queryContext()` API are - Perform multi-key operations on a single `ChronicleMap` correctly in concurrent environment, by acquiring locks on all keys before updating the entries - Define own replication/reconciliation logic for distributed Chronicle Maps - - Dump statistics of a Chronicle Map data structure -- each segment's load, size in bytes of each + - Dump statistics of the Chronicle Map instance -- each segment's load, size in bytes of each entry, etc. #### Example - Simple logging diff --git a/docs/compatibility.md b/docs/compatibility.md index eabda15ae..eafbc4530 100644 --- a/docs/compatibility.md +++ b/docs/compatibility.md @@ -1,6 +1,6 @@ # Chronicle Map 3 Compatibility -## Java implementation compatibility +## The Java Implementation Compatibility ### Public API @@ -26,7 +26,7 @@ of some types could change, "high level" configurations in `ChronicleMapBuilder` Chronicle Map instances with different properties, etc. Such changes should be documented in release notes. -### Persistence file binary form +### Persistence File Binary Form A persisted Chronicle Map instance (a binary file), produced by a production release of the library, should be accessible and updatable by any subsequent production release of the library in 3.x @@ -41,19 +41,19 @@ To support this, new version of root implementation class, e. g. `VanillaChronicleMap_3_5` should be created and put along with older version, which should keep behave the same as before. -### Replication protocol +### Replication Protocol The replication protocol might change between Chronicle Map versions with different "minor" number. Chronicle Map nodes running different versions of the library shouldn't communicate to each other. Though this might lead to failures, but not to data corruption on either of the nodes. -## Specification compatibility +## The Specification Compatibility -Chronicle Map Data Structure Specification could be updated without revision change while it remains -backward compatible, i. e. implementation of the newer version of specification could access and -update any persisted Chronicle Map instance, produced by older version. In addition, potential -*concurrent* accessing implementations of older versions ("within" the same revision) shouldn't lose -correctness and corrupt the data. +The Chronicle Map Data Store Specification could be updated without changing the revision while it +remains backward compatible, i. e. the implementation of the newer version of the specification +could access and update any persisted Chronicle Map instance, produced by an older version. +In addition, potential *concurrent* accessing implementations of older versions ("within" the same +revision) shouldn't lose correctness and corrupt the data. If more substantial changes to the specification are required, the revision number should be updated. diff --git a/docs/versioning.md b/docs/versioning.md index 0766725e8..a51662797 100644 --- a/docs/versioning.md +++ b/docs/versioning.md @@ -1,6 +1,6 @@ # Chronicle Map 3 Versioning -## Java implementation versioning +## The Java implementation Versioning 3.minor.patch @@ -14,10 +14,10 @@ number of segments, or segment's capacity, etc. - Any dependency is updated, even if only "patch" number is updated, e. g. `chronicle-bytes` dependency is updated from `1.1.1` to `1.1.2`. - - Bug fixing, performance optimization or feature addition requires to change binary or serialized - form of the Chronicle Map data structure (therefore revise the Chronicle Map Specification, see - below), or binary replication protocol, or serialization form of some marshaller classes, that - requires to [create new versions of classes](compatibility.md). + - Bug fixing, performance optimization or feature addition requires to change the binary or + serialized form of Chronicle Map data store (therefore revise the Chronicle Map data store + specification, see below), or the binary replication protocol, or the serialized form of some + marshaller classes, that requires to [create new versions of classes](compatibility.md). "Minor" number shouldn't be updated, i. e. the next release could update only the "patch" number, when @@ -33,16 +33,18 @@ commit* these changes are made. Suffixes `-alpha`, `-beta`, `-rc` etc. mean this is a pre-production release. -## Specification versioning +## The Specification Versioning -The first Chronicle Map 3 production release designates the Chronicle Map Specification revision 1. +The first Chronicle Map 3 production release designates the Chronicle Map data store specification, +revision 1. When the specification should be updated in a way that is not [backward compatible]( -compatibility.md#specification-compatibility), the revision should be updated (revision 2, 3, ...) -and the Java implementation of this revision should have "minor" number updated in it's version. +compatibility.md#the-specification-compatibility), the revision should be updated (revision 2, 3, +...) and the Java implementation of this revision should have "minor" number updated in it's +version. -During development after the release of the revision `x`, while the specification is backward -compatible, it should remain versioned `x`. In the same commit where specification becomes +During the development after the release of the revision `x`, while the specification is backward +compatible, it should remain versioned `x`. In the same commit where the specification becomes incompatible, it's revision should be set to `-pre` until the next production release. Releases and updates to the "minor" number in the library version should be synchronized with diff --git a/spec/0-purposes.md b/spec/0-purposes.md index ecdc0efd4..76e16f541 100644 --- a/spec/0-purposes.md +++ b/spec/0-purposes.md @@ -1,4 +1,4 @@ -# Rational behind Chronicle Map Specification +# Rational behind The Chronicle Map Data Store Specification - To provide the ability to write a small, drastically simplified Chronicle Map implementation in Java for a specific use-case. It could hard-code many aspects which the reference implementation has diff --git a/spec/1-design-goals.md b/spec/1-design-goals.md index 0a026d12a..911d3b51e 100644 --- a/spec/1-design-goals.md +++ b/spec/1-design-goals.md @@ -1,104 +1,111 @@ -# Chronicle Map Design Goals, Assumptions and Guarantees +# Chronicle Map Data Store Design Goals, Assumptions and Guarantees - Chronicle Map is an in-memory, highly concurrent key-value store. Generally keys and values are arbitrary sequences of bytes. - Chronicle Map optionally persists to a *single file* via the memory-mapping facility, present in Windows and POSIX-compatible operating systems (`mmap`). The file shouldn't necessarily materialize - on disk. It's OK for Chronicle Map if the file resides a memory-mounted FS. The whole Chronicle Map - state is contained in the file contents i. e. it is possible to move, copy or send the file to - another machine, access it and observe exactly the same Chronicle Map state. Chronicle Map doesn't - use the file metadata. + on disk. It's OK for Chronicle Map if the file resides a memory-mounted file system. The whole + Chronicle Map state is contained in the file contents i. e. it is possible to move, copy or send + the file to another machine, access it and observe exactly the same Chronicle Map state. Chronicle + Map doesn't use the metadata of the file. - Chronicle Map supports fully-featured concurrent access from multiple processes, mapping the same file. - - Chronicle Map supports *isolated* multi-key operations, either on the single key-value domain - (a single Chronicle Map) or several ones. But the multi-key updates to persisted Chronicle Map(s) - are *not* atomic (basically isolated and atomic in ACID sense here; see the detailed definition - below in [Guarantees](#Guarantees) section). - + - Chronicle Map supports *isolated* multi-key operations, either on a single key-value domain + (a single Chronicle Map instance) or several ones. But multi-key updates to persisted Chronicle + Map(s) are *not* atomic (basically, *isolated* and *atomic* terms used in the sense of + [ACID](https://en.wikipedia.org/wiki/ACID) here; see the detailed definition below + in [Guarantees](#guarantees-1) section). - Chronicle Map runs only on little-endian architectures. - - Chronicle Map is *not* distributed: the whole data structure is placed in memory of the single - machine (in a single file, if persisted). + - Chronicle Map is *not* distributed: the whole data store resides the memory of a single machine + (a single file, if persisted). ## Guarantees ### Assumptions +Chronicle Map provides some guarantees under the following assumptions: + #### CPU - - CPU should support atomic 32-bit compare-and-swap operations with aligned memory, i. e. if - several threads, possibly belonging to different processes, try to perform compare-and-swap - operation on the same 4-byte block, one thread succeeds and all the rest fail. - - Aligned 32-bit or 64-bit writes should be atomic, that means either all 4(8) bytes are written to - memory, or nothing is written. + - CPU supports atomic 32-bit compare-and-swap operations with aligned memory, i. e. if several + threads, possibly belonging to different processes, try to perform compare-and-swap operation on + the same 4-byte block, at most one thread will succeed and all the rest will fail. + - Aligned 32-bit or 64-bit writes are atomic, that means either all 4(8) bytes are written to + the memory, or nothing is written (in case of power outage, unconditional thread interruption, + etc.). The two above points are true for CPUs with x86 and x86_64 architectures. - If some values Xi are written at addresses Ai, then 32- or 64-bit value Y - is written at address B, there should be a way to ensure some memory order. In particular, if a - concurrent thread reads value Y from address B, then it should read Xii addresses (but not the old values, stored at these addresses). + is written at address B, there is a way to ensure some memory order. In particular, if a + concurrent thread reads value Y from address B, after that it will read Xii addresses (but not the old values, stored at those addresses). - > In Java Memory model terms, this assumption means "writes to Ai addresses + > In terms of the Java Memory model, this assumption means "writes to Ai addresses > *happens-before* the write of Y value at address B, which *synchronizes-with* read of this value - > from a concurrent thread". E. g. in Java there are some trivial ways to ensure this memory - > ordering: volatile write or compare-and-swap at address B with volatile reads of value Y - > from B in a concurrent thread. + > from a concurrent thread". I. e. in Java there are some trivial ways to ensure such memory + > ordering, for example, volatile write or compare-and-swap at address B with volatile read of + > value Y at address B from a concurrent thread. #### Memory-mapping - - The memory-mapping implementation shouldn't corrupt the mapped file, even if operating system + - The memory-mapping implementation doesn't corrupt mapped files, even if operating system execution was interrupted in any way (black out, virtual machine crash, etc.) It means Chronicle - Map expects some memory might be "stale", i. e. the bits written shortly before operating system - failure might not be persisted, but it doesn't expect to read the bit values that have never been - written to the memory. - - Writes to disk should be at least 4 or 8 bytes atomic, i. e. either aligned 4- or 8-byte blocks - are fully written to the disk, or not written at all. + Map expects some memory might be "stale", i. e. the values written shortly before operating system + failure might not be persisted, but it doesn't expect to read values that have never been written + to the memory. + - Writes to the disk are at least 4 or 8 bytes atomic, i. e. either aligned 4- or 8-byte blocks + are fully written to the disk, or not written at all (in case of power loss, virtual machine crash, + etc.). > Modern non-volatile drives (HDDs and SSDs) support atomic sector writes, which are usually at > least 512 bytes in size. -On the other hand, it doesn't assume anything specific about how memory if flushed to disk: +Chronicle Map *doesn't* assume anything specific about how memory if flushed to the disk: - - The order in which memory is flushed (up or down by addresses, random, concurrent, etc.) - - The maximum timeout dirty memory could not be flushed to disk. i.e. it assumes some memory could - remain dirty forever. - - The maximum fraction of dirty memory not written to disk i.e. it assumes that all mapped memory - could be dirty. + - The order in which memory is flushed (up or down by addresses, random, concurrent, etc.). + - The maximum timeout dirty memory might not be flushed to the disk, i. e. Chronicle Map assumes + some memory might remain dirty forever. + - The maximum fraction of dirty memory not written to the disk, i. e. Chronicle Map assumes that + *all* the mapped memory might be dirty. ### Guarantees -If the above assumptions are met, Chronicle Map aims to satisfy the following guarantee: +If the above assumptions are met, Chronicle Map aims to satisfy the following guarantees: - - Single-key and multi-key accesses and updates to Chronicle Map (multi-key updates could span - several Chronicle Maps) are concurrently isolated, i. e. accesses involving a certain key are - totally ordered across accessing threads and processes. All updates made to the entry during the - previous update are visible during the subsequent access. After a multi-key update, a subsequent - multi-key query involving a subset of the updated keys (and possibly some more keys) views all - updates made to all entries corresponding to the keys from that subset. + - Single-key and multi-key accesses and updates to the Chronicle Map (multi-key updates could span + several Chronicle Map instances) are concurrently isolated, i. e. accesses involving a certain key + are totally ordered across accessing threads and processes. All updates made to the entry during + the previous update are visible during the subsequent accesses. After a multi-key update, + a subsequent multi-key query involving a subset of the updated keys (and possibly some more keys) + observes all updates made to all the entries corresponding to the keys from that subset. - Under any conditions (concurrent access, inter-process access) and using any type of access (querying the entry by the key or iteration), only entries that were ever stored in the Chronicle - Map could be observed. Reading corrupted or half written values by some keys, or observing some - keys which were never stored during the iteration is disallowed. + Map instance are observed. Reading corrupted or half written values by some keys, or observing some + keys that were never stored during the iteration is impossible. -If Chronicle Map is persisted to a file and the operating system fails to flush all dirty memory to -disk due to a power-off or any other failure, when the file is mapped to memory and accessed again, -it might be needed to perform a special recovery procedure on the store first, which identifies and -purges corrupted entries from the Chronicle Map. Therefore, *some entries updated shortly before the -failure could be lost.* +If the Chronicle Map instance is persisted to the file and the operating system fails to flush all +the dirty memory to the disk due to a power-off or any other failure, when the file is mapped to +the memory and accessed again, it is required to perform a *special recovery procedure* on the +Chronicle Map first, which identifies and purges corrupted entries from the Chronicle Map. +Therefore, *some entries updated shortly before the failure could be lost.* + + > The aforementioned *special recovery procedure* is `ChronicleHashBuilder.recoverPersistedTo()` + > or `ChronicleHashInstanceBuilder.recover()` in the reference Java implementation. ## Goals The ultimate goal of Chronicle Map design is efficiency: - If the number of entries in Chronicle Map is much greater than the number of accessor hardware - threads, it should scale well up to the total number of hardware threads present in the system. - - Chronicle Map should be a low-latency key-value store, meaning that for any particular - percentile (90%, 99%, 99.9% etc.) the latency of Chronicle Map operations should be the - best or among the best compared to latencies of other key-value stores filled with the same data - and providing similar guarantees. - - Chronicle Map shouldn't severely overuse memory and disk space for any range and distribution of - key and values sizes. Chronicle Map exhibits best-in-class memory footprint if keys and values are - small and fixed-sized. + threads, it scales well up to the total number of hardware threads present in the system. + - Chronicle Map is a low-latency key-value store, meaning that for any particular percentile (90%, + 99%, 99.9% etc.) the latency of Chronicle Map operations should be the best or among the best + compared to latencies of other key-value stores filled with the same data and providing the same or + stronger guarantees. + - Chronicle Map memory and disk space overuse rate is limited by 25% for any range and distribution + of key and values lengths (if configured properly). Chronicle Map exhibits the least memory + footprint among key-value stores, if keys and values have constant lengths. ### Assumptions @@ -106,27 +113,26 @@ Chronicle Map targets the goals outlined above under the following assumptions: - It runs on AMD64 (aka x86_64) architecture - It runs on Linux - - The total volume of memory, used by Chronicle Map fits the available physical memory. In other - words, the memory accessed by Chronicle Map should never be evicted to disk. + - The total volume of memory, used by the Chronicle Map fits the available physical memory. + In other words, the memory accessed by the Chronicle Map is never evicted to the disk. -If these assumptions are not held, Chronicle Map must still operate correctly and providing the same -guarantees, but it shouldn't be expected to be very efficient. +If these assumptions are not held, Chronicle Map will still operate correctly and provide the same +guarantees, but it shouldn't be expected to be similarly efficient. ## Non-goals - - Alphabetical key ordering (sorted keys). + - Alphabetical ordering of the keys (sorted keys). - Durability. There are ways to make Chronicle Map look like a durable key-value store, e. g. - performing `msync` after each operation. However, if required, it would be better use a database + to call `msync` after each operation. However, if required, it would be better to use a data store designed for durability. -The following items are non-goals but might become goals in the future, at least under certain -conditions or with additional assumptions: +The following items are currently non-goals, but might become goals in the future, at least under +certain conditions or assumptions: - Entry ordering by insertion, update or access times. - - Hard latency guarantees. e.g. Chronicle Map doesn't specify "all operations complete in less than - one second." Persistence via memory-mapping implies that any memory write could trigger flushing - the whole file to disk, that could take long time. - - Entry expirations. This could be emulated by storing the last update (access) time in the value - bytes, checking it on each access to Chronicle Map and, for example, running a background thread - that traverses the Chronicle Map and cleans expired entries. However it is not guaranteed that - Chronicle Map would perform better than a cache with expirations implemented as core functionality. + - Hard latency guarantees. Currently Chronicle Map doesn't specify "all operations complete in less + than one second." Persistence via memory-mapping implies that any memory write could trigger + flushing the whole file to the disk, that could take a long time. + - Entry expirations. Currently entry expirations could be emulated by storing the last update + (access) time in the value bytes, checking it on each access to the Chronicle Map and, for example, + running a background thread that traverses the Chronicle Map and cleans expired entries. diff --git a/spec/2-design-overview.md b/spec/2-design-overview.md index 5ac16076c..740399fec 100644 --- a/spec/2-design-overview.md +++ b/spec/2-design-overview.md @@ -1,31 +1,34 @@ -# Chronicle Map Design Overview +# Chronicle Map Data Store Design Overview ## Logic -**Chronicle Map is split into N *completely independent*, ordered segments**. The number of segments -is chosen during Chronicle Map creation and is never changed. Each segment has an associated 3-level -lock (read, *update* and write levels). Typical steps of performing a query to a Chronicle Map: +**Each Chronicle Map is split into N *completely independent*, ordered segments**. +The number of segments is chosen during the Chronicle Map creation and is never changed. Each +segment has an associated 3-level lock (the *read*, *update* and *write* levels). Typical steps of +performing a query to a Chronicle Map: - 1. Compute the hash code of the queried key. Hash code size is 64 bits. - 2. Choose the segment that should hold the key, by the hash code - 3. Acquire the segment lock on the needed level on - 4. Search for the entry with the queried key in the segment - 5. Perform the query operation on the entry (read the value, update the value, etc.) - 6. Release the segment lock + 1. Compute the hash code of the queried key. Hash code length is 64 bits. + 2. Identify the segment that should hold the key, based on the hash code. + 3. Acquire the segment lock on the needed level. + 4. Search for the entry with the queried key in the segment. + 5. Perform the actual query operation on the entry, if the entry is found (e. g. read the value, + update the value, etc.), or insert the entry, if the queried key was absent in the segment, and + insertion of a previously absent entry is implied by the logic of the query being performed. + 6. Release the segment lock. For multi-key queries, - 1. All segments in which the involved keys should be held are identified - 2. Acquire the locks of all involved segments. Within each involved Chronicle Map, segment locks - are acquired in the order of their segments. + 1. Identify all the segments in which the involved keys should be held. + 2. Acquire the locks of all the involved segments. Within each involved Chronicle Map instance, + acquire the segment locks in the order of their segments. - > Acquiring segment locks in the predictable order is needed to avoid dead-locks, e. g. when the - > first thread acquires lock of the segment #1 and then tries to acquire the lock of the + > Acquiring segment locks always in the same order is needed to avoid dead-locks, e. g. when the + > first thread acquires the lock of the segment #1 and then tries to acquire the lock of the > segment #2, and the second thread does the opposite: locks the segment #2, then tries to lock the > segment #1. - 3. Perform the query operations - 4. Release the locks of all involved segment in reverse order + 3. Perform the query operations. + 4. Release the locks of all the involved segments in the reverse order. Bulk operations (iterations, background tasks) visit each segment once in any order. For each segment, they acquire the segment lock, then visit each entry stored in the segment, then release @@ -33,43 +36,49 @@ the segment lock. ### Lock levels -**Read level** is shared, several threads and processes could hold the segment lock on read level at -the same time. This level is used for read queries on a single or multiple keys, like simple value -reading by the key. This level couldn't be upgraded to update or write level. +The **read level** is shared, several threads and processes could hold the segment lock on the read +level at the same time. This level is used for read queries on a single or multiple keys, like +simple value reading by the key. This level couldn't be upgraded to the update or write level. -> Read level couldn't upgrade to more exclusive levels, because this is dead-lock prone. Two threads -> could acquire the same lock on read level, then both try upgrade to write level - and blocked by -> each other. (Upgrading to write level through update level wouldn't eliminate this issue too.) +> The read level couldn't upgrade to the higher levels, because this is dead-lock prone. Two threads +> could acquire the same lock on the read level, then both try to upgrade to the write level - and +> blocked by each other. (Upgrading to the write level through the update level doesn't resolve +> this issue, too.) -**Update level** is partially exclusive: a thread could hold the segment lock on update level while -some other threads or processes are holding the same lock on read level, but not update or write -level. This level is used for "typically read" queries, that occasionally should be able to write -hence upgrade to write level without releasing the lock, for example reading value by the key, if -the entry is present or storing a new value, if the entry is absent. Also, bulk operations should -acquire segment locks at least on update level, even if they never perform writes. Lock on update -level could be upgraded to write level. +The **update level** is partially exclusive: a thread could hold the segment lock on the update +level while some other threads or processes are holding the same lock on the read level, but not +the update or write level. -**Write level** is fully exclusive: when a thread holds the segment lock on write level, no other -thread or process could hold the same lock on any level. Write level is used for write queries. +This level is used for "typically read" types of queries, which occasionally should be able to write +to the data store, hence need to upgrade to the write level without releasing the lock, for example, +reading value by the key, if the entry is already present in the Chronicle Map, or storing a new +value, if the entry is absent. Also, bulk operations acquire segment locks at least on the update +level, even if they never perform writes. + +A segment lock acquired on the update level could be upgraded to the write level. + +The **write level** is totally exclusive: when a thread holds the segment lock on the write level, +no other thread or process could hold the same lock on any level. The write level is used for write +queries. ## Lock implementation -Lock is represented by two 32-bit words in Chronicle Map memory. *Count word* holds the counts of -threads holding the lock at the moment on read, update and write levels. *Wait word* holds the -number of pending threads, trying to acquire the lock on write level, at the moment. These two words -are updated only via compare-and-swap operations. In addition to the checks essential for keeping -exclusiveness invariants, described in the previous section, when a thread tries to acquire the lock -on read level, it should check there are no threads pending for write access, to prevent starvation -of the latter. +Each lock is represented by two 32-bit words in the Chronicle Map memory. *Count word* holds the +numbers of threads holding the lock at the moment on the read, update and write levels. *Wait word* +holds the number of pending threads, trying to acquire the lock on the write level, at the moment. +These two words are updated only via compare-and-swap operations. In addition to the checks +essential for keeping exclusiveness invariants, described in the previous section, when a thread +tries to acquire the lock on the read level, it checks there are no threads pending for write +access, to prevent starvation of the latter. -As Chronicle Map is designed for inter-process communication and it doesn't use operating system -level locks, it doesn't have queues of threads, waiting for lock acquisition and conditional -wake-ups. Threads should try to acquire a lock in a spin loop, probably yielding and/or sleeping -between the attempts, after certain numbers of unsuccessful attempts. +As Chronicle Map is designed for inter-process communication, and it doesn't use locks provided by +an operation system, Chronicle Map doesn't implement queues of threads, waiting for lock acquisition +and conditional wake-ups. Threads acquire locks in a spin loop, yielding and/or sleeping after a +certain threshold numbers of unsuccessful attempts. -> Chronicle Map implementations specialized for single-process access may implement thread some kind -> of thread queues and wake-ups on their runtime level; certain lock acquisition strategy is out of -> scope of the Chronicle Map data structure specification. +> Chronicle Map implementations specialized for single-process access to instances may implement +> some kind of thread queues and wake-ups on their runtime level; certain lock acquisition strategy +> is out of scope of the Chronicle Map data store specification. ## The segment data structure @@ -77,62 +86,69 @@ Under normal conditions, a segment is represented by a single *tier*. ### Segment tier -Segment tier basically consists of two parts: +A segment tier basically consists of two parts: 1. **Hash lookup**, a flat, power-of-two sized open addressing hash table with collision resolution via linear probing, in which the role of keys is played by parts of the Chronicle Map's key hash - codes (different part of the hash code, from what is used for choosing the segment), the values in - this hash table are allocation identifiers within the segment tier's *entry space*. + codes (different part of the hash code, from what is used for choosing the segment), values in this + hash table are allocation identifiers within this segment tier's *entry space*. - > In the reference Chronicle Map implementation hash lookup table size is chosen in order to keep + > In the reference Chronicle Map implementation hash lookup's table size is chosen in order to keep > table load factor between 0.33 and 0.66. - 2. **Entry space** is the area to allocate blocks of memory to store the actual Chronicle Map's + 2. The **entry space** is the area to allocate blocks of memory to store the actual Chronicle Map's entries (key and value bytes sequences), along with some metadata. - > The specific memory allocation algorithm may vary, depending on characteristics of the keys - > and values stored in the specific Chronicle Map, time-space tradeoff chosen, etc. One important - > property that the allocator should likely obey, is *not* requiring to touch all it's memory up - > front, for example zeroing out or splitting the memory in blocks, going though all of them and - > creating an intrusive linked list. Touching entry space memory on demand takes advantage from - > the Linux memory mapping feature - lazy page allocation. This, in turn, allows to overcommit - > memory for segment tier's entry spaces without sacrificing actual memory usage much (because - > hash lookup slots, that should be overcommited too, are relatively small). This, in turn, - > mitigates variance of segments' filling rates and makes slow *tier chaining* (see below) very - > unlikely. + > The specific memory allocation algorithm may vary, depending on the characteristics of the keys + > and values stored in the specific Chronicle Map, the choice in the context of the time-space + > tradeoff, etc. One important property that the allocator should obey, is *not* requiring to touch + > all it's memory up front, for example zeroing out or splitting the memory in blocks, going though + > all of them and creating an intrusive linked list. Touching entry space memory on demand takes + > advantage from the Linux memory mapping feature - lazy page allocation. This, in turn, allows to + > overcommit memory for segment tier's entry spaces without sacrificing actual memory usage much + > (because hash lookup slots, that should be overcommitted too, are relatively small). This, + > in turn, mitigates variance of segments' filling rates and makes slow *tier chaining* (see below) + > very unlikely. > The reference Chronicle Map implementation uses the following allocation algorithm: it is > initialized with the *chunk size* depending of the configured properties of keys and values, - > which are going to be stored in the Chronicle Map, virtually splits the entry space into M - > chunks, and a bit set, where each bit corresponds to a single chunk of the same index. Allocation - > procedure computes the number of chunks that cover the requested allocation size and finds the - > same number of continuous zero bits in the bit set. Allocation identifier is the index of the - > first chunk of the allocated block. + > which are going to be stored in the Chronicle Map. The entry space is virtually split into M + > chunks and a bit set, where each bit corresponds to the single chunk of the same index. + > The allocation procedure computes the number of chunks that cover the requested allocation size + > and finds that number of continuous zero bits in the bit set. The allocation identifier is the + > index of the first chunk of the allocated block. -Hash lookup slots' size is either 4 or 8 bytes. Allocation identifiers use the minimum required bits -to identify themselves within the entry space, the rest bits of hash lookup slots are used to store -bits of the Chronicle Map's key hash codes. +The hash lookup slots' size is either 4 or 8 bytes. Allocation identifiers use the minimum required +bits to identify themselves within the entry space, the rest bits of hash lookup slots are used to +store bits of the Chronicle Map's key hash codes, to minimize full collisions on the hash lookup +level. + +> Hash lookup slots are disallowed to be of any integral byte size from 3 to 8 bytes, despite it +> would be more agile than limiting to 4 or 8 bytes, because cross cache line reads and writes on +> x86 are inherently not atomic, that makes unsafe to perform some write operations with a segment +> locked on the update level. Also, unaligned slots could be corrupted if the writing process is +> interrupted unconditionally, or the operating system fails. ### Tier chaining -If segment tier is filled up, i. e. on some entry insertion request entry space fails to allocate +If a segment tier is filled up, i. e. on some entry insertion request entry space fails to allocate a memory block sufficient to place the new entry, *a whole new segment tier is allocated and chained after the previous tier.* All tiers, either first in their segments or chained, are identical. > A segment with one or several chained tiers might be much slower for key search, than a "normal" -> segment with the single tier. Tier chaining is an *exceptional* mechanism for dealing with segment +> segment of a single tier. Tier chaining is an *exceptional* mechanism for dealing with segment > overflow. Chronicle Map shows it's efficiency properties when a very little fraction or none of > it's segments has chained tiers. Chronicle Map should be configured accordingly. -### Procedure of searching for a key within a segment +### The procedure of searching for a key within a segment
    -
  1. Extract the part of the queried key's hash code, that should serve as a key in hash lookup - tables.
  2. +
  3. Extract the part of the queried key's hash code, that serves as a key in hash lookup tables of + the tiers.
  4. -
  5. [For a segment tier, the first tier initially] +
  6. [For each segment tier in the chain, the first tier initially] -

    Search for the slot with the extracted hash lookup key in the tier's hash lookup, using +

    Search for a slot with the extracted hash lookup key in the tier's hash lookup, using standard search algorithm for hash tables with open addressing and collision resolution via linear probing. @@ -142,14 +158,15 @@ after the previous tier.* All tiers, either first in their segments or chained, location, byte-by-byte.

    1. -
    2. If keys are identical, the entry for the queried within the segment is found, exit the - procedure.
    3. +
    4. If the keys are identical, the entry for the queried key within the segment is found, exit + the procedure.
    5. -
    6. If keys are not identical (it means parts of their hash codes collide), continue the search - in the current tier's hash lookup table (step #2), without resetting the slot index.
    7. +
    8. If the keys are not identical (because of a full collision of parts of their hash codes for + storing in hash lookup tables), continue the search in the current tier's hash lookup table + (step #2), without resetting the hash lookup slot index.
    -
  7. If slot is not found, and there are no more chained tiers, the entry for the queried key +
  8. If such slot is not found, and there are no more chained tiers, the entry for the queried key within the segment is not found, exit the procedure.

    If there is a chained tier, go to it and start searching again from step #2.

  9. diff --git a/spec/3-memory-layout.md b/spec/3-memory-layout.md index ed7050198..5d7abc615 100644 --- a/spec/3-memory-layout.md +++ b/spec/3-memory-layout.md @@ -1,44 +1,25 @@ -# Chronicle Map Memory Layout +# Chronicle Map Data Store Memory Layout and Initialization -Newly created Chronicle Map (or just mapped from an already existing persistence file) takes one big -continuous block of memory. It's structure, from lower addresses to higher: +A newly created Chronicle Map instance (or just mapped from an already existing persistence file) +takes one big continuous block of memory. Its structure, from lower addresses to higher: - 1. Self-bootstrapping header - 2. Global mutable state + 1. [Self-bootstrapping header](#self-bootstrapping-header) + 2. [Global mutable state](#global-mutable-state) 3. Alignment to the next page boundary by addresses (page size of the current memory mapping), - if a new Chronicle Map is created, if the existing one (e. g. persisted) is loaded, the alignment - is specified in the global mutable state (however, specifically this field of global mutable state - is actually immutable). + if a new Chronicle Map is created, if an existing one (i. e. persisted) is loaded, the alignment + is specified in the global mutable state (specifically this field of global mutable state is + actually immutable, once written). - > The purpose of this alignment is to minimize the number of pages spanned by the following segment - > headers area. Segment headers area is frequently accessed and updated, so we want to minimize the - > number of TLB cache entries it almost constantly reside and the number of pages to flush to disk, - > if the Chronicle Map is persisted. + > The purpose of this alignment is to minimize the number of pages spanned by the following + > *segment headers area*. The segment headers area is frequently accessed and updated, so the pages + > it spans almost always reside in TLB cache and always need to be flushed to the disk. - > Need to store the alignment, because other accessing processes and/or implementations of - > Chronicle Map may use different page size for memory mapping. + > The alignment is persisted because other accessing processes, operation systems and + > implementations of Chronicle Map may use different page size for memory mapping. - 4. Segment headers area - 5. Main segments area - 6. Zero to several *extra tier bulks*. If the Chronicle Map is persisted to a file, each extra tier - bulk is pre-aligned to the granularity of offsets in files supported by memory mapping facility in - the operating system running at the moment of extra tier bulk allocation, then these alignments are - stored for each extra tier bulk in the global mutable state. - - > The reason to make alignment before each extra tier bulk is different from that for alignment - > before segment headers area. Extra tier bulks are allocated in runtime, there are zero bulks in - > a newly created Chronicle Map. If the Chronicle Map is persisted, it should extend the persisted - > file and map the newly allocated file space into memory. In processes which will load this - > Chronicle Map later the already existing Chronicle Map space and the new extra tier bulk will - > be served by a single memory mapping, hence continuous range of addresses. But in the processes - > *currently working* with this Chronicle Map (including the process triggered extra tier bulk - > allocation itself), i. e. already using a *shorter* memory-mapping, this new extra tier bulk - > should be loaded as a *separate* memory mapping. This separate memory mapping with offset in - > the persistence file, should have the offset aligned accordingly. - - > Granularity of offsets in files, supported by memory mapping facility is equal to *native* page - > size in Linux, i. e. most likely 4KB, and "file allocation granularity" in Windows, i. e. most - > likely 64KB. + 4. [Segment headers area](#segment-headers-area) + 5. [Main segments area](#main-segments-area) + 6. Zero to several [extra tier bulks](#extra-tier-bulks). ## Self-Bootstrapping Header diff --git a/spec/README.md b/spec/README.md index 9267e1d81..2af4b635c 100644 --- a/spec/README.md +++ b/spec/README.md @@ -1,6 +1,4 @@ -# The Chronicle Map Specification, revision 1-pre - -This directory contains the Chronicle Map Specification. +# The Chronicle Map Data Store Specification, revision 1-pre This is currently work-in-progress. diff --git a/src/main/java/net/openhft/chronicle/hash/ChronicleHashBuilder.java b/src/main/java/net/openhft/chronicle/hash/ChronicleHashBuilder.java index f8c3038d9..82a360eaa 100755 --- a/src/main/java/net/openhft/chronicle/hash/ChronicleHashBuilder.java +++ b/src/main/java/net/openhft/chronicle/hash/ChronicleHashBuilder.java @@ -286,7 +286,7 @@ public interface ChronicleHashBuilder, * *

    It is strongly advised not to configure {@code maxBloatFactor} to more than {@code 10.0}, * almost certainly, you either should configure {@code ChronicleHash}es completely differently, - * or this data structure doesn't fit you case. + * or this data store doesn't fit to your case. * * @param maxBloatFactor the maximum number ot times, the created hash container is supposed * to bloat up beyond the {@link #entries(long)} @@ -393,19 +393,19 @@ public interface ChronicleHashBuilder, /** * Configures timeout after which entries, marked as removed in the Chronicle Hash, constructed - * by this builder, are allowed to be completely removed from the data structure. In replicated + * by this builder, are allowed to be completely removed from the data store. In replicated * Chronicle nodes, when {@code remove()} on the key is called, the corresponding entry - * is not immediately erased from the data structure, to let the distributed system eventually + * is not immediately erased from the data store, to let the distributed system eventually * converge on some value for this key (or converge on the fact, that this key is removed). * Chronicle Hash watch in runtime after the entries, and if one is removed and not updated * in any way for this {@code removedEntryCleanupTimeout}, Chronicle is allowed to remove this - * entry completely from the data structure. This timeout should depend on your distributed + * entry completely from the data store. This timeout should depend on your distributed * system topology, and typical replication latencies, that should be determined experimentally. * *

    Default timeout is 1 minute. * * @param removedEntryCleanupTimeout timeout, after which stale removed entries could be erased - * from Chronicle Hash data structure completely + * from Chronicle Hash data store completely * @param unit time unit, in which the timeout is given * @return this builder back * @throws IllegalArgumentException is the specified timeout is less than 1 millisecond diff --git a/src/main/java/net/openhft/chronicle/hash/impl/VanillaChronicleHash.java b/src/main/java/net/openhft/chronicle/hash/impl/VanillaChronicleHash.java index ff4ee4059..ee3bdb2f1 100644 --- a/src/main/java/net/openhft/chronicle/hash/impl/VanillaChronicleHash.java +++ b/src/main/java/net/openhft/chronicle/hash/impl/VanillaChronicleHash.java @@ -734,7 +734,7 @@ public abstract class VanillaChronicleHash builder = ChronicleMapBuilder - .of(Data.class, Data.class) + ChronicleMapBuilder builder = ChronicleMapBuilder + .of(IData.Data.class, IData.Data.class) .keyReaderAndDataAccess(new DataReader(), new DataDataAccess()) .valueReaderAndDataAccess(new DataReader(), new DataDataAccess()) .actualChunkSize(64) .entries(1000); - try (ChronicleMap map = newInstance(builder)) { + try (ChronicleMap map = newInstance(builder)) { for (int i = 0; i < 100; i++) { - Data key = new Data(); - Data value = new Data(); + IData.Data key = new IData.Data(); + IData.Data value = new IData.Data(); key.setText("key-" + i); key.setNumber(i); value.setNumber(i); @@ -2685,30 +2685,30 @@ public class CHMUseCasesTest { } } - private static class DataDataAccess extends BytesMarshallableDataAccess { + private static class DataDataAccess extends BytesMarshallableDataAccess { public DataDataAccess() { - super(Data.class); + super(IData.Data.class); } @Override - protected Data createInstance() { - return new Data(); + protected IData.Data createInstance() { + return new IData.Data(); } @Override - public DataAccess copy() { + public DataAccess copy() { return new DataDataAccess(); } } - private static class DataReader extends BytesMarshallableReader { + private static class DataReader extends BytesMarshallableReader { public DataReader() { - super(Data.class); + super(IData.Data.class); } @Override - protected Data createInstance() { - return new Data(); + protected IData.Data createInstance() { + return new IData.Data(); } } } @@ -2721,53 +2721,53 @@ interface IData extends BytesMarshallable { String getText(); int getNumber(); -} -class Data implements IData, BytesMarshallable { - static final long MAGIC = 0x8081828384858687L; - static final long MAGIC2 = 0xa0a1a2a3a4a5a6a7L; + class Data implements IData, BytesMarshallable { + static final long MAGIC = 0x8081828384858687L; + static final long MAGIC2 = 0xa0a1a2a3a4a5a6a7L; - String text; - int number; + String text; + int number; - @Override - public String getText() { - return text; - } + @Override + public String getText() { + return text; + } - @Override - public void setText(String text) { - this.text = text; - } + @Override + public void setText(String text) { + this.text = text; + } - @Override - public int getNumber() { - return number; - } + @Override + public int getNumber() { + return number; + } - @Override - public void setNumber(int number) { - this.number = number; - } + @Override + public void setNumber(int number) { + this.number = number; + } - @Override - public void readMarshallable(@NotNull Bytes in) throws IllegalStateException { - long magic = in.readLong(); - if (magic != MAGIC) - throw new AssertionError("Start " + Long.toHexString(magic)); - text = in.readUTFΔ(); - number = in.readInt(); - long magic2 = in.readLong(); - if (magic2 != MAGIC2) - throw new AssertionError("End " + Long.toHexString(magic2)); - } + @Override + public void readMarshallable(@NotNull Bytes in) throws IllegalStateException { + long magic = in.readLong(); + if (magic != MAGIC) + throw new AssertionError("Start " + Long.toHexString(magic)); + text = in.readUTFΔ(); + number = in.readInt(); + long magic2 = in.readLong(); + if (magic2 != MAGIC2) + throw new AssertionError("End " + Long.toHexString(magic2)); + } - @Override - public void writeMarshallable(@NotNull Bytes out) { - out.writeLong(MAGIC); - out.writeUTFΔ(text); - out.writeInt(number); - out.writeLong(MAGIC2); + @Override + public void writeMarshallable(@NotNull Bytes out) { + out.writeLong(MAGIC); + out.writeUTFΔ(text); + out.writeInt(number); + out.writeLong(MAGIC2); + } } }