mirror of
https://github.com/OpenHFT/Chronicle-Map.git
synced 2021-09-13 01:26:12 +00:00
Make the specification agnostic to the implementation runtime thread model (expand to green threads)
This commit is contained in:
leventov
+13
-2
@@ -25,6 +25,14 @@
|
||||
|
||||
Chronicle Map provides some guarantees under the following assumptions:
|
||||
|
||||
#### Threads
|
||||
|
||||
Throughout this specification, the term *thread* means the execution thread of the runtime of the
|
||||
Chronicle Map implementation. It might be an OS thread or a "greener" thread.
|
||||
|
||||
- Operations declared for execution in some threads are performed sequentially in the same order,
|
||||
with full memory visibility between operations.
|
||||
|
||||
#### CPU
|
||||
|
||||
- CPU supports atomic 64-bit compare-and-swap operations with aligned memory, i. e. if several
|
||||
@@ -36,6 +44,8 @@ Chronicle Map provides some guarantees under the following assumptions:
|
||||
|
||||
The two above points are true for CPUs with x86 and x86_64 architectures.
|
||||
|
||||
#### Memory ordering
|
||||
|
||||
- If some values X<sub>i</sub> are written at addresses A<sub>i</sub>, then 32- or 64-bit value Y
|
||||
is written at address B, there is a way to ensure some memory order. In particular, if a
|
||||
concurrent thread reads value Y from address B, after that it will read X<sub>i</sub values from
|
||||
@@ -102,8 +112,9 @@ Therefore, *some entries updated shortly before the failure could be lost.*
|
||||
|
||||
The ultimate goal of Chronicle Map design is efficiency:
|
||||
|
||||
- If the number of entries in Chronicle Map is much greater than the number of accessor hardware
|
||||
threads, it scales well up to the total number of hardware threads present in the system.
|
||||
- If the number of entries in Chronicle Map is much greater than the number of accessor CPUs, it
|
||||
scales (i. e. adding the last CPU still adds to the total throughput) up to the total number of
|
||||
CPUs present in the system.
|
||||
- Chronicle Map is a low-latency key-value store, meaning that for any particular percentile (90%,
|
||||
99%, 99.9% etc.) the latency of Chronicle Map operations should be the best or among the best
|
||||
compared to latencies of other key-value stores filled with the same data and providing the same or
|
||||
|
||||
+10
-10
@@ -36,9 +36,9 @@ the segment lock.
|
||||
|
||||
### Lock levels
|
||||
|
||||
The **read level** is shared, several threads and processes could hold the segment lock on the read
|
||||
level at the same time. This level is used for read queries on a single or multiple keys, like
|
||||
simple value reading by the key. This level couldn't be upgraded to the update or write level.
|
||||
The **read level** is shared, several threads (across OS processes) could hold the segment lock on
|
||||
the read level at the same time. This level is used for read queries on a single or multiple keys,
|
||||
like simple value reading by the key. This level couldn't be upgraded to the update or write level.
|
||||
|
||||
> The read level couldn't upgrade to the higher levels, because this is dead-lock prone. Two threads
|
||||
> could acquire the same lock on the read level, then both try to upgrade to the write level - and
|
||||
@@ -46,8 +46,8 @@ simple value reading by the key. This level couldn't be upgraded to the update o
|
||||
> this issue, too.)
|
||||
|
||||
The **update level** is partially exclusive: a thread could hold the segment lock on the update
|
||||
level while some other threads or processes are holding the same lock on the read level, but not
|
||||
the update or write level.
|
||||
level while some other threads (across OS processes) are holding the same lock on the read level,
|
||||
but not the update or write level.
|
||||
|
||||
This level is used for "typically read" types of queries, which occasionally should be able to write
|
||||
to the data store, hence need to upgrade to the write level without releasing the lock, for example,
|
||||
@@ -58,10 +58,10 @@ level, even if they never perform writes.
|
||||
A segment lock acquired on the update level could be upgraded to the write level.
|
||||
|
||||
The **write level** is totally exclusive: when a thread holds the segment lock on the write level,
|
||||
no other thread or process could hold the same lock on any level. The write level is used for write
|
||||
queries.
|
||||
no other thread (across OS processes) could hold the same lock on any level. The write level is used
|
||||
for write queries.
|
||||
|
||||
## Lock implementation
|
||||
## [Lock implementation](3_2-lock-structure.md)
|
||||
|
||||
Each lock is represented by two 32-bit words in the Chronicle Map memory. *Count word* holds the
|
||||
numbers of threads holding the lock at the moment on the read, update and write levels. *Wait word*
|
||||
@@ -77,8 +77,8 @@ and conditional wake-ups. Threads acquire locks in a spin loop, yielding and/or
|
||||
certain threshold numbers of unsuccessful attempts.
|
||||
|
||||
> Chronicle Map implementations specialized for single-process access to instances may implement
|
||||
> some kind of thread queues and wake-ups on their runtime level; certain lock acquisition strategy
|
||||
> is out of scope of the Chronicle Map data store specification.
|
||||
> some kind of thread queues, wake-ups and cooperative scheduling on the runtime level; certain lock
|
||||
> acquisition strategy is out of scope of the Chronicle Map data store specification.
|
||||
|
||||
## The segment data structure
|
||||
|
||||
|
||||
@@ -254,16 +254,16 @@ The segment tier counters structure is 64 bytes long:
|
||||
|
||||
#### Free list
|
||||
|
||||
A free list starts with offset [`tierHashLookupOuterSize`
|
||||
](3_1-header-fields.md#tierhashlookupoutersize) + 64 from the beginning address of a containing
|
||||
A free list starts with offset [`tierHashLookupOuterSize`](
|
||||
3_1-header-fields.md#tierhashlookupoutersize) + 64 from the beginning address of a containing
|
||||
segment tier.
|
||||
|
||||
This is a list of [`actualChunksPerSegmentTier`](3_1-header-fields.md#actualchunkspersegmenttier)
|
||||
bits, each corresponding to a chunk in the [entry space](#entry-space). If the chunk is allocated
|
||||
for some entry, the corresponding bit is set to 1, otherwise it is clear.
|
||||
|
||||
> The reference Java implementation: [`SingleThreadedFlatBitSetFrame`
|
||||
> ](https://github.com/OpenHFT/Chronicle-Algorithms/blob/chronicle-algorithms-1.1.6/src/main/java/net/openhft/chronicle/algo/bitset/SingleThreadedFlatBitSetFrame.java).
|
||||
> The reference Java implementation: [`SingleThreadedFlatBitSetFrame`](
|
||||
> https://github.com/OpenHFT/Chronicle-Algorithms/blob/chronicle-algorithms-1.1.6/src/main/java/net/openhft/chronicle/algo/bitset/SingleThreadedFlatBitSetFrame.java).
|
||||
|
||||
#### Entry space
|
||||
|
||||
|
||||
@@ -55,14 +55,21 @@ The lock structure is 8 bytes (64 bits) long.
|
||||
|
||||
Attempt to acquire the read or update lock using the corresponding *try acquire* procedure, until it
|
||||
succeeds, optionally inserting some directives aimed to yield the resources after failed attempts,
|
||||
for example, x86's `PAUSE` instruction, or an operation system command to schedule another thread to
|
||||
the current CPU, or a command to clock down the current CPU, etc.
|
||||
for example, x86's `PAUSE` instruction, or an OS command to schedule another OS thread to the
|
||||
current CPU, or a runtime command to schedule another green thread to the current OS thread, or a
|
||||
command to clock down the current CPU, etc.
|
||||
|
||||
If the runtime of the Chronicle Map implementation has cooperative (rather than preemptive)
|
||||
scheduling of [threads](1-design-goals.md#threads), thread switch must be enabled after failed
|
||||
attempts. Some explicit command is executed, letting the runtime to switch between green threads.
|
||||
|
||||
> Possibility for thread switch is required to avoid dead locks between threads.
|
||||
|
||||
There is no single exact scheme that implementations should follow, it is unspecified. The
|
||||
implementation may vary for the read and update locks as well.
|
||||
|
||||
> The reference Java implementation doesn't insert any resources yielding directives after failed
|
||||
> try acquire attempts, for both read and update lock acquisition.
|
||||
> acquisition attempts, for both read and update lock acquisition.
|
||||
|
||||
Implementations must not try to acquire the read or update lock indefinitely, after some finite
|
||||
number of failed attempts or some finite time elapsed since the time-limited lock acquisition
|
||||
@@ -142,16 +149,23 @@ procedure.
|
||||
procedure fails.
|
||||
3. Read the lock state.
|
||||
4. If the count word of the lock state is not equal to 0 (in case of acquiring the write lock) or
|
||||
0x40000000 (in case of upgrading update to write lock), go to the step 7.
|
||||
0x40000000 (in case of upgrading update to write lock), go to the 7th step.
|
||||
5. If the wait count is 0, this procedure fails.
|
||||
6. Perform a CAS operation on the whole lock structure, comparing the state, read on the step 3,
|
||||
6. Perform a CAS operation on the whole lock structure, comparing the state, read on the 3rd step,
|
||||
with an updated state with the wait count decremented, and count word of 0x80000000 (the write lock
|
||||
flag set). If the CAS operation succeeds, the time-limited acquisition succeeds. If the CAS
|
||||
operation fails, continue with the step 7.
|
||||
operation fails, continue with the 7th step.
|
||||
7. Check if this time-limited procedure has run out of the time limit or the number of attempts.
|
||||
If it is, the time-limited acquisition fails, [deregister a wait](#deregister-wait) before exiting
|
||||
from this procedure (the result of the deregister wait procedure doesn't matter). Otherwise,
|
||||
continue from the step 3 of this procedure.
|
||||
continue from the 3rd step of this procedure, optionally performing some directives aimed to yield
|
||||
the resources at first, as described in the section about [time-limited read or update lock
|
||||
acquisition](#time-limited-read-or-update-lock-acquisition).
|
||||
|
||||
If the runtime of the Chronicle Map implementation has cooperative (rather than preemptive)
|
||||
scheduling of [threads](1-design-goals.md#threads), thread switch must be enabled before continuing
|
||||
with the 3rd step. Some explicit command is executed, letting the runtime to switch between green
|
||||
threads.
|
||||
|
||||
Implementations doesn't make locking attempts indefinitely [for the same reasons, as in case of
|
||||
acquiring read and update lock](#time-limited-read-or-update-lock-acquisition).
|
||||
|
||||
@@ -28,9 +28,9 @@ existing Map, persisted to some file).
|
||||
> This step follows [double-checked locking](https://en.wikipedia.org/wiki/Double-checked_locking)
|
||||
> pattern to avoid file locking in most cases when an already existing Chronicle Map is opened.
|
||||
|
||||
This step is serialized for the persistence file among threads within the current process, because
|
||||
file locking mechanism in operating systems is per-process, not per-thread. I. e. concurrent
|
||||
threads within the same process don't lock the same file at the same time.
|
||||
This step is serialized for the persistence file among threads within the current OS process,
|
||||
because file locking mechanism in operating systems is per-process, not per-thread. I. e.
|
||||
concurrent threads within the same OS process don't lock the same file at the same time.
|
||||
|
||||
Concurrent threads perform any initialization operations (Chronicle Map creation and [extra tier
|
||||
bulk allocation](#extra-tier-bulk-allocation)) with canonicalized file descriptor, that is closed
|
||||
@@ -59,8 +59,8 @@ existing Map, persisted to some file).
|
||||
bit of the 32-bit word by offset 8 from the beginning of the file, read and written in the
|
||||
little-endian order. Read the [Size Prefix Blob](
|
||||
https://github.com/OpenHFT/RFC/blob/master/Size-Prefixed-Blob/Size-Prefixed-Blob-0.1.md)
|
||||
specification for details. This step is the event threads and processes [waiting until Chronicle
|
||||
Map is ready](#wait-until-chronicle-map-is-ready) actually wait for.
|
||||
specification for details. This step is the event that is actually awaited by the threads (across
|
||||
OS processes) [waiting until Chronicle Map is ready](#wait-until-chronicle-map-is-ready).
|
||||
|
||||
### Wait until Chronicle Map is ready
|
||||
|
||||
@@ -68,7 +68,7 @@ Check until the readiness bit is set to 0. The readiness bit it the highest bit
|
||||
offset 8 from the beginning of the Chronicle Map persistence file, read in the little-endian order.
|
||||
|
||||
Optionally yield processor resources after failed checks, e. g. make the waiting thread sleep for
|
||||
some time. The exact yielding operation (or if it is making the current process sleeping, the exact
|
||||
some time. The exact yielding operation (or if it is making the current thread sleeping, the exact
|
||||
sleep interval) is unspecified.
|
||||
|
||||
> The reference Java implementations turns the waiting thread to sleep for 100 ms after failed
|
||||
@@ -77,8 +77,7 @@ sleep interval) is unspecified.
|
||||
Implementations must not wait until the Chronicle Map is ready indefinitely, after some finite
|
||||
number of failed checks, or some finite time elapsed since the start, the wait procedure must fail.
|
||||
|
||||
> Indefinite wait is dead lock prone, if the concurrent process or thread doing the initialization
|
||||
> fails itself.
|
||||
> Indefinite wait is dead lock prone, if the thread doing the initialization fails itself.
|
||||
|
||||
> The reference Java implementation checks until the Chronicle Map is ready for approximately 1
|
||||
> minute, then throws an exception.
|
||||
|
||||
Reference in New Issue
Block a user