diff --git a/spec/3-memory-layout.md b/spec/3-memory-layout.md index 67e61b2ac..00e89cf7f 100644 --- a/spec/3-memory-layout.md +++ b/spec/3-memory-layout.md @@ -54,7 +54,7 @@ offsets, etc. See the specification of the fields on [Map Header Fields](3_1-hea ## Global mutable state -The global mutable state is 36 bytes long. +The global mutable state is 33 bytes long. 1. Bytes 0..7 - the global mutable state lock, a [lock structure](3_2-lock-structure.md). @@ -62,12 +62,12 @@ The global mutable state is 36 bytes long. > (the write lock) level is used. This is done for sharing the lock implementation for the > global mutable state and segment headers (see below). - 2. Bytes 8..11 - the number of allocated [extra tier bulks](#extra-tier-bulks). A non-negative - 32-bit value, stored in the little-endian order. + 2. Bytes 8..10 - the number of allocated [extra tier bulks](#extra-tier-bulks). An unsigned 24-bit + value, stored in the little-endian order. See also the [extra tier bulk allocation](5-initialization.md#entra-tier-bulk-allocation) operation. - 3. Bytes 12..19 - the index of the first *free* segment tier. A non-negative 64-bit value, stored + 3. Bytes 11..15 - the index of the first *free* segment tier. An unsigned 40-bit value, stored in the little-endian order. Extra segment tiers are allocated in bulks, so there is usually a chain of allocated, but unused yet segment tiers. This field points to the head of this chain, or has value `0`, if there are no free segment tiers. @@ -79,11 +79,15 @@ The global mutable state is 36 bytes long. [`actualSegments`](3_1-header-fields.md#actualsegments) − 1) has tier index `actualSegments`, the first tier of the first extra tier bulk has tier index `actualSegments` + 1, etc. Tier indexes are 1-counted, because value 0 has some special meaning. - 4. Bytes 20..27 - the number of used extra segment tiers. A non-negative 64-bit value, stored in + 4. Bytes 16..20 - the number of used extra segment tiers. An unsigned 40-bit value, stored in the little-endian order. - 5. Bytes 28..35 - the offset of the segment headers area from the beginning of the memory of this - Chronicle Map instance. This field determines the size of [the 4th area of the general Chronicle - Map structure](#segment-headers-alignment). + 5. Bytes 21..24 - the offset of the segment headers area from the beginning of the memory of this + Chronicle Map instance. An unsigned 32-bit value, stored in the little-endian ordered. This field + determines the size of [the 4th area of the general Chronicle Map structure]( + #segment-headers-alignment). + 6. Bytes 25..32 - the Chronicle Map data store size, the offset to the end of the [main segments + area](#main-segments area) or the last [extra tier bulk](#extra-tier-bulks). A non-negative 64-bit + value, stored in the little-endian order. > The reference Java implementation: [`VanillaGlobalMutableState` > ](../src/main/java/net/openhft/chronicle/hash/VanillaGlobalMutableState.java). diff --git a/spec/3_2-lock-structure.md b/spec/3_2-lock-structure.md index eec64312b..4951fd715 100644 --- a/spec/3_2-lock-structure.md +++ b/spec/3_2-lock-structure.md @@ -158,7 +158,7 @@ acquiring read and update lock](#time-limited-read-or-update-lock-acquisition). > ## The reference Java implementation > -> Attempt operations: [`VanillaReadWriteUpdateWithWaitsLockingStrategy`]( +> Attempt, release and downgrade operations: [`VanillaReadWriteUpdateWithWaitsLockingStrategy`]( > https://github.com/OpenHFT/Chronicle-Algorithms/blob/master/src/main/java/net/openhft/chronicle/algo/locks/VanillaReadWriteUpdateWithWaitsLockingStrategy.java) > > Time-limited operations: [`BigSegmentHeader`]( diff --git a/spec/5-initialization.md b/spec/5-initialization.md index 120464c1e..db77d499c 100644 --- a/spec/5-initialization.md +++ b/spec/5-initialization.md @@ -50,10 +50,12 @@ existing Map, persisted to some file). 3-memory-layout.md#entry-space), i. e. zero out this tier's hash lookup, segment tier counters area and free list. 6. White the segment headers offset into the 5th field of the global mutable state. - 7. If the Chronicle Map is persisted, ensure all data written to the file is flushed to the disk. + 7. Write the offset to the end of the main segments area into the 6th field of the global mutable + state. + 8. If the Chronicle Map is persisted, ensure all data written to the file is flushed to the disk. For example on Linux, this could be done with `msync` and `fdatasync` calls, on Windows - with `FlushFileBuffers` system call. - 8. If the Chronicle Map is persisted, write the readiness bit into the header, i. e. the highest + 9. If the Chronicle Map is persisted, write the readiness bit into the header, i. e. the highest bit of the 32-bit word by offset 8 from the beginning of the file, read and written in the little-endian order. Read the [Size Prefix Blob]( https://github.com/OpenHFT/RFC/blob/master/Size-Prefixed-Blob/Size-Prefixed-Blob-0.1.md) @@ -125,6 +127,8 @@ by [global mutable state](3-memory-layout.md#global-mutable-state) lock. 7. Write the [index](3-memory-layout.md#tier-index) of the first tier in this extra tier bulk into the 3rd field (the index of the first *free* segment tier) of this Chronicle Map's global mutable state. + 8. Increment the value in the 6th field (the data store size) of the global mutable state by + [`tierBulkSizeInBytes`](3_1-header-fields.md#tierbulksizeinbytes). ## Tier allocation diff --git a/src/main/java/net/openhft/chronicle/hash/VanillaGlobalMutableState.java b/src/main/java/net/openhft/chronicle/hash/VanillaGlobalMutableState.java index 24678eb02..4b32eb051 100644 --- a/src/main/java/net/openhft/chronicle/hash/VanillaGlobalMutableState.java +++ b/src/main/java/net/openhft/chronicle/hash/VanillaGlobalMutableState.java @@ -18,6 +18,8 @@ package net.openhft.chronicle.hash; import net.openhft.chronicle.bytes.Byteable; import net.openhft.chronicle.values.Group; +import net.openhft.chronicle.values.Range; +import net.openhft.chronicle.values.Values; /** *

This interface is a private API, don't use in client code

@@ -28,17 +30,27 @@ public interface VanillaGlobalMutableState extends Byteable { @Group(1) int getAllocatedExtraTierBulks(); - void setAllocatedExtraTierBulks(int allocatedExtraTierBulks); + void setAllocatedExtraTierBulks(@Range(min = 0, max = 16777215) int allocatedExtraTierBulks); @Group(2) long getFirstFreeTierIndex(); - void setFirstFreeTierIndex(long firstFreeTierIndex); + void setFirstFreeTierIndex(@Range(min = 0, max = 1099511627775L) long firstFreeTierIndex); @Group(3) long getExtraTiersInUse(); - void setExtraTiersInUse(long extraTiersInUse); + void setExtraTiersInUse(@Range(min = 0, max = 1099511627775L) long extraTiersInUse); @Group(4) long getSegmentHeadersOffset(); - void setSegmentHeadersOffset(long segmentHeadersOffset); + void setSegmentHeadersOffset(@Range(min = 0, max = 4294967295L) long segmentHeadersOffset); + + @Group(5) + long getDataStoreSize(); + void setDataStoreSize(@Range(min = 0, max = Long.MAX_VALUE) long dataStoreSize); + long addDataStoreSize(long addition); + + public static void main(String[] args) { + System.setProperty("chronicle.values.dumpCode", "true"); + Values.nativeClassFor(VanillaGlobalMutableState.class); + } } diff --git a/src/main/java/net/openhft/chronicle/hash/impl/VanillaChronicleHash.java b/src/main/java/net/openhft/chronicle/hash/impl/VanillaChronicleHash.java index f875a2929..96788f0ee 100644 --- a/src/main/java/net/openhft/chronicle/hash/impl/VanillaChronicleHash.java +++ b/src/main/java/net/openhft/chronicle/hash/impl/VanillaChronicleHash.java @@ -16,6 +16,7 @@ package net.openhft.chronicle.hash.impl; +import net.openhft.chronicle.algo.MemoryUnit; import net.openhft.chronicle.algo.locks.*; import net.openhft.chronicle.bytes.BytesStore; import net.openhft.chronicle.bytes.MappedBytesStoreFactory; @@ -434,6 +435,7 @@ public abstract class VanillaChronicleHash GIGABYTES.toBytes(1)) { + segmentHeadersOffset = computeSegmentHeadersOffset(); + } long sizeInBytesWithoutTiers = computeSizeInBytesWithoutTiers(segmentHeadersOffset); long sizeBeyondSegments = Math.max(raf.length() - sizeInBytesWithoutTiers, 0); - int allocatedExtraTierBulks = (int) (sizeBeyondSegments / tierBulkSizeInBytes); - long mapSize = pageAlign(sizeInBytesWithoutTiers + - allocatedExtraTierBulks * tierBulkSizeInBytes); - initBytesStoreAndHeadersViews(map(mapSize, 0)); + long dataStoreSize = globalMutableState().getDataStoreSize(); + int allocatedExtraTierBulks = globalMutableState().getAllocatedExtraTierBulks(); + if (dataStoreSize < sizeInBytesWithoutTiers || + ((dataStoreSize - sizeInBytesWithoutTiers) % tierBulkSizeInBytes != 0)) { + dataStoreSize = sizeInBytesWithoutTiers + allocatedExtraTierBulks * tierBulkSizeInBytes; + } else { + allocatedExtraTierBulks = + (int) ((dataStoreSize - sizeInBytesWithoutTiers) / tierBulkSizeInBytes); + } + initBytesStoreAndHeadersViews(map(dataStoreSize, 0)); resetGlobalMutableStateLock(); recoverAllocatedExtraTierBulks(allocatedExtraTierBulks); recoverSegmentHeadersOffset(segmentHeadersOffset); + recoverDataStoreSize(dataStoreSize); initOffsetsAndBulks(); } @@ -530,6 +540,14 @@ public abstract class VanillaChronicleHash implements VanillaChronicleMap map = wire.getValueIn().typedMarshallable(); assert map != null; map.initBeforeMapping(file, raf, headerBuffer.limit(), recover); - long expectedFileLength = map.expectedFileSize(); - if (!recover && expectedFileLength != file.length()) { + long dataStoreSize = map.globalMutableState().getDataStoreSize(); + if (!recover && dataStoreSize > file.length()) { throw new IOException("The file " + file + " the map is serialized from " + "has unexpected length " + file.length() + ", probably corrupted. " + - "Expected length is " + expectedFileLength); + "Data store size is " + dataStoreSize); } map.initTransientsFromBuilder(this); if (!recover) { @@ -1683,11 +1683,6 @@ public final class ChronicleMapBuilder implements try { VanillaChronicleMap map = newMap(); - // TODO this method had been moved -// if(OS.warnOnWindows(map.sizeInBytesWithoutTiers())){ -// throw new IllegalStateException("Windows cannot support this configuration"); -// } - BytesStore bytesStore = lazyNativeBytesStoreWithFixedCapacity(map.sizeInBytesWithoutTiers()); map.createMappedStoreAndSegments(bytesStore); diff --git a/src/main/java/net/openhft/chronicle/map/ReplicatedGlobalMutableState.java b/src/main/java/net/openhft/chronicle/map/ReplicatedGlobalMutableState.java index dd9e2de51..950790ef7 100644 --- a/src/main/java/net/openhft/chronicle/map/ReplicatedGlobalMutableState.java +++ b/src/main/java/net/openhft/chronicle/map/ReplicatedGlobalMutableState.java @@ -17,21 +17,30 @@ package net.openhft.chronicle.map; import net.openhft.chronicle.hash.VanillaGlobalMutableState; -import net.openhft.chronicle.values.Array; -import net.openhft.chronicle.values.Group; +import net.openhft.chronicle.values.*; interface ReplicatedGlobalMutableState extends VanillaGlobalMutableState { - @Group(5) - int getCurrentCleanupSegmentIndex(); - void setCurrentCleanupSegmentIndex(int currentCleanupSegmentIndex); - @Group(6) + int getCurrentCleanupSegmentIndex(); + void setCurrentCleanupSegmentIndex( + @Range(min = 0, max = Integer.MAX_VALUE) int currentCleanupSegmentIndex); + + @Group(7) + @Align(offset = 1) + int getModificationIteratorsCount(); + int addModificationIteratorsCount(@Range(min = 0, max = 128) int addition); + + @Group(8) + // Align to prohibit array filling the highest bit of the previous 2-byte block, that + // complicates generated code for all fields, defined in this interface. + @Align(offset = 2) @Array(length = 128) boolean getModificationIteratorInitAt(int index); void setModificationIteratorInitAt(int index, boolean init); - @Group(7) - int getModificationIteratorsCount(); - int addModificationIteratorsCount(int addition); + public static void main(String[] args) { + System.setProperty("chronicle.values.dumpCode", "true"); + Values.nativeClassFor(ReplicatedGlobalMutableState.class); + } }