Add dataStoreSize field to Chronicle Map's global mutable state

2021-09-13 01:26:12 +00:00 · 2016-02-13 21:27:30 +07:00
parent e5c37a1e8f
commit 05ccde9671
7 changed files with 87 additions and 43 deletions
@@ -54,7 +54,7 @@ offsets, etc. See the specification of the fields on [Map Header Fields](3_1-hea

 ## Global mutable state

-The global mutable state is 36 bytes long.
+The global mutable state is 33 bytes long.

 1. Bytes 0..7 - the global mutable state lock, a [lock structure](3_2-lock-structure.md).

@@ -62,12 +62,12 @@ The global mutable state is 36 bytes long.
 > (the write lock) level is used. This is done for sharing the lock implementation for the
 > global mutable state and segment headers (see below).

- 2. Bytes 8..11 - the number of allocated [extra tier bulks](#extra-tier-bulks). A non-negative
- 32-bit value, stored in the little-endian order.
+ 2. Bytes 8..10 - the number of allocated [extra tier bulks](#extra-tier-bulks). An unsigned 24-bit
+ value, stored in the little-endian order.

 See also the [extra tier bulk allocation](5-initialization.md#entra-tier-bulk-allocation)
 operation.
- 3. Bytes 12..19 - the index of the first *free* segment tier. A non-negative 64-bit value, stored
+ 3. Bytes 11..15 - the index of the first *free* segment tier. An unsigned 40-bit value, stored
 in the little-endian order. Extra segment tiers are allocated in bulks, so there is usually a chain
 of allocated, but unused yet segment tiers. This field points to the head of this chain, or has
 value `0`, if there are no free segment tiers.
@@ -79,11 +79,15 @@ The global mutable state is 36 bytes long.
 [`actualSegments`](3_1-header-fields.md#actualsegments) &minus; 1) has tier index `actualSegments`,
 the first tier of the first extra tier bulk has tier index `actualSegments` + 1, etc. Tier indexes
 are 1-counted, because value 0 has some special meaning.
- 4. Bytes 20..27 - the number of used extra segment tiers. A non-negative 64-bit value, stored in
+ 4. Bytes 16..20 - the number of used extra segment tiers. An unsigned 40-bit value, stored in
 the little-endian order.
- 5. Bytes 28..35 - the offset of the segment headers area from the beginning of the memory of this
- Chronicle Map instance. This field determines the size of [the 4th area of the general Chronicle
- Map structure](#segment-headers-alignment).
+ 5. Bytes 21..24 - the offset of the segment headers area from the beginning of the memory of this
+ Chronicle Map instance. An unsigned 32-bit value, stored in the little-endian ordered. This field
+ determines the size of [the 4th area of the general Chronicle Map structure](
+ #segment-headers-alignment).
+ 6. Bytes 25..32 - the Chronicle Map data store size, the offset to the end of the [main segments
+ area](#main-segments area) or the last [extra tier bulk](#extra-tier-bulks). A non-negative 64-bit
+ value, stored in the little-endian order.

 > The reference Java implementation: [`VanillaGlobalMutableState`
 > ](../src/main/java/net/openhft/chronicle/hash/VanillaGlobalMutableState.java).
@@ -158,7 +158,7 @@ acquiring read and update lock](#time-limited-read-or-update-lock-acquisition).

 > ## The reference Java implementation
 >
-> Attempt operations: [`VanillaReadWriteUpdateWithWaitsLockingStrategy`](
+> Attempt, release and downgrade operations: [`VanillaReadWriteUpdateWithWaitsLockingStrategy`](
 > https://github.com/OpenHFT/Chronicle-Algorithms/blob/master/src/main/java/net/openhft/chronicle/algo/locks/VanillaReadWriteUpdateWithWaitsLockingStrategy.java)
 >
 > Time-limited operations: [`BigSegmentHeader`](
@@ -50,10 +50,12 @@ existing Map, persisted to some file).
 3-memory-layout.md#entry-space), i. e. zero out this tier's hash lookup, segment tier counters area
 and free list.
 6. White the segment headers offset into the 5th field of the global mutable state.
- 7. If the Chronicle Map is persisted, ensure all data written to the file is flushed to the disk.
+ 7. Write the offset to the end of the main segments area into the 6th field of the global mutable
+ state.
+ 8. If the Chronicle Map is persisted, ensure all data written to the file is flushed to the disk.
 For example on Linux, this could be done with `msync` and `fdatasync` calls, on Windows - with
 `FlushFileBuffers` system call.
- 8. If the Chronicle Map is persisted, write the readiness bit into the header, i. e. the highest
+ 9. If the Chronicle Map is persisted, write the readiness bit into the header, i. e. the highest
 bit of the 32-bit word by offset 8 from the beginning of the file, read and written in the
 little-endian order. Read the [Size Prefix Blob](
 https://github.com/OpenHFT/RFC/blob/master/Size-Prefixed-Blob/Size-Prefixed-Blob-0.1.md)
@@ -125,6 +127,8 @@ by [global mutable state](3-memory-layout.md#global-mutable-state) lock.
 7. Write the [index](3-memory-layout.md#tier-index) of the first tier in this extra tier bulk into
 the 3rd field (the index of the first *free* segment tier) of this Chronicle Map's global mutable
 state.
+ 8. Increment the value in the 6th field (the data store size) of the global mutable state by
+ [`tierBulkSizeInBytes`](3_1-header-fields.md#tierbulksizeinbytes).

 ## Tier allocation

@@ -18,6 +18,8 @@ package net.openhft.chronicle.hash;

 import net.openhft.chronicle.bytes.Byteable;
 import net.openhft.chronicle.values.Group;
+import net.openhft.chronicle.values.Range;
+import net.openhft.chronicle.values.Values;

 /**
 * <h1>This interface is a private API, don't use in client code</h1>
@@ -28,17 +30,27 @@ public interface VanillaGlobalMutableState extends Byteable {

    @Group(1)
    int getAllocatedExtraTierBulks();
-    void setAllocatedExtraTierBulks(int allocatedExtraTierBulks);
+    void setAllocatedExtraTierBulks(@Range(min = 0, max = 16777215) int allocatedExtraTierBulks);

    @Group(2)
    long getFirstFreeTierIndex();
-    void setFirstFreeTierIndex(long firstFreeTierIndex);
+    void setFirstFreeTierIndex(@Range(min = 0, max = 1099511627775L) long firstFreeTierIndex);

    @Group(3)
    long getExtraTiersInUse();
-    void setExtraTiersInUse(long extraTiersInUse);
+    void setExtraTiersInUse(@Range(min = 0, max = 1099511627775L) long extraTiersInUse);

    @Group(4)
    long getSegmentHeadersOffset();
-    void setSegmentHeadersOffset(long segmentHeadersOffset);
+    void setSegmentHeadersOffset(@Range(min = 0, max = 4294967295L) long segmentHeadersOffset);
+
+    @Group(5)
+    long getDataStoreSize();
+    void setDataStoreSize(@Range(min = 0, max = Long.MAX_VALUE) long dataStoreSize);
+    long addDataStoreSize(long addition);
+
+    public static void main(String[] args) {
+        System.setProperty("chronicle.values.dumpCode", "true");
+        Values.nativeClassFor(VanillaGlobalMutableState.class);
+    }
 }
@@ -16,6 +16,7 @@

 package net.openhft.chronicle.hash.impl;

+import net.openhft.chronicle.algo.MemoryUnit;
 import net.openhft.chronicle.algo.locks.*;
 import net.openhft.chronicle.bytes.BytesStore;
 import net.openhft.chronicle.bytes.MappedBytesStoreFactory;
@@ -434,6 +435,7 @@ public abstract class VanillaChronicleHash<K,
            zeroOutNewlyMappedChronicleMapBytes();
            // write the segment headers offset after zeroing out
            globalMutableState.setSegmentHeadersOffset(segmentHeadersOffset);
+            globalMutableState.setDataStoreSize(sizeInBytesWithoutTiers());
        } else {
            initBulks();
        }
@@ -481,24 +483,32 @@ public abstract class VanillaChronicleHash<K,
    }

    public final void createMappedStoreAndSegments() throws IOException {
-        // TODO this method had been moved -- not clear where
-        //OS.warnOnWindows(sizeInBytesWithoutTiers());
-        long mapSize = expectedFileSize();
-        createMappedStoreAndSegments(map(mapSize, 0));
+        createMappedStoreAndSegments(map(dataStoreSize(), 0));
    }

    public final void basicRecover() throws IOException {
-        long segmentHeadersOffset = computeSegmentHeadersOffset();
+        long segmentHeadersOffset = globalMutableState().getSegmentHeadersOffset();
+        if (segmentHeadersOffset <= 0 || segmentHeadersOffset % 4096 != 0 ||
+                segmentHeadersOffset > GIGABYTES.toBytes(1)) {
+            segmentHeadersOffset = computeSegmentHeadersOffset();
+        }
        long sizeInBytesWithoutTiers = computeSizeInBytesWithoutTiers(segmentHeadersOffset);
        long sizeBeyondSegments = Math.max(raf.length() - sizeInBytesWithoutTiers, 0);
-        int allocatedExtraTierBulks = (int) (sizeBeyondSegments / tierBulkSizeInBytes);
-        long mapSize = pageAlign(sizeInBytesWithoutTiers +
-                allocatedExtraTierBulks * tierBulkSizeInBytes);
-        initBytesStoreAndHeadersViews(map(mapSize, 0));
+        long dataStoreSize = globalMutableState().getDataStoreSize();
+        int allocatedExtraTierBulks = globalMutableState().getAllocatedExtraTierBulks();
+        if (dataStoreSize < sizeInBytesWithoutTiers ||
+                ((dataStoreSize - sizeInBytesWithoutTiers) % tierBulkSizeInBytes != 0)) {
+            dataStoreSize = sizeInBytesWithoutTiers + allocatedExtraTierBulks * tierBulkSizeInBytes;
+        } else {
+            allocatedExtraTierBulks =
+                    (int) ((dataStoreSize - sizeInBytesWithoutTiers) / tierBulkSizeInBytes);
+        }
+        initBytesStoreAndHeadersViews(map(dataStoreSize, 0));

        resetGlobalMutableStateLock();
        recoverAllocatedExtraTierBulks(allocatedExtraTierBulks);
        recoverSegmentHeadersOffset(segmentHeadersOffset);
+        recoverDataStoreSize(dataStoreSize);
        initOffsetsAndBulks();
    }

@@ -530,6 +540,14 @@ public abstract class VanillaChronicleHash<K,
        }
    }

+    private void recoverDataStoreSize(long dataStoreSize) {
+        if (globalMutableState.getDataStoreSize() != dataStoreSize) {
+            LOG.error("data store size of map at {} corrupted. stored: {}, should be: {}",
+                    file, globalMutableState.getDataStoreSize(), dataStoreSize);
+            globalMutableState.setDataStoreSize(dataStoreSize);
+        }
+    }
+
    private boolean persisted() {
        return file != null;
    }
@@ -609,11 +627,11 @@ public abstract class VanillaChronicleHash<K,
        return segmentHeadersOffset + actualSegments * (segmentHeaderSize + tierSize);
    }

-    public final long expectedFileSize() {
+    public final long dataStoreSize() {
        long sizeInBytesWithoutTiers = sizeInBytesWithoutTiers();
        int allocatedExtraTierBulks = !createdOrInMemory ?
                globalMutableState.getAllocatedExtraTierBulks() : 0;
-        return pageAlign(sizeInBytesWithoutTiers + allocatedExtraTierBulks * tierBulkSizeInBytes);
+        return sizeInBytesWithoutTiers + allocatedExtraTierBulks * tierBulkSizeInBytes;
    }

    @Override
@@ -800,6 +818,7 @@ public abstract class VanillaChronicleHash<K,
        // after we are sure the new bulk is initialized, update the global mutable state
        globalMutableState.setAllocatedExtraTierBulks(allocatedExtraTierBulks + 1);
        globalMutableState.setFirstFreeTierIndex(firstTierIndex);
+        globalMutableState.addDataStoreSize(tierBulkSizeInBytes);
    }

    public void msync() throws IOException {
@@ -930,6 +949,7 @@ public abstract class VanillaChronicleHash<K,
     * @see net.openhft.chronicle.bytes.MappedFile#acquireByteStore(long, MappedBytesStoreFactory)
     */
    private NativeBytesStore map(long mapSize, long mappingOffsetInFile) throws IOException {
+        mapSize = pageAlign(mapSize);
        long minFileSize = mappingOffsetInFile + mapSize;
        FileChannel fileChannel = raf.getChannel();
        if (fileChannel.size() < minFileSize) {
@@ -1593,11 +1593,11 @@ public final class ChronicleMapBuilder<K, V> implements
            VanillaChronicleMap<K, V, ?> map = wire.getValueIn().typedMarshallable();
            assert map != null;
            map.initBeforeMapping(file, raf, headerBuffer.limit(), recover);
-            long expectedFileLength = map.expectedFileSize();
-            if (!recover && expectedFileLength != file.length()) {
+            long dataStoreSize = map.globalMutableState().getDataStoreSize();
+            if (!recover && dataStoreSize > file.length()) {
                throw new IOException("The file " + file + " the map is serialized from " +
                        "has unexpected length " + file.length() + ", probably corrupted. " +
-                        "Expected length is " + expectedFileLength);
+                        "Data store size is " + dataStoreSize);
            }
            map.initTransientsFromBuilder(this);
            if (!recover) {
@@ -1683,11 +1683,6 @@ public final class ChronicleMapBuilder<K, V> implements

        try {
            VanillaChronicleMap<K, V, ?> map = newMap();
-            // TODO this method had been moved
-//            if(OS.warnOnWindows(map.sizeInBytesWithoutTiers())){
-//                throw new IllegalStateException("Windows cannot support this configuration");
-//            }
-
            BytesStore bytesStore =
                    lazyNativeBytesStoreWithFixedCapacity(map.sizeInBytesWithoutTiers());
            map.createMappedStoreAndSegments(bytesStore);
@@ -17,21 +17,30 @@
 package net.openhft.chronicle.map;

 import net.openhft.chronicle.hash.VanillaGlobalMutableState;
-import net.openhft.chronicle.values.Array;
-import net.openhft.chronicle.values.Group;
+import net.openhft.chronicle.values.*;

 interface ReplicatedGlobalMutableState extends VanillaGlobalMutableState {

-    @Group(5)
-    int getCurrentCleanupSegmentIndex();
-    void setCurrentCleanupSegmentIndex(int currentCleanupSegmentIndex);
-
    @Group(6)
+    int getCurrentCleanupSegmentIndex();
+    void setCurrentCleanupSegmentIndex(
+            @Range(min = 0, max = Integer.MAX_VALUE) int currentCleanupSegmentIndex);
+
+    @Group(7)
+    @Align(offset = 1)
+    int getModificationIteratorsCount();
+    int addModificationIteratorsCount(@Range(min = 0, max = 128) int addition);
+
+    @Group(8)
+    // Align to prohibit array filling the highest bit of the previous 2-byte block, that
+    // complicates generated code for all fields, defined in this interface.
+    @Align(offset = 2)
    @Array(length = 128)
    boolean getModificationIteratorInitAt(int index);
    void setModificationIteratorInitAt(int index, boolean init);

-    @Group(7)
-    int getModificationIteratorsCount();
-    int addModificationIteratorsCount(int addition);
+    public static void main(String[] args) {
+        System.setProperty("chronicle.values.dumpCode", "true");
+        Values.nativeClassFor(ReplicatedGlobalMutableState.class);
+    }
 }