diff --git a/README.md b/README.md index 3dd309147..007ac0f6d 100644 --- a/README.md +++ b/README.md @@ -1,13 +1,14 @@ # Chronicle Map -

Documentation: Tutorial, -Javadoc

+#### Quick links +**Documentation: Tutorial, +Javadoc** -

Community support: Issues, +**Community support: Issues, Chronicle mailing list, Stackoverflow, -Chronicle User's group

+Chronicle User's group** ### 3 min to understand everything about Chronicle Map @@ -37,7 +38,7 @@ applications. Notably trading, financial market applications. Chronicle Map has two meanings: [the language-agnostic data store](spec) and [the implementation of this data store for the JVM](src). Currently, this is the only implementation. -**From Java perspective,** Chronicle Map is a `ConcurrentMap` implementation which stores the +**From Java perspective,** `ChronicleMap` is a `ConcurrentMap` implementation which stores the entries *off-heap*, serializing/deserializing key and value objects to/from off-heap memory transparently. Chronicle Map supports - Key and value objects caching/reusing for making *zero allocations (garbage) on @@ -136,13 +137,17 @@ Chronicle Map doesn't support - [Difference between Chronicle Map 2 and 3](#difference-between-chronicle-map-2-and-3) - [Download the library](#download-the-library) - [Create a `ChronicleMap` Instance](#create-a-chroniclemap-instance) + - [In-memory Chronicle Map](#in-memory-chronicle-map) + - [Persisted Chronicle Map](#persisted-chronicle-map) + - [Recovery](#recovery) - [Key and Value Types](#key-and-value-types) - [Custom serializers](#custom-serializers) - - [Custom `CharSequence` encoding](#custom-charsequence-encoding) - - [Custom serialization checklist](#custom-serialization-checklist) - - [`ChronicleMap` instance usage patterns](#chroniclemap-instance-usage-patterns) + - [Custom `CharSequence` encoding](#custom-charsequence-encoding) + - [Custom serialization checklist](#custom-serialization-checklist) + - [`ChronicleMap` usage patterns](#chroniclemap-usage-patterns) - [Single-key queries](#single-key-queries) - [Multi-key queries](#multi-key-queries) + - [Entry checksums](#entry-checksums) - [Close `ChronicleMap`](#close-chroniclemap) - [Behaviour Customization](#behaviour-customization) - [Example - Simple logging](#example---simple-logging) @@ -153,6 +158,9 @@ Chronicle Map doesn't support Functional changes in Chronicle Map 3: + - Chronicle Map 3 has [formal data store specification](spec), that verbalizes [the guarantees + which the data store provides](spec/1-design-goals.md#guarantees-2) and gives a way to verify those + guarantees. - Added support for multi-key queries. - "Listeners" mechanism fully reworked, see the [Behaviour Customization](#behaviour-customization) section. This has a number of important consequences, most notable is: @@ -166,13 +174,16 @@ Functional changes in Chronicle Map 3: limitation is removed, though the number of entries still has to be configured on the Chronicle Map instance creation, exceeding this configured limit is possible, but discouraged. See the [Number of entries configuration](#number-of-entries-configuration) section. + - Chronicle Map 3 supports [entry checksums](#entry-checksums) that allows to detect data + corruption, hence brings additional safety. + - Chronicle Map 3 allows to [recover](#recovery) after failures and corruptions. - A number of smaller improvements and fixes. Non-functional changes: - Chronicle Map 3 requires Java version 8 or newer, while Chronicle Map 2 supports Java 7. - - Chronicle Map 3 has [specification](spec), [versioning policy](docs/versioning.md) and - [compatibility policy](docs/compatibility.md). Chronicle Map 2 doesn't have such documents. + - Chronicle Map 3 has [versioning policy](docs/versioning.md) and [compatibility policy]( + docs/compatibility.md). Chronicle Map 2 doesn't have such documents. If you use Chronicle Map 2, you might be looking for [Chronicle Map 2 Tutorial]( https://github.com/OpenHFT/Chronicle-Map/tree/2.1#contents) or [Chronicle Map 2 Javadoc]( @@ -218,9 +229,11 @@ and define the snapshot version in your pom.xml, for example: ### Create a `ChronicleMap` Instance -Creating an instance of ChronicleMap is a little more complex than just calling a constructor. +Creating an instance of `ChronicleMap` is a little more complex than just calling a constructor. To create an instance you have to use the `ChronicleMapBuilder`. +#### In-memory Chronicle Map + ``` java import net.openhft.chronicle.map.* ..... @@ -249,15 +262,18 @@ ChronicleMap cityPostalCodes = ChronicleMap .create(); ``` -This snippet creates a `ChronicleMap`, supposed to store about 50 000 city name -> postal code -mappings. It is accessible within a single Java process - the process it is created within. The -data is accessible while the process is alive. +This snippet creates an in-memory Chronicle Map store, supposed to store about 50 000 *city name -> +postal code* mappings. It is accessible within a single JVM process - the process it is created +within. The data is accessible while the process is alive, when the process is terminated, the data +is vanished. + +#### Persisted Chronicle Map Replace `.create()` calls with `.createPersistedTo(cityPostalCodesFile)`, if you want the Chronicle Map to either - - Outlive the process it was created within, e. g. to support hot Java application redeploy - - Be accessible from multiple processes on the same server + - Outlive the process it was created within, e. g. to support hot application redeploy + - Be accessible from *multiple concurrent processes* on the same server - Persist the data to disk The `cityPostalCodesFile` has to represent the same location on your server among all Java @@ -266,6 +282,26 @@ processes, wishing to access this Chronicle Map instance, e. g. The name and location of the file is entirely up to you. +Note than when you create a `ChronicleMap` instance with `.createPersistedTo(file)`, and the given +file already exists in the system, you *open a view to the existing Chronicle Map data store from +this JVM process* rather than creating a new Chronicle Map data store. I. e. it could already +contain some entries. No special action with the data is performed during such operation. If you +want to clean up corrupted entries and ensure that the data store is in correct state, see +[Recovery](#recovery) section. + + > ##### ""`ChronicleMap` instance" vs "Chronicle Map data store" + > + > In this tutorial, *`ChronicleMap` instance* (or simply *`ChronicleMap`*) term is used to refer to + > *on-heap object*, providing access to a *Chronicle Map data store* (or *Chronicle Map key-value + > store*, or *Chronicle Map store*, or simply *Chronicle Map*, with space between two words in + > contrast to `ChronicleMap`), which could be purely in-memory, or persisted to disk. Currently + > Java implementation doesn't allow to create multiple accessor `ChronicleMap` objects to a single + > *in-memory Chronicle Map store*, i. e. there is always a one-to-one correspondence, that may lead + > to confusion. *Persisted Chronicle Map store*, however, allows to create multiple accessor + > *`ChronicleMap` instances* either within a single JVM process (though [it is not recommended to + > do that](#single-chroniclemap-instance-per-jvm)), or from concurrent JVM processes, that is + > perfectly OK. + When no processes access the file, it could be freely moved to another location in the system, and even to another server, even running different operating system, opened from another location and you will observe the same data. @@ -297,11 +333,86 @@ though the `ChronicleMapBuilder` instance.** --- -**Single `ChronicleMap` instance per JVM.** If you want to access the Chronicle Map instance -concurrently within the Java process, you should *not* create a separate `ChronicleMap` instance per + +**Single `ChronicleMap` instance per JVM.** If you want to access a Chronicle Map data store +concurrently within a Java process, you should *not* create a separate `ChronicleMap` instance per thread. Within the JVM environment, `ChronicleMap` instance *is* a `ConcurrentMap`, and could be accessed concurrently the same way as e. g. `ConcurrentHashMap`. +#### Recovery + +If a process, accessing a persisted Chronicle Map, terminated abnormally: crashed, `SIGKILL`ed, or +terminated because the host operating system crashed, or the machine lost power, the Chronicle Map +might remain in an inaccessible or corrupted state. When the Chronicle Map is opened next time from +another process, it should be done via `.recoverPersistedTo()` method in `ChronicleMapBuilder`. +Unlike `createPersistedTo()`, this method scans all memory of Chronicle Map store for +inconsistencies, if some found, it cleans them up. + +*`.recoverPersistedTo()` needs to access the Chronicle Map exclusively. If a concurrent process is +accessing the Chronicle Map while another process is attempting to perform recovery, result of +operations on the accessing process side, and results of recovery are unspecified. The data could be +corrupted further. You* must *ensure no other process is accessing the Chronicle Map store when +calling for `.recoverPersistedTo()` on this store.* + +Example: + +```java +ChronicleMap cityPostalCodes = ChronicleMap + .of(CharSequence.class, PostalCodeRange.class) + .averageKey("Amsterdam") + .entries(50_000) + .recoverPersistedTo(cityPostalCodesFile, true); +// or +ChronicleMap cityPostalCodes = ChronicleMap + .of(CharSequence.class, PostalCodeRange.class) + // assuming ChronicleMapBuilder configurations at the moment of + // cityPostalCodes Chronicle Map creation are not known + .recoverPersistedTo(cityPostalCodesFile, false); +``` + +The second parameter in `recoverPersistedTo()` method is called `sameBuilderConfig`, it should be +`true` if `ChronicleMapBuilder` is configured in exactly the same way, as when the Chronicle Map was +created, and using the same version of Chronicle Map library, or `false`, if initial configurations +are not known, of current version of Chronicle Map library differs from the version, used to create +this Chronicle Map initially. + +If `sameBuilderConfig` is `true`, `recoverPersistedTo()` checks that the recovered Chronicle Map's +header memory (containing serialized configurations) is not corrupted, because it "knows" all the +right configurations and what should be written to the header. If the header is corrupted, it +recovers it. + +If `sameBuilderConfig` is `false`, `recoverPersistedTo()` relies on the configurations written to +the Chronicle Map's header, assuming it is not corrupted. If it is corrupted, at best it will lead +to a runtime exception thrown from `recoverPersistedTo()` without touching the data, stored in the +Chronicle Map (but this effectively means inability to recover the Chronicle Map), at worst it will +proceed "successfully" but will corrupt the Chronicle Map further. + +However, the subject header memory is never updated on ordinary operations with Chronicle Map, so it +couldn't be corrupted if an accessing process crashed, or the operating system crashed, or even the +machine lost power. Only hardware memory or disk corruption or a bug in the file system could lead +to Chronicle Map header memory corruption. + +`.recoverPersistedTo()` is harmless if the previous process accessing the Chronicle Map terminated +normally, however this is a computationally expensive procedure that should generally be avoided. + +To share configuration, Chronicle Map creation and recovery code you could use +`.createOrRecoverPersistedTo(persistenceFile)` method in `ChronicleMapBuilder`, which is equivalent +to `createPersistedTo(persistenceFile)` call, if the persistence file doesn't yet exist, and to +`recoverPersistedTo(persistenceFile, true)`, if the file already exists, e. g.: + +```java +ChronicleMap cityPostalCodes = ChronicleMap + .of(CharSequence.class, PostalCodeRange.class) + .averageKey("Amsterdam") + .entries(50_000) + .createOrRecoverPersistedTo(cityPostalCodesFile); +``` + +If the Chronicle Map is configured to store entry checksums along with entries, recovery procedure +checks for each entry that the checksums is correct, otherwise it assumes the entry is corrupted and +deletes it from the Chronicle Map. If checksums are to stored, recovery procedure cannot guarantee +correctness of entry data. See [Entry checksums](#entry-checksums) section for more information. + #### Key and Value Types Either key or value type of `ChronicleMap` could be: @@ -322,10 +433,10 @@ Either key or value type of `ChronicleMap` could be: - Types supported out of the box, but not particularly efficiently. You might want to implement more efficient [custom serializers](#custom-serializers) for them: - - Any class implementing `java.io.Externalizable`. The implementation class should have a public - no-arg constructor. - - Any type implementing `java.io.Serializable`, including boxed primitive types (except listed - above) and array types + - Any class implementing `java.io.Externalizable`. The implementation class should have a public + no-arg constructor. + - Any type implementing `java.io.Serializable`, including boxed primitive types (except listed + above) and array types - Any other type, if [custom serializers](#custom-serializers) are provided. @@ -1198,7 +1309,7 @@ class. `ChronicleMapBuilder` by `keyMarshallers()`, `keyReaderAndDataAccess()`, `valueMarshallers()` or `valueReaderAndDataAccess()` methods. -### `ChronicleMap` instance usage patterns +### `ChronicleMap` usage patterns #### Single-key queries @@ -1831,3 +1942,68 @@ System.out.println(map2.get("1")); } } ``` + +### Entry checksums + +Chronicle Map 3 is able to store entry checksums along with entries. With entry checksums it is +possible to identify partially written entries (in case of operating system or power failure while) +and corrupted entries (in case of hardware memory or disk corruption) and clean them up during +[recovery](#recovery) procedure. + +Entry checksums are 32-bit numbers, computed by a hash function with good avalanche effect. +Theoretically there is still about a one-billionth chance that after entry corruption it passes the +sum check. + +By default, entry checksums are ON if the Chronicle Map is persisted to disk (i. e. created via +`createPersistedTo()` method), and OFF if the Chronicle Map is purely in-memory. Storing checksums +for a purely in-memory Chronicle Map hardly makes any practical sense, but you might want to disable +storing checksums for a persisted Chronicle Map by calling `.checksumEntries(false)` on the +`ChronicleMapBuilder` used to create a map. It makes sense if you don't need extra safety checksums +provide. + +Entry checksums are computed automatically when an entry is inserted into a Chronicle Map, and +re-computed automatically on operations which update the whole value, e. g. `map.put()`, +`map.replace()`, `map.compute()`, `mapEntry.doReplaceValue()` (See `MapEntry` interface in +[Javadocs](http://www.javadoc.io/doc/net.openhft/chronicle-map/). But if you update values directly, +bypassing Chronicle Map logic, keeping entry checksum up-to-date is also your responsibility. + +It is strongly recommended to update off-heap memory of values directly only within a +[context](#working-with-an-entry-within-a-context-section) and update or write lock held. Within a +context, you are provided with an *entry* object of `MapEntry` type. To re-compute entry checksum +manually, cast that object to `ChecksumEntry` type and call `.updateChecksum()` method on it: + +```java +try (ChronicleMap map = ChronicleMap + .of(Integer.class, LongValue.class) + .entries(1) + // Entry checksums make sense only for persisted Chronicle Maps, and are ON by + // default for such maps + .createPersistedTo(file)) { + + LongValue value = Values.newHeapInstance(LongValue.class); + value.setValue(42); + map.put(1, value); + + try (ExternalMapQueryContext c = map.queryContext(1)) { + // Update lock required for calling ChecksumEntry.checkSum() + c.updateLock().lock(); + MapEntry entry = c.entry(); + Assert.assertNotNull(entry); + ChecksumEntry checksumEntry = (ChecksumEntry) entry; + Assert.assertTrue(checksumEntry.checkSum()); + + // to access off-heap bytes, should call value().getUsing() with Native value + // provided. Simple get() return Heap value by default + LongValue nativeValue = + entry.value().getUsing(Values.newNativeReference(LongValue.class)); + // This value bytes update bypass Chronicle Map internals, so checksum is not + // updated automatically + nativeValue.setValue(43); + Assert.assertFalse(checksumEntry.checkSum()); + + // Restore correct checksum + checksumEntry.updateChecksum(); + Assert.assertTrue(checksumEntry.checkSum()); + } +} +``` diff --git a/pom.xml b/pom.xml index 344d65771..c60abb41e 100755 --- a/pom.xml +++ b/pom.xml @@ -255,6 +255,8 @@ net/openhft/chronicle/hash/ChronicleHashBuilderPrivateAPI.java net/openhft/chronicle/hash/VanillaGlobalMutableState.java + net/openhft/chronicle/hash/ReplicatedHashSegmentContext.java + net/openhft/chronicle/map/MapDiagnostics.java net/openhft/chronicle/map/VanillaChronicleMap.java net/openhft/chronicle/map/Replica.java net/openhft/chronicle/map/ReplicatedChronicleMap.java diff --git a/spec/1-design-goals.md b/spec/1-design-goals.md index 2b09118c0..763062ead 100644 --- a/spec/1-design-goals.md +++ b/spec/1-design-goals.md @@ -106,7 +106,8 @@ Chronicle Map first, which identifies and purges corrupted entries from the Chro Therefore, *some entries updated shortly before the failure could be lost.* > The aforementioned *special recovery procedure* is `ChronicleHashBuilder.recoverPersistedTo()` - > or `ChronicleHashInstanceBuilder.recover()` in the reference Java implementation. + > or `ChronicleHashInstanceBuilder.recover()` in the reference Java implementation. See + > [Recovery](../README.md#recovery) section in the Chronicle Map tutorial for more information. ## Goals diff --git a/src/main/java/net/openhft/chronicle/hash/AbstractData.java b/src/main/java/net/openhft/chronicle/hash/AbstractData.java index de2504d90..ea19ece5e 100644 --- a/src/main/java/net/openhft/chronicle/hash/AbstractData.java +++ b/src/main/java/net/openhft/chronicle/hash/AbstractData.java @@ -16,8 +16,6 @@ package net.openhft.chronicle.hash; -import net.openhft.chronicle.bytes.RandomDataInput; - /** * Defines reasonable defaults for {@code Data}'s {@code equals()}, {@code hashCode()} and * {@code toString()}. They should be default implementations in the {@code Data} interface itself, diff --git a/src/main/java/net/openhft/chronicle/hash/ChecksumEntry.java b/src/main/java/net/openhft/chronicle/hash/ChecksumEntry.java index b3b652c5e..2d2929a15 100644 --- a/src/main/java/net/openhft/chronicle/hash/ChecksumEntry.java +++ b/src/main/java/net/openhft/chronicle/hash/ChecksumEntry.java @@ -18,10 +18,16 @@ package net.openhft.chronicle.hash; import net.openhft.chronicle.map.MapAbsentEntry; import net.openhft.chronicle.map.MapEntry; +import net.openhft.chronicle.set.SetEntry; /** * Abstracts entries of hash containers, created by {@link ChronicleHashBuilder}s - * with {@link ChronicleHashBuilder#checksumEntries(boolean)} configured to {@code true}. + * with {@link ChronicleHashBuilder#checksumEntries(boolean)} configured to {@code true}. There is + * no method that returns {@code ChecksumEntry}, {@link MapEntry} or {@link SetEntry} could be + * casted to {@code ChecksumEntry} to access it's methods. + * + *

See Entry checksums + * section in the Chronicle Map tutorial for usage examples of this interface. */ public interface ChecksumEntry { diff --git a/src/main/java/net/openhft/chronicle/hash/ChronicleHash.java b/src/main/java/net/openhft/chronicle/hash/ChronicleHash.java index 67a8279c3..d093925e2 100644 --- a/src/main/java/net/openhft/chronicle/hash/ChronicleHash.java +++ b/src/main/java/net/openhft/chronicle/hash/ChronicleHash.java @@ -29,8 +29,7 @@ import java.util.function.Consumer; import java.util.function.Predicate; /** - * This interface defines common {@link ChronicleMap} and {@link ChronicleSet}, related to off-heap - * memory management and file-mapping. Not usable by itself. + * Common base interface for {@link ChronicleMap} and {@link ChronicleSet}. */ public interface ChronicleHash, SC extends HashSegmentContext, EQC extends ExternalHashQueryContext> extends Closeable { @@ -45,6 +44,11 @@ public interface ChronicleHash, SC extends HashSegment */ File file(); + /** + * Returns the number of entries in this store. + * + * @return the number of entries in this store + */ long longSize(); /** @@ -53,13 +57,16 @@ public interface ChronicleHash, SC extends HashSegment Class keyClass(); /** - * Returns the context to perform arbitrary operations with the given key in this map. - * Conventionally, try-with-resources block should wrap the returned context:

{@code
+     * Returns a context to perform arbitrary operations with the given key in this store.
+     * Conventionally, should be used in a try-with-resources block: 
{@code
      * try (ExternalHashQueryContext q = hash.queryContext(key)) {
      *     // ... do something
      * }}
- * See {@link HashQueryContext} and {@link MapMethods} for a lot of inspiration about using this - * functionality. + * + *

See documentation to {@link HashQueryContext} interface and methods in {@link MapMethods} + * interface for examples of using contexts. Also see + * Working with an entry within a context section in the Chronicle Map tutorial. * * @param key the queried key * @return the context to perform operations with the key @@ -72,9 +79,16 @@ public interface ChronicleHash, SC extends HashSegment EQC queryContext(K key); /** - * Equivalent to {@link #queryContext(Object)}, but accepts {@code Data} instead of key as - * an object. Useful, when you already have {@code Data}, calling this method instead of {@link - * #queryContext(Object)} might help to avoid unnecessary deserialization. + * Returns a context to perform arbitrary operations with the given key, provided in + * {@link Data} form. Equivalent to {@link #queryContext(Object)}, but accepts {@code Data} + * instead of object key. This method is useful, when you already have {@code Data}, calling + * this method instead of {@link #queryContext(Object)} might help to avoid unnecessary + * deserialization. + * + *

See documentation to {@link HashQueryContext} interface and methods in {@link MapMethods} + * interface for examples of using contexts. Also see + * Working with an entry within a context section in the Chronicle Map tutorial. * * @param key the queried key as {@code Data} * @return the context to perform operations with the key @@ -82,19 +96,19 @@ public interface ChronicleHash, SC extends HashSegment @NotNull EQC queryContext(Data key); /** - * Returns the context to perform arbitrary operations with the given key, provided in the + * Returns a context to perform arbitrary operations with the given key, provided in the * serialized form. See {@link #queryContext(Object)} and {@link #queryContext(Data)} for more * information on contexts semantics and usage patterns. * * @param keyBytes the bytes store with the key bytes to query * @param offset actual offset of the key bytes within the given BytesStore - * @param size length of the key bytes sequence + * @param size length of the key bytes sequence within the given BytesStore * @return the context to perform operations with the key */ @NotNull EQC queryContext(BytesStore keyBytes, long offset, long size); /** - * Returns the context of the segment with the given index. Segments are indexed from 0 to + * Returns a context of the segment with the given index. Segments are indexed from 0 to * {@link #segments()}{@code - 1}. * * @see HashSegmentContext @@ -150,10 +164,6 @@ public interface ChronicleHash, SC extends HashSegment * collection. This could be a problem if, for example, you target rare garbage collections, * but load and drop {@code ChronicleHash}es regularly. * - *

- * TODO what about commit guarantees, when ChronicleMap is used with memory-mapped files, if - * {@code ChronicleMap}/{@code ChronicleSet} closed/not? - * *

After this method call behaviour of all methods of {@code ChronicleMap} * or {@code ChronicleSet} is undefined. Any method call on the map might throw * any exception, or even JVM crash. Shortly speaking, don't call use map closing. diff --git a/src/main/java/net/openhft/chronicle/hash/ChronicleHashBuilder.java b/src/main/java/net/openhft/chronicle/hash/ChronicleHashBuilder.java index 513518542..62d67f4c1 100755 --- a/src/main/java/net/openhft/chronicle/hash/ChronicleHashBuilder.java +++ b/src/main/java/net/openhft/chronicle/hash/ChronicleHashBuilder.java @@ -33,8 +33,8 @@ import java.util.HashSet; import java.util.concurrent.ConcurrentHashMap; /** - * This interface defines the meaning of configurations, common to {@link ChronicleMapBuilder} and - * {@link ChronicleSetBuilder}, i. e. Chronicle hash container configurations. + * Base interface for {@link ChronicleMapBuilder} and {@link ChronicleSetBuilder}, i. e. defines + * Chronicle hash container configurations. * *

{@code ChronicleHashBuilder} is mutable. Configuration methods mutate the builder and return * the builder itself back to support chaining pattern, rather than the builder copies with diff --git a/src/main/java/net/openhft/chronicle/hash/Data.java b/src/main/java/net/openhft/chronicle/hash/Data.java index bc7c5de2b..4df7349e4 100644 --- a/src/main/java/net/openhft/chronicle/hash/Data.java +++ b/src/main/java/net/openhft/chronicle/hash/Data.java @@ -17,45 +17,63 @@ package net.openhft.chronicle.hash; import net.openhft.chronicle.algo.hashing.LongHashFunction; +import net.openhft.chronicle.bytes.BytesStore; import net.openhft.chronicle.bytes.BytesUtil; import net.openhft.chronicle.bytes.RandomDataInput; import net.openhft.chronicle.bytes.RandomDataOutput; +import net.openhft.chronicle.map.ChronicleMap; +import net.openhft.chronicle.map.MapEntry; +import net.openhft.chronicle.map.MapQueryContext; +import net.openhft.chronicle.set.ChronicleSet; import org.jetbrains.annotations.Nullable; import static net.openhft.chronicle.algo.bytes.Access.checkedRandomDataInputAccess; /** - * Dual bytes/object access to keys/values/elements. + * Dual bytes/object access to keys or values (for {@link ChronicleMap}) and elements (for {@link + * ChronicleSet}) throughout the Chronicle Map library. * *

Bytes access: {@link #bytes()} + {@link #offset()} + {@link #size()}. * *

Object access: {@link #get()}. * - *

In most cases, each particular value wraps either some object or some bytes. Object + *

In most cases, each particular {@code Data} wraps either some object or some bytes. Object * is marshalled to bytes lazily on demand, and bytes are lazily deserialized to object, - * accordingly. + * accordingly. * * @param type of the accessed objects */ public interface Data { /** - * Returns the accessor object to the {@code Data}'s bytes. For safety, this interface returns - * read-only object, because this object could expose bytes source that must be immutable, e. g. - * an array behind {@code String} object. + * Returns the accessor object to the {@code Data}'s bytes. + * + *

If this {@code Data} wraps some bytes, this method just returns a reference to that bytes. + * + *

If this {@code Data} wraps an object, this method performs serialization internally and + * returns a reference to the output buffer, caching the result for subsequent calls of this + * method. + * + *

For safety, this interface returns read-only object, because it could expose bytes source + * that must be immutable, e. g. a `char[]` array behind a {@code String}. But in cases when the + * {@code Data} instance wraps off-heap bytes, e. g. {@link MapEntry#value()}, it is allowed to + * cast the object, returned from this method, to {@link BytesStore}, and write into the + * off-heap memory. You should only ensure that current context (e. g. {@link MapQueryContext}) + * is locked exclusively, in order to avoid data races. */ RandomDataInput bytes(); /** - * Returns the offset to the {@code Data}'s bytes sequence, within the {@link #bytes()} object. - * For example, the first byte of the bytes representation of this {@code Data} instance is - * {@code data.bytes().readByte(data.offset())}. + * Returns the offset to the {@code Data}'s bytes sequence within the {@link RandomDataInput}, + * returned from {@link #bytes()} method. For example, the first byte of the bytes + * representation of some {@code data} is {@code data.bytes().readByte(data.offset())}. */ long offset(); /** * Returns the size of this {@code Data}'s bytes sequence. It spans from {@link #offset()} to - * {@code offset() + size() - 1} bytes within the {@link #bytes()} object. + * {@code offset() + size() - 1} bytes within the {@link RandomDataInput}, returned from + * {@link #bytes()} method. */ long size(); @@ -76,7 +94,7 @@ public interface Data { * *

Default implementation compares {@link #bytes()} of this {@code Data}, but custom * implementation may only check if {@linkplain #get() object} of this {@code Data} would - * be serialized to the same bytes sequence, if this {@code Data} is object-based and obtaining + * be serialized to the same bytes sequence, if this {@code Data} wraps an object and obtaining * {@link #bytes()} requires serialization internally. * * @param source the bytes source, to compare this {@code Data}'s bytes with @@ -106,16 +124,25 @@ public interface Data { } /** - * Returns "cached" object, generally not eligible for using outside some context, or a block, - * synchronized with locks, or lambda, etc. - * - *

If the {@code Data} is object wrapper -- this method just returns this object. + * Returns object view of this {@code Data}. + * + *

If this {@code Data} wraps some object, this method just returns that object. + * + *

If this {@code Data} wraps some bytes, this method performs deserialization internally + * and returns the resulting on-heap object, caching it for subsequent calls of this method. The + * returned object could be reused, therefore it is generally disallowed to use the + * object, returned from this method, outside some context, or a block, synchronized with + * locks, or lambda, etc., which provided the access to this {@code Data} instance. */ T get(); /** - * Reads the object from the value's bytes, trying to reuse the given object (might be {@code - * null}). + * Deserialize and return an object from the {@code Data}'s bytes, trying to reuse the given + * {@code using} object (might be {@code null}). This method either returns the given {@code + * using} object back (if reuse is possible) or creates a new object, rather than some + * internally cached and reused one, therefore it is always allowed to use the object, + * returned from this method, outside some context, or a block, synchronized with locks, + * or lambda, etc., which provided the access to this {@code Data} instance. */ T getUsing(@Nullable T using); diff --git a/src/main/java/net/openhft/chronicle/hash/locks/package-info.java b/src/main/java/net/openhft/chronicle/hash/locks/package-info.java new file mode 100644 index 000000000..be59753b7 --- /dev/null +++ b/src/main/java/net/openhft/chronicle/hash/locks/package-info.java @@ -0,0 +1,20 @@ +/* + * Copyright (C) 2015 higherfrequencytrading.com + * + * This program is free software: you can redistribute it and/or modify + * it under the terms of the GNU Lesser General Public License as published by + * the Free Software Foundation, either version 3 of the License. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public License + * along with this program. If not, see . + */ + +/** + * Contains abstractions of inter-process locks, used in Chronicle Map. + */ +package net.openhft.chronicle.hash.locks; \ No newline at end of file diff --git a/src/main/java/net/openhft/chronicle/hash/serialization/ListMarshaller.java b/src/main/java/net/openhft/chronicle/hash/serialization/ListMarshaller.java index 79bb08584..54fc5987e 100644 --- a/src/main/java/net/openhft/chronicle/hash/serialization/ListMarshaller.java +++ b/src/main/java/net/openhft/chronicle/hash/serialization/ListMarshaller.java @@ -36,14 +36,12 @@ import static net.openhft.chronicle.hash.serialization.StatefulCopyable.copyIfNe * README section. * *

Usage:

{@code
- * ListMarshaller valueMarshaller = new ListMarshaller<>(
- *     IntegerMarshaller.INSTANCE, IntegerMarshaller.INSTANCE);
  * ChronicleMap> regNumbers = ChronicleMap
  *     .of(String.class, (Class>) (Class) List.class)
- *     .entries(10_000)
  *     .averageKey("John Smith")
- *     .valueMarshaller(valueMarshaller)
+ *     .valueMarshaller(ListMarshaller.of(IntegerMarshaller.INSTANCE))
  *     .averageValue(ImmutableList.of(1, 2, 3))
+ *     .entries(10_000)
  *     .create();
  * }
* diff --git a/src/main/java/net/openhft/chronicle/hash/serialization/SetMarshaller.java b/src/main/java/net/openhft/chronicle/hash/serialization/SetMarshaller.java index 7b3d0a4e3..4094763a0 100644 --- a/src/main/java/net/openhft/chronicle/hash/serialization/SetMarshaller.java +++ b/src/main/java/net/openhft/chronicle/hash/serialization/SetMarshaller.java @@ -38,14 +38,14 @@ import static net.openhft.chronicle.hash.serialization.StatefulCopyable.copyIfNe * README section. * *

Usage:

{@code
- * SetMarshaller valueMarshaller = new SetMarshaller<>(
+ * SetMarshaller valueMarshaller = SetMarshaller.of(
  *     new StringBytesReader(), CharSequenceBytesWriter.INSTANCE);
  * ChronicleMap> map = ChronicleMap
  *     .of(String.class, (Class>) (Class) Set.class)
- *     .entries(10_000)
  *     .averageKey("fruits")
  *     .valueMarshaller(valueMarshaller)
  *     .averageValue(ImmutableSet.of("apples", "bananas", "grapes"))
+ *     .entries(10_000)
  *     .create();
  * }
* diff --git a/src/main/java/net/openhft/chronicle/hash/serialization/package-info.java b/src/main/java/net/openhft/chronicle/hash/serialization/package-info.java index 30531c96b..0db3acd01 100644 --- a/src/main/java/net/openhft/chronicle/hash/serialization/package-info.java +++ b/src/main/java/net/openhft/chronicle/hash/serialization/package-info.java @@ -15,7 +15,7 @@ */ /** - * The package contains interfaces for serializing objects between Java heap and {@link + * Contains interfaces for serializing objects between Java heap and {@link * net.openhft.chronicle.bytes.Bytes} or {@link net.openhft.chronicle.bytes.BytesStore}, used by * Chronicle Map to store objects off-heap, and read them back from off-heap memory to on-heap * objects. diff --git a/src/main/java/net/openhft/chronicle/map/ChronicleMap.java b/src/main/java/net/openhft/chronicle/map/ChronicleMap.java index cf8084bb8..529da29a5 100755 --- a/src/main/java/net/openhft/chronicle/map/ChronicleMap.java +++ b/src/main/java/net/openhft/chronicle/map/ChronicleMap.java @@ -28,7 +28,8 @@ import java.util.Map; import java.util.concurrent.ConcurrentMap; /** - * Extension of {@link ConcurrentMap} interface, stores the data off-heap. + * {@code ChronicleMap} provides concurrent access to a Chronicle Map key-value store from a + * JVM process. * *

For information on

  • how to construct a {@code ChronicleMap}
  • {@code * ChronicleMap} flavors and properties
  • available configurations
see {@link diff --git a/src/main/java/net/openhft/chronicle/map/ChronicleMapBuilder.java b/src/main/java/net/openhft/chronicle/map/ChronicleMapBuilder.java index 4796fbf9b..f44b62d3e 100755 --- a/src/main/java/net/openhft/chronicle/map/ChronicleMapBuilder.java +++ b/src/main/java/net/openhft/chronicle/map/ChronicleMapBuilder.java @@ -1777,7 +1777,7 @@ public final class ChronicleMapBuilder implements /** * Inject your SPI code around basic {@code ChronicleMap}'s operations with entries: - * removing entries, replacing the entries' value and inserting the new entry. + * removing entries, replacing entries' value and inserting new entries. * *

This affects behaviour of ordinary map.put(), map.remove(), etc. calls, as well as removes * and replacing values during iterations, remote map calls and diff --git a/src/main/java/net/openhft/chronicle/map/MapEntryOperations.java b/src/main/java/net/openhft/chronicle/map/MapEntryOperations.java index 0f4403272..9970066e0 100644 --- a/src/main/java/net/openhft/chronicle/map/MapEntryOperations.java +++ b/src/main/java/net/openhft/chronicle/map/MapEntryOperations.java @@ -108,7 +108,9 @@ public interface MapEntryOperations { * @implNote default implementation calls {@link MapEntry#doRemove()} on the given entry * and returns {@code null}. * - * @param entry the entry to remove + * @param entry the entry to remove + * @return result of operation, understandable by higher-level SPIs, e. g. custom + * {@link MapMethods} implementation * @throws IllegalStateException if some locking/state conditions required to perform remove * operation are not met * @throws RuntimeException if removal was unconditionally unsuccessful due to any reason @@ -125,6 +127,8 @@ public interface MapEntryOperations { * entry.doReplaceValue(newValue)} and returns {@code null}. * * @param entry the entry to replace the value in + * @return result of operation, understandable by higher-level SPIs, e. g. custom + * {@link MapMethods} implementation * @throws IllegalStateException if some locking/state conditions required to perform replace * operation are not met * @throws RuntimeException if value replacement was unconditionally unsuccessful due @@ -142,6 +146,8 @@ public interface MapEntryOperations { * @implNote default implementation calls {@link MapAbsentEntry#doInsert(Data) * absentEntry.doInsert(value)} and returns {@code null}. * + * @return result of operation, understandable by higher-level SPIs, e. g. custom + * {@link MapMethods} implementation * @throws IllegalStateException if some locking/state conditions required to perform insertion * operation are not met * @throws RuntimeException if insertion was unconditionally unsuccessful due to any reason diff --git a/src/main/java/net/openhft/chronicle/map/MapMethods.java b/src/main/java/net/openhft/chronicle/map/MapMethods.java index 8c954bf7b..968f0392f 100644 --- a/src/main/java/net/openhft/chronicle/map/MapMethods.java +++ b/src/main/java/net/openhft/chronicle/map/MapMethods.java @@ -32,6 +32,13 @@ import static net.openhft.chronicle.map.MapMethodsSupport.tryReturnCurrentValueI */ public interface MapMethods { + /** + * Backing {@link ChronicleMap#containsKey(Object)} method. + * + * @implNote the default implementation is equivalent to

{@code
+     * return q.entry() != null;
+     * }
+ */ default boolean containsKey(MapQueryContext q) { return q.entry() != null; } diff --git a/src/main/java/net/openhft/chronicle/map/ReturnValue.java b/src/main/java/net/openhft/chronicle/map/ReturnValue.java index 807c0a7b2..6b25960e9 100644 --- a/src/main/java/net/openhft/chronicle/map/ReturnValue.java +++ b/src/main/java/net/openhft/chronicle/map/ReturnValue.java @@ -19,6 +19,27 @@ package net.openhft.chronicle.map; import net.openhft.chronicle.hash.Data; import org.jetbrains.annotations.NotNull; +/** + * Abstracts returning a value from a query to a {@link ChronicleMap}, used in {@link MapMethods}. + * This interface is not supposed to be implemented in user code. + * + * @param the type of values in the {@code ChronicleMap} + */ public interface ReturnValue { + /** + * Calling this method on a {@code ReturnValue} object, provided as an argument in a method from + * {@link MapMethods}, designates that the {@code ChronicleMap}'s method (backed by this + * {@code MapMethods}'s method) should return the given {@code value}. + * + *

It is not allowed to call {@code returnValue()} twice during a single {@code MapMethods}'s + * method call. + * + *

Not calling {@code returnValue()} during {@code MapMethods}'s method call will make the + * backed {@code ChronicleMap}'s method to return {@code null}. + * + * @param value a {@code Data} object wrapping a value, which should be returned from a + * {@code ChronicleMap}'s method, backed by the {@code MapMethods}'s method, to + * which this {@code ReturnValue} instance was passed as an argument + */ void returnValue(@NotNull Data value); } diff --git a/src/main/java/net/openhft/chronicle/map/package-info.java b/src/main/java/net/openhft/chronicle/map/package-info.java new file mode 100644 index 000000000..e83c5da63 --- /dev/null +++ b/src/main/java/net/openhft/chronicle/map/package-info.java @@ -0,0 +1,21 @@ +/* + * Copyright (C) 2015 higherfrequencytrading.com + * + * This program is free software: you can redistribute it and/or modify + * it under the terms of the GNU Lesser General Public License as published by + * the Free Software Foundation, either version 3 of the License. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public License + * along with this program. If not, see . + */ + +/** + * Contains {@link net.openhft.chronicle.map.ChronicleMap} interface, {@code ChronicleMap} context + * interfaces, {@link net.openhft.chronicle.map.ChronicleMapBuilder} class and related stuff. + */ +package net.openhft.chronicle.map; \ No newline at end of file diff --git a/src/main/java/net/openhft/chronicle/set/ChronicleSet.java b/src/main/java/net/openhft/chronicle/set/ChronicleSet.java index 974afe6cf..ebdf3cf61 100644 --- a/src/main/java/net/openhft/chronicle/set/ChronicleSet.java +++ b/src/main/java/net/openhft/chronicle/set/ChronicleSet.java @@ -20,10 +20,24 @@ import net.openhft.chronicle.hash.ChronicleHash; import java.util.Set; +/** + * {@code ChronicleSet} provides concurrent access to a Chronicle Map key-value store with + * zero-sized values from a JVM process, wrapped as an extension of {@link Set} interface. + * + * @param the set key type + * @see net.openhft.chronicle.map.ChronicleMap + */ public interface ChronicleSet extends Set, ChronicleHash, SetSegmentContext, ExternalSetQueryContext> { + /** + * Delegates to {@link ChronicleSetBuilder#of(Class)} for convenience. + * + * @param keyClass class of the key type of the {@code ChronicleSet} to create + * @param the key type of the {@code ChronicleSet} to create + * @return a new {@code ChronicleSetBuilder} for the given key class + */ static ChronicleSetBuilder of(Class keyClass) { return ChronicleSetBuilder.of(keyClass); } diff --git a/src/main/java/net/openhft/chronicle/set/ChronicleSetBuilder.java b/src/main/java/net/openhft/chronicle/set/ChronicleSetBuilder.java index dca82d590..c49904ca5 100755 --- a/src/main/java/net/openhft/chronicle/set/ChronicleSetBuilder.java +++ b/src/main/java/net/openhft/chronicle/set/ChronicleSetBuilder.java @@ -55,6 +55,15 @@ public final class ChronicleSetBuilder chronicleMapBuilder.privateAPI()); } + /** + * Returns a new {@code ChronicleSetBuilder} instance which is able to {@linkplain #create() + * create} sets with the specified key class. + * + * @param keyClass class object used to infer key type and discover it's properties via + * reflection + * @param key type of the sets, created by the returned builder + * @return a new builder for the given key class + */ public static ChronicleSetBuilder of(Class keyClass) { return new ChronicleSetBuilder<>(keyClass); } @@ -250,6 +259,14 @@ public final class ChronicleSetBuilder return this; } + /** + * Inject your SPI code around basic {@code ChronicleSet}'s operations with entries: + * removing entries and inserting new entries. + * + *

This affects behaviour of ordinary set.add(), set.remove(), calls, as well as removes + * during iterations, updates during remote calls and + * internal replication operations. + */ public ChronicleSetBuilder entryOperations(SetEntryOperations entryOperations) { chronicleMapBuilder.entryOperations(new MapEntryOperations() { @Override diff --git a/src/main/java/net/openhft/chronicle/set/SetEntryOperations.java b/src/main/java/net/openhft/chronicle/set/SetEntryOperations.java index 05dc6013f..468605e13 100644 --- a/src/main/java/net/openhft/chronicle/set/SetEntryOperations.java +++ b/src/main/java/net/openhft/chronicle/set/SetEntryOperations.java @@ -18,15 +18,47 @@ package net.openhft.chronicle.set; import org.jetbrains.annotations.NotNull; +/** + * SPI interface fro customizing "low-level" modification operations on {@link ChronicleSet} + * entries. + * + * @param the set key type + * @param methods return type, used for communication between lower- and higher-level SPI + * @see ChronicleSetBuilder#entryOperations(SetEntryOperations) + */ public interface SetEntryOperations { - + + /** + * Removes the given entry from the set. + * + * @implNote default implementation calls {@link SetEntry#doRemove()} on the given entry and + * returns {@code null}. + * + * @param entry the entry to remove + * @return result of operation, understandable by higher-level SPIs + * @throws IllegalStateException if some locking/state conditions required to perform remove + * operation are not met + * @throws RuntimeException if removal was unconditionally unsuccessful due to any reason + */ default R remove(@NotNull SetEntry entry) { entry.doRemove(); return null; } - - default R insert(@NotNull SetAbsentEntry entry) { - entry.doInsert(); + + /** + * Inserts the new entry into the set, of {@link SetAbsentEntry#absentKey() the key} from + * the given insertion context (absentEntry). + * + * @implNote default implementation calls {@link SetAbsentEntry#doInsert()} and returns + * {@code null}. + * + * @return result of operation, understandable by higher-level SPIs + * @throws IllegalStateException if some locking/state conditions required to perform insertion + * operation are not met + * @throws RuntimeException if insertion was unconditionally unsuccessful due to any reason + */ + default R insert(@NotNull SetAbsentEntry absentEntry) { + absentEntry.doInsert(); return null; } } diff --git a/src/main/java/net/openhft/chronicle/set/package-info.java b/src/main/java/net/openhft/chronicle/set/package-info.java index 15f493723..48eebcad8 100644 --- a/src/main/java/net/openhft/chronicle/set/package-info.java +++ b/src/main/java/net/openhft/chronicle/set/package-info.java @@ -15,7 +15,7 @@ */ /** - * Contains {@link net.openhft.chronicle.set.ChronicleSet} interface and - * {@link net.openhft.chronicle.set.ChronicleSetBuilder} class. + * Contains {@link net.openhft.chronicle.set.ChronicleSet} interface, {@code ChronicleSet} context + * interfaces and {@link net.openhft.chronicle.set.ChronicleSetBuilder} class. */ package net.openhft.chronicle.set; \ No newline at end of file diff --git a/src/test/java/net/openhft/chronicle/map/fromdocs/ChecksumEntryTest.java b/src/test/java/net/openhft/chronicle/map/fromdocs/ChecksumEntryTest.java new file mode 100644 index 000000000..7d5bc3a8c --- /dev/null +++ b/src/test/java/net/openhft/chronicle/map/fromdocs/ChecksumEntryTest.java @@ -0,0 +1,72 @@ +/* + * Copyright (C) 2015 higherfrequencytrading.com + * + * This program is free software: you can redistribute it and/or modify + * it under the terms of the GNU Lesser General Public License as published by + * the Free Software Foundation, either version 3 of the License. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public License + * along with this program. If not, see . + */ + +package net.openhft.chronicle.map.fromdocs; + +import net.openhft.chronicle.core.values.LongValue; +import net.openhft.chronicle.hash.ChecksumEntry; +import net.openhft.chronicle.map.ChronicleMap; +import net.openhft.chronicle.map.ExternalMapQueryContext; +import net.openhft.chronicle.map.MapEntry; +import net.openhft.chronicle.set.Builder; +import net.openhft.chronicle.values.Values; +import org.junit.Assert; +import org.junit.Test; + +import java.io.File; +import java.io.IOException; + +public class ChecksumEntryTest { + + @Test + public void testChecksumEntriesWithValueInterface() throws IOException { + File file = Builder.getPersistenceFile(); + + try (ChronicleMap map = ChronicleMap + .of(Integer.class, LongValue.class) + .entries(1) + // Entry checksums make sense only for persisted Chronicle Maps, and are ON by + // default for such maps + .createPersistedTo(file)) { + + LongValue value = Values.newHeapInstance(LongValue.class); + value.setValue(42); + map.put(1, value); + + try (ExternalMapQueryContext c = map.queryContext(1)) { + // Update lock required for calling ChecksumEntry.checkSum() + c.updateLock().lock(); + MapEntry entry = c.entry(); + Assert.assertNotNull(entry); + ChecksumEntry checksumEntry = (ChecksumEntry) entry; + Assert.assertTrue(checksumEntry.checkSum()); + + // to access off-heap bytes, should call value().getUsing() with Native value + // provided. Simple get() return Heap value by default + LongValue nativeValue = + entry.value().getUsing(Values.newNativeReference(LongValue.class)); + // This value bytes update bypass Chronicle Map internals, so checksum is not + // updated automatically + nativeValue.setValue(43); + Assert.assertFalse(checksumEntry.checkSum()); + + // Restore correct checksum + checksumEntry.updateChecksum(); + Assert.assertTrue(checksumEntry.checkSum()); + } + } + } +} diff --git a/src/test/java/net/openhft/chronicle/set/SetEntryOperationsTest.java b/src/test/java/net/openhft/chronicle/set/SetEntryOperationsTest.java index 9d979f6d9..f0cc6b3a2 100644 --- a/src/test/java/net/openhft/chronicle/set/SetEntryOperationsTest.java +++ b/src/test/java/net/openhft/chronicle/set/SetEntryOperationsTest.java @@ -41,9 +41,9 @@ public class SetEntryOperationsTest { } @Override - public Void insert(@NotNull SetAbsentEntry entry) { + public Void insert(@NotNull SetAbsentEntry absentEntry) { insertCounter.addAndGet(1); - entry.doInsert(); + absentEntry.doInsert(); return null; } })