diff --git a/.gitignore b/.gitignore
index 4e4647b1a..bd747a416 100644
--- a/.gitignore
+++ b/.gitignore
@@ -8,3 +8,4 @@
*.war
*.ear
*.iml
+.idea*
diff --git a/README.md b/README.md
index 9e26588df..245ce3a54 100644
--- a/README.md
+++ b/README.md
@@ -6,7 +6,7 @@ requirements. - [Contact Us](sales@higherfrequencytrading.com)*
# Chronicle Map
-Replicate your Key Value Store across your network, with consistency, durability and performance.
+Replicate your Key Value Store across your network, with eventual consistency, persistence and performance.
#### Maven Artifact Download
@@ -30,7 +30,7 @@ Click here to get the [Latest Version Number](http://search.maven.org/#search%7C
* [Maven Download](https://github.com/OpenHFT/Chronicle-Map#maven-artifact-download-1)
* [Snapshot Download](https://github.com/OpenHFT/Chronicle-Map#maven-snapshot-download)
* [Key Value Object Types](https://github.com/OpenHFT/Chronicle-Map#key-value-object-types)
- * [Off Heap and How to improve performance]()
+ * [Off Heap and How to improve performance](https://github.com/OpenHFT/Chronicle-Map#off-heap-storage-and-how-using-a-proxy-object-can-improve-performance)
* [Sharing Data Between Two or More Maps](https://github.com/OpenHFT/Chronicle-Map#sharing-data-between-two-or-more-maps)
* [Entries](https://github.com/OpenHFT/Chronicle-Map#entries)
* [Size of Space Reserved on Disk](https://github.com/OpenHFT/Chronicle-Map#size-of-space-reserved-on-disk)
@@ -54,6 +54,7 @@ Click here to get the [Latest Version Number](http://search.maven.org/#search%7C
* [Heart Beat Interval](https://github.com/OpenHFT/Chronicle-Map#heart-beat-interval)
* [Multi Chronicle Maps - Network Distributed](https://github.com/OpenHFT/Chronicle-Map#multiple-chronicle-maps---network-distributed)
* [Stateless Client](https://github.com/OpenHFT/Chronicle-Map#stateless-client)
+* [How to speed up the Chronicle Map Stateless Client](https://github.com/OpenHFT/Chronicle-Map#how-to-speed-up-the-chronicle-map-stateless-client)
* [Questions/Answers](https://github.com/OpenHFT/Chronicle-Map#questions-and-answers)
#### Miscellaneous
@@ -168,7 +169,7 @@ and define the snapshot version in your pom.xml, for example:
net.openhftchronicle-map
- 1.0.1-SNAPSHOT
+
```
@@ -237,7 +238,7 @@ between processes by just using memory and in around 40 nanoseconds.
``` java
ConcurrentMap map1, map2;
-// this could could be on one process
+// this could be on one process
map1 = ChronicleMapBuilder.of(Integer.class, CharSequence.class).createPersistedTo(file);
// this could be on the other process
@@ -355,7 +356,7 @@ These methods let you provide the object which the data will be written to, but
``` java
CharSequence using = new StringBuilder();
CharSequence myResult = map.getUsing("key", using);
-// at this point the myString and myResult will both point to the same object
+// at this point "using" and "myResult" will both point to the same object
```
The `map.getUsing()` method is similar to `map.get()`, but because Chronicle Map stores its data off
@@ -381,6 +382,12 @@ the chronicle map is able to distribute entries between processes with extremely
as both processes are sharing the same off heap memory space. Since your objects are already stored off
heap ( as a series of bytes ), replicating entries over the network adds relatively low over head.
+For more information on the benefit's of off heap memory, see our article on - [On
+heap vs off
+heap memory
+usage]
+(http://vanillajava.blogspot.co.uk/2014/12/on-heap-vs-off-heap-memory-usage.html)
+
One of the downsides of an Off Heap Map is that whenever you wish to get a value ( on heap ) from an entry
which is off heap, for example calling :
@@ -461,10 +468,10 @@ the off heap data structures directly, this reduced serialisation can give you a
Below we show you how you can work directly with the off heap entries.
``` java
- ChronicleMap chm = ChronicleMapBuilder
- .of(String.class, BondVOInterface.class)
- .keySize(10)
- .create();
+ChronicleMap map = ChronicleMapBuilder
+ .of(String.class, BondVOInterface.class)
+ .keySize(10)
+ .create();
```
notice that the
@@ -476,29 +483,29 @@ notice that the
value class, in our case `BondVOInterface.class` is an `interface` rather than a `class`, now
like before, we can
use the `getUsing(key,using)` method, but this time we have to create the ‘using’ instance slightly
-differently, we have to call the `newDirectReference(..)` method.
+differently, we have to call either the `map.newValueInstance()` or `map.newKeyInstance()` method.
``` java
-BondVOInterface using = DataValueClasses.newDirectReference(BondVOInterface.class);
+BondVOInterface using = map.newValueInstance();
```
the call to `getUsing(key,value)` is the same as we had in the earlier example
``` java
-BondVOInterface using = DataValueClasses.newDirectReference(BondVOInterface.class);
+BondVOInterface using = map.newValueInstance();
BondVOInterface bond = map.getUsing(key,using);
assert using == bond; // this will always be the same instance
```
`getUsing(key,using)` won’t create any on heap objects, and it won’t deserialize the ‘bond’ entry from off heap to on
heap, all it does is sets the bond as a proxy to the off heap memory, this proxy object was created by
-`DataValueClasses.newDirectReference(BondVOInterface.class)`, it allows us access to the fields of our
+`map.newValueInstance()`, it allows us access to the fields of our
entry, directly into the off heap storage.
As above, ideally you would reuse the `using` variable.
``` java
-BondVOInterface using = DataValueClasses.newDirectReference(BondVOInterface.class);
+BondVOInterface using = map.newValueInstance();
for(int i=1;i<=10;i++) {
Value bond = map.getUsing(key,using); // this won’t create a new value each time.
@@ -543,8 +550,7 @@ context.close() // the lock will get released when this is called
#### acquireUsingLocked()
Just like getUsing(), acquireUsing() will also recycle the value you pass it, the following
-code is a pattern that you will often come across, `acquireUsing(key,value)` offers this
-functionality i the example, below with a single method call :
+code is a pattern that you will often come across,
``` java
V acquireUsing(key,value) {
@@ -563,6 +569,10 @@ V acquireUsing(key,value) {
}
```
+`acquireUsing(key,value)` offers this functionality from a single method call, an be cause it
+implemented inside the chronicle map we are able to reduce the hash lookups and make it more
+efficient.
+
If you are only accessing ChronicleMap from a single thread. If you are not doing replication
and don't care about atomic reads. Then its simpler ( and faster ) to use acquireUsing() otherwise we
recommend you use `acquireUsingLocked(key,value)`
@@ -599,6 +609,9 @@ entries are no where near the max entry size.
## Serialization
+
+
+
Chronicle Map stores your data into off heap memory, so when you give it a Key or Value, it will
serialise these objects into bytes.
@@ -1041,11 +1054,15 @@ unique for each map you have.
A stateless client is an instance of a `ChronicleMap` or a `ChronicleSet` that does not hold any
data
locally, all the Map or Set operations are delegated via a Remote Procedure Calls ( RPC ) to
- another `ChronicleMap` or `ChronicleSet` which we will refer to as the server. The server will
- hold all your data, the server can not it’s self be a stateless client. Your stateless client must
- be connected to the server via TCP/IP. The stateless client will delegate all your method calls to
- the remote server. The stateless client operations will block, in other words the stateless client
- will wait for the server to send a response before continuing to the next operation. The stateless
+ another `ChronicleMap` or `ChronicleSet` which we will refer to as the server. The server
+ holds all your data, the server can not it’s self be a stateless client. Your stateless client must
+ be connected to the server via TCP/IP.
+
+ 
+
+ The stateless client delegates all your method calls to
+ the remote server. The stateless client operations will block, in other words the stateless
+ client waits for the server to send a response before continuing to the next operation. The stateless
client could be consider to be a ClientProxy to `ChronicleMap` or `ChronicleSet` running
on another host.
@@ -1103,24 +1120,32 @@ but in your example you should choose the host of the state-full server and the
```
the ".statelessClient(..)" returns an instance of `StatelessClientConfig`, which has only a few
-own configurations and `create()` method to create a new stateless client.
-method tells `ChronicleMap` that its going to build a stateless client.
-If you don’t add this line a normal state-full `ChronicleMap` will be created. For this example
+of its own configurations, such as the `create()` method, which can be used to create a new
+stateless client.
+If you don’t add this line, a normal state-full `ChronicleMap` will be created. For this example
we ran both
the client an the server on the same host ( hence the “localhost" setting ),
but in a real life example the stateless client will typically be on a different server than the
state-full host. If you are aiming to create a stateless client and server on the same host, its
-better not to do this, as the stateless client connects to the server via TCP/IP,
-you would get better performance if you connect to the server via heap memory, to read more about
-sharing a map with heap memory
+better not to do this, as the stateless client connects to the server via TCP/IP. It better to
+share the maps via memory as this will give you better performance ( read more about
+this at [Sharing Data Between Two or More Maps](https://github
+.com/OpenHFT/Chronicle-Map#sharing-data-between-two-or-more-maps).
+
click [here](https://github.com/OpenHFT/Chronicle-Map#sharing-data-between-two-or-more-maps )
### How to speed up the Chronicle Map Stateless Client
-By default when you make a method call to a `ChronicleMap` stateless client,
-your method call is wrapped into an event which is sent over tcp and processed by the server,
-your stateless client will block until an acknowledgement has been received from the server that
-your event was processed. When you are calling methods that return a value like get() this
+When calling the stateless client, you will get better throughput if you invoke your requests from a
+ number of threads, this is because by default when you make a method call to a `ChronicleMap`
+ stateless client, your method call is wrapped into an event which is sent over tcp and processed
+ by the server,
+your stateless client will block until an acknowledgement has been received from the server that
+your event was processed.
+
+
+
+When you are calling methods that return a value like get() this
blocking adds no additional overhead, because you have to wait for the return value anyway,
In some cases you could get better performance if you don't have to wait for the acknowledgement, This maybe the case when you are calling the `put()` method, but the problem with this method is it returns the old value even though you may not use it.
diff --git a/pom.xml b/pom.xml
index 7092ad641..8e2f67a20 100644
--- a/pom.xml
+++ b/pom.xml
@@ -17,18 +17,20 @@
~ limitations under the License.
-->
-
+net.openhftjava-parent-pom1.1.2
-
+ 4.0.0chronicle-map
- 2.0.13b-SNAPSHOT
+ 2.0.15b-SNAPSHOTOpenHFT/Chronicle-MapChronicle-Map
@@ -46,7 +48,7 @@
net.openhftlang
- 6.4.16
+ 6.4.18-SNAPSHOT
@@ -73,21 +75,35 @@
1.3.6
+
+ org.ops4j.pax.url
+ pax-url-aether
+ 2.2.0
+
+
+ truexstreamxstream
+ trueorg.codehaus.jettisonjettison
+
+ true
+ org.ops4j.pax.url
+ pax-url-aether
+
+
net.openhftcompiler
@@ -172,6 +188,23 @@
+
+
+
+ org.apache.maven.plugins
+ maven-jar-plugin
+ 2.1
+
+
+
+
+ true
+ true
+
+
+
+
+
diff --git a/src/main/java/net/openhft/chronicle/hash/ChronicleHashBuilder.java b/src/main/java/net/openhft/chronicle/hash/ChronicleHashBuilder.java
index 7a12a592d..8dce56d17 100755
--- a/src/main/java/net/openhft/chronicle/hash/ChronicleHashBuilder.java
+++ b/src/main/java/net/openhft/chronicle/hash/ChronicleHashBuilder.java
@@ -23,6 +23,7 @@ import net.openhft.chronicle.hash.replication.TcpTransportAndNetworkConfig;
import net.openhft.chronicle.hash.replication.TimeProvider;
import net.openhft.chronicle.hash.serialization.*;
import net.openhft.chronicle.map.ChronicleMap;
+import net.openhft.chronicle.map.ChronicleMapBuilder;
import net.openhft.chronicle.set.ChronicleSet;
import net.openhft.chronicle.set.ChronicleSetBuilder;
import net.openhft.lang.io.Bytes;
@@ -46,7 +47,7 @@ import java.util.concurrent.TimeUnit;
/**
* This interface defines the meaning of configurations, common to {@link
- * net.openhft.chronicle.map.ChronicleMapBuilder} and {@link ChronicleSetBuilder}, i.
+ * ChronicleMapBuilder} and {@link ChronicleSetBuilder}, i.
* e. Chronicle hash container configurations.
*
*
{@code ChronicleHashBuilder} is mutable. Configuration methods mutate the builder and return
@@ -57,7 +58,7 @@ import java.util.concurrent.TimeUnit;
* @param the type of keys in hash containers, created by this builder
* @param the container type, created by this builder, i. e. {@link ChronicleMap} or {@link
* ChronicleSet}
- * @param the concrete builder type, i. e. {@link net.openhft.chronicle.map.ChronicleMapBuilder}
+ * @param the concrete builder type, i. e. {@link ChronicleMapBuilder}
* or {@link ChronicleSetBuilder}
*/
public interface ChronicleHashBuilderIf key is a boxed primitive type or {@link Byteable} subclass, i. e. if key size is known
+ * statically, it is automatically accounted and shouldn't be specified by user.
+ *
* @param keySize number of bytes, taken by serialized form of keys
* @return this builder back
+ * @throws IllegalStateException if key size is known statically and shouldn't be configured
+ * by user
+ * @throws IllegalArgumentException if the given {@code keySize} is non-positive
* @see #constantKeySizeBySample(Object)
* @see #entrySize(int)
*/
@@ -131,7 +138,7 @@ public interface ChronicleHashBuilderThere are three major patterns of this configuration usage:
Key (and value, in
* {@code ChronicleMap} case) sizes are constant. Configure them via {@link
- * #constantKeySizeBySample(Object)} and {@link net.openhft.chronicle.map.ChronicleMapBuilder#constantValueSizeBySample(Object)}
+ * #constantKeySizeBySample(Object)} and {@link ChronicleMapBuilder#constantValueSizeBySample(Object)}
* methods, and you will experience no memory waste at all.
Key (and/or value size, in
* {@code ChronicleMap} case) varies moderately. Specify them using corresponding methods, or
* specify entry size directly by calling this method, by sizes somewhere between average and
@@ -145,10 +152,11 @@ public interface ChronicleHashBuilderHowever, remember that
Operations with entries that span several "entry sizes"
* are a bit slower, than entries which take a single "entry size". That is why "chunk" approach
* is not recommended, when key (and/or value size) varies moderately.
The maximum
- * number of chunks could be taken by an entry is 64. {@link IllegalArgumentException} is thrown
- * on attempt to insert too large entry, compared to the configured or computed entry size.
- *
The number of "entries" {@linkplain #entries(long) in the whole map} and {@linkplain
- * #actualEntriesPerSegment(long) per segment} is actually the number "chunks".
+ * number of chunks could be taken by an entry is {@link #maxEntryOversizeFactor(int)}, of max
+ * 64 chunks. {@link IllegalArgumentException} is thrown on attempt to insert too large entry,
+ * compared to the configured or computed entry size.
The number of "entries"
+ * {@linkplain #entries(long) in the whole map} and {@linkplain #actualEntriesPerSegment(long)
+ * per segment} is actually the number "chunks".
*
*
Example: if values in your {@code ChronicleMap} are adjacency lists of some social graph,
* where nodes are represented as {@code long} ids, and adjacency lists are serialized in
@@ -169,9 +177,22 @@ public interface ChronicleHashBuilder
If key (and/or value, in {@code ChronicleMap} case) size varies
* moderately, you should pass to this method the maximum number of entries + 5-25%, depending
* on your data properties and configured {@linkplain #keySize(int) key}/{@linkplain
- * net.openhft.chronicle.map.ChronicleMapBuilder#valueSize(int)
+ * ChronicleMapBuilder#valueSize(int)
* value}/{@linkplain #entrySize(int) entry} sizes.
If your data size varies in a wide
* range, pass the maximum number of entries multiplied by average data size and divided by the
* configured "entry size" (i. e. chunk size). See an example in the documentation to {@link
@@ -292,7 +313,7 @@ public interface ChronicleHashBuilder keyMarshaller);
+ B keyMarshaller(@NotNull BytesMarshaller keyMarshaller);
/**
* Configures the marshallers, used to serialize/deserialize keys to/from off-heap memory in
@@ -332,11 +353,10 @@ public interface ChronicleHashBuilderDefault key deserialization factory is {@link NewInstanceObjectFactory}, which creates a
* new key instance using {@link Class#newInstance()} default constructor. You could provide an
@@ -348,9 +368,8 @@ public interface ChronicleHashBuilder keyDeserializationFactory);
@@ -401,8 +420,6 @@ public interface ChronicleHashBuilder statelessClient(InetSocketAddress remoteAddress);
-
ChronicleHashInstanceConfig instance();
/**
@@ -414,12 +431,10 @@ public interface ChronicleHashBuilderThis method is a shortcut for {@code instance().create()}.
*
* @return a new off-heap hash container
- * @throws IOException if any IO error relates to off-heap memory allocation, or establishing
- * replication connections, occurs
* @see #createPersistedTo(File)
* @see #instance()
*/
- C create() throws IOException;
+ C create();
/**
* Opens a hash container residing the specified file, or creates a new one if the file not yet
@@ -450,5 +465,7 @@ public interface ChronicleHashBuilder statelessClient(InetSocketAddress remoteAddress);
+
+ C createStatelessClient(InetSocketAddress serverAddress) throws IOException;
}
diff --git a/src/main/java/net/openhft/chronicle/hash/ChronicleHashInstanceConfig.java b/src/main/java/net/openhft/chronicle/hash/ChronicleHashInstanceConfig.java
index 21b01661b..8afd3d188 100644
--- a/src/main/java/net/openhft/chronicle/hash/ChronicleHashInstanceConfig.java
+++ b/src/main/java/net/openhft/chronicle/hash/ChronicleHashInstanceConfig.java
@@ -53,5 +53,7 @@ public interface ChronicleHashInstanceConfig {
ChronicleHashInstanceConfig persistedTo(File file);
+ ChronicleHashInstanceConfig name(String name);
+
C create() throws IOException;
}
diff --git a/src/main/java/net/openhft/chronicle/hash/StatelessClientConfig.java b/src/main/java/net/openhft/chronicle/hash/StatelessClientConfig.java
index 01836241e..38add9b6a 100644
--- a/src/main/java/net/openhft/chronicle/hash/StatelessClientConfig.java
+++ b/src/main/java/net/openhft/chronicle/hash/StatelessClientConfig.java
@@ -18,8 +18,10 @@ package net.openhft.chronicle.hash;
import net.openhft.chronicle.map.ChronicleMap;
import net.openhft.chronicle.set.ChronicleSet;
+import org.jetbrains.annotations.NotNull;
import java.io.IOException;
+import java.net.InetSocketAddress;
import java.util.concurrent.TimeUnit;
/**
@@ -41,5 +43,7 @@ import java.util.concurrent.TimeUnit;
public interface StatelessClientConfig {
StatelessClientConfig timeout(long timeout, TimeUnit units);
+ StatelessClientConfig name(String name);
+
C create() throws IOException;
}
diff --git a/src/main/java/net/openhft/chronicle/hash/hashing/Access.java b/src/main/java/net/openhft/chronicle/hash/hashing/Access.java
new file mode 100644
index 000000000..44088b816
--- /dev/null
+++ b/src/main/java/net/openhft/chronicle/hash/hashing/Access.java
@@ -0,0 +1,140 @@
+/*
+ * Copyright 2014 Higher Frequency Trading http://www.higherfrequencytrading.com
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package net.openhft.chronicle.hash.hashing;
+
+import java.nio.ByteOrder;
+
+/**
+ * Strategy of reading bytes, defines the abstraction of {@code T} class instances as ordered byte
+ * sequence. All {@code getXXX(input, offset)} should be consistent to each other in terms of
+ * ordered byte sequence each {@code T} instance represents. For example, if some {@code
+ * Access} implementation returns {@link ByteOrder#LITTLE_ENDIAN} on {@link #byteOrder(Object)
+ * byteOrder(input)} call, the following expressions should always have the same value:
+ *
{@code getXXX(input, offset)} methods could throw unchecked exceptions when requested bytes
+ * range is outside of the bounds of the byte sequence, represented by the given {@code input}.
+ * However, they could omit checks for better performance.
+ *
+ *
{@code Access} API is designed for inputs, that actually represent byte sequences that lay
+ * continuously in memory. Theoretically {@code Access} strategy could be implemented for
+ * non-continuous byte sequences, or abstractions which aren't actually present in memory as they
+ * are accessed, but this should be awkward, and hashing using such {@code Access} is expected to
+ * be slow.
+ *
+ * @param the type of the object to access
+ */
+public interface Access {
+
+ /**
+ * Reads {@code [offset, offset + 7]} bytes of the byte sequence represented by the given
+ * {@code input} as a single {@code long} value.
+ *
+ * @param input the object to access
+ * @param offset offset to the first byte to read within the byte sequence represented
+ * by the given object
+ * @return eight bytes as a {@code long} value, in {@linkplain #byteOrder(Object) the expected
+ * order}
+ */
+ long getLong(T input, long offset);
+
+ /**
+ * Shortcut for {@code getInt(input, offset) & 0xFFFFFFFFL}. Could be implemented more
+ * efficiently.
+ *
+ * @param input the object to access
+ * @param offset offset to the first byte to read within the byte sequence represented
+ * by the given object
+ * @return four bytes as an unsigned int value, in {@linkplain #byteOrder(Object) the expected
+ * order}
+ */
+ long getUnsignedInt(T input, long offset);
+
+ /**
+ * Reads {@code [offset, offset + 3]} bytes of the byte sequence represented by the given
+ * {@code input} as a single {@code int} value.
+ *
+ * @param input the object to access
+ * @param offset offset to the first byte to read within the byte sequence represented
+ * by the given object
+ * @return four bytes as an {@code int} value, in {@linkplain #byteOrder(Object) the expected
+ * order}
+ */
+ int getInt(T input, long offset);
+
+ /**
+ * Shortcut for {@code getShort(input, offset) & 0xFFFF}. Could be implemented more
+ * efficiently.
+ *
+ * @param input the object to access
+ * @param offset offset to the first byte to read within the byte sequence represented
+ * by the given object
+ * @return two bytes as an unsigned short value, in {@linkplain #byteOrder(Object) the expected
+ * order}
+ */
+ int getUnsignedShort(T input, long offset);
+
+ /**
+ * Reads {@code [offset, offset + 1]} bytes of the byte sequence represented by the given
+ * {@code input} as a single {@code short} value, returned widened to {@code int}.
+ *
+ * @param input the object to access
+ * @param offset offset to the first byte to read within the byte sequence represented
+ * by the given object
+ * @return two bytes as a {@code short} value, in {@linkplain #byteOrder(Object) the expected
+ * order}, widened to {@code int}
+ */
+ int getShort(T input, long offset);
+
+ /**
+ * Shortcut for {@code getByte(input, offset) & 0xFF}. Could be implemented more efficiently.
+ *
+ * @param input the object to access
+ * @param offset offset to the byte to read within the byte sequence represented
+ * by the given object
+ * @return a byte by the given {@code offset}, interpreted as unsigned
+ */
+ int getUnsignedByte(T input, long offset);
+
+ /**
+ * Reads a single byte at the given {@code offset} in the byte sequence represented by the given
+ * {@code input}, returned widened to {@code int}.
+ *
+ * @param input the object to access
+ * @param offset offset to the byte to read within the byte sequence represented
+ * by the given object
+ * @return a byte by the given {@code offset}, widened to {@code int}
+ */
+ int getByte(T input, long offset);
+
+ /**
+ * The byte order in which all multi-byte {@code getXXX()} reads from the given {@code input}
+ * are performed.
+ *
+ * @param input the accessed object
+ * @return the byte order of all multi-byte reads from the given {@code input}
+ */
+ ByteOrder byteOrder(T input);
+}
diff --git a/src/main/java/net/openhft/chronicle/hash/hashing/Accesses.java b/src/main/java/net/openhft/chronicle/hash/hashing/Accesses.java
new file mode 100644
index 000000000..d9b71abd5
--- /dev/null
+++ b/src/main/java/net/openhft/chronicle/hash/hashing/Accesses.java
@@ -0,0 +1,69 @@
+/*
+ * Copyright 2014 Higher Frequency Trading http://www.higherfrequencytrading.com
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package net.openhft.chronicle.hash.hashing;
+
+import net.openhft.lang.io.Bytes;
+
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+
+/**
+ * Static methods returning useful {@link Access} implementations.
+ */
+public final class Accesses {
+
+ /**
+ * Returns the {@code Access} delegating {@code getXXX(input, offset)} methods to {@code
+ * sun.misc.Unsafe.getXXX(input, offset)}.
+ *
+ *
Usage example:
{@code
+ * class Pair {
+ * long first, second;
+ *
+ * static final long pairDataOffset =
+ * theUnsafe.objectFieldOffset(Pair.class.getDeclaredField("first"));
+ *
+ * static long hashPair(Pair pair, LongHashFunction hashFunction) {
+ * return hashFunction.hash(pair, Accesses.unsafe(), pairDataOffset, 16L);
+ * }
+ * }}
+ *
+ *
{@code null} is a valid input, on accepting {@code null} {@code Unsafe} just interprets
+ * the given offset as a wild memory address.
+ *
+ * @param the type of objects to access
+ * @return the unsafe memory {@code Access}
+ */
+ public static Access unsafe() {
+ return (Access) UnsafeAccess.INSTANCE;
+ }
+
+ /**
+ * Returns the {@code Access} to any {@link ByteBuffer}.
+ *
+ * @return the {@code Access} to {@link ByteBuffer}s
+ */
+ public static Access toByteBuffer() {
+ return ByteBufferAccess.INSTANCE;
+ }
+
+ public static Access toBytes() {
+ return BytesAccess.INSTANCE;
+ }
+
+ private Accesses() {}
+}
diff --git a/src/main/java/net/openhft/chronicle/hash/hashing/ByteBufferAccess.java b/src/main/java/net/openhft/chronicle/hash/hashing/ByteBufferAccess.java
new file mode 100644
index 000000000..703111dff
--- /dev/null
+++ b/src/main/java/net/openhft/chronicle/hash/hashing/ByteBufferAccess.java
@@ -0,0 +1,64 @@
+/*
+ * Copyright 2014 Higher Frequency Trading http://www.higherfrequencytrading.com
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package net.openhft.chronicle.hash.hashing;
+
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+
+enum ByteBufferAccess implements Access {
+ INSTANCE;
+
+ @Override
+ public long getLong(ByteBuffer input, long offset) {
+ return input.getLong((int) offset);
+ }
+
+ @Override
+ public long getUnsignedInt(ByteBuffer input, long offset) {
+ return Primitives.unsignedInt(getInt(input, offset));
+ }
+
+ @Override
+ public int getInt(ByteBuffer input, long offset) {
+ return input.getInt((int) offset);
+ }
+
+ @Override
+ public int getUnsignedShort(ByteBuffer input, long offset) {
+ return Primitives.unsignedShort(getShort(input, offset));
+ }
+
+ @Override
+ public int getShort(ByteBuffer input, long offset) {
+ return input.getShort((int) offset);
+ }
+
+ @Override
+ public int getUnsignedByte(ByteBuffer input, long offset) {
+ return Primitives.unsignedByte(getByte(input, offset));
+ }
+
+ @Override
+ public int getByte(ByteBuffer input, long offset) {
+ return input.get((int) offset);
+ }
+
+ @Override
+ public ByteOrder byteOrder(ByteBuffer input) {
+ return input.order();
+ }
+}
diff --git a/src/main/java/net/openhft/chronicle/hash/hashing/BytesAccess.java b/src/main/java/net/openhft/chronicle/hash/hashing/BytesAccess.java
new file mode 100644
index 000000000..4e67b40f8
--- /dev/null
+++ b/src/main/java/net/openhft/chronicle/hash/hashing/BytesAccess.java
@@ -0,0 +1,66 @@
+/*
+ * Copyright 2014 Higher Frequency Trading http://www.higherfrequencytrading.com
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package net.openhft.chronicle.hash.hashing;
+
+import net.openhft.lang.io.Bytes;
+
+import java.nio.ByteOrder;
+
+enum BytesAccess implements Access {
+ INSTANCE;
+
+ @Override
+ public long getLong(Bytes input, long offset) {
+ return input.readLong(offset);
+ }
+
+ @Override
+ public long getUnsignedInt(Bytes input, long offset) {
+ return input.readUnsignedInt(offset);
+ }
+
+ @Override
+ public int getInt(Bytes input, long offset) {
+ return input.readInt(offset);
+ }
+
+ @Override
+ public int getUnsignedShort(Bytes input, long offset) {
+ return input.readUnsignedShort(offset);
+ }
+
+ @Override
+ public int getShort(Bytes input, long offset) {
+ return input.readShort(offset);
+ }
+
+ @Override
+ public int getUnsignedByte(Bytes input, long offset) {
+ return input.readUnsignedByte(offset);
+ }
+
+ @Override
+ public int getByte(Bytes input, long offset) {
+ return input.readByte(offset);
+ }
+
+ @Override
+ public ByteOrder byteOrder(Bytes input) {
+ return input.byteOrder();
+ }
+
+}
diff --git a/src/main/java/net/openhft/chronicle/hash/hashing/Hasher.java b/src/main/java/net/openhft/chronicle/hash/hashing/Hasher.java
new file mode 100644
index 000000000..a7c97393c
--- /dev/null
+++ b/src/main/java/net/openhft/chronicle/hash/hashing/Hasher.java
@@ -0,0 +1,194 @@
+/*
+ * Copyright 2014 Higher Frequency Trading
+ *
+ * http://www.higherfrequencytrading.com
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package net.openhft.chronicle.hash.hashing;
+
+import net.openhft.lang.io.Bytes;
+
+import static net.openhft.chronicle.hash.hashing.UnsafeAccess.BYTE_BASE;
+import static net.openhft.chronicle.hash.hashing.UnsafeAccess.CHAR_BASE;
+
+public final class Hasher {
+
+ // MurmurHash3, derived from
+ // https://github.com/google/guava/blob/fa95e381e665d8ee9639543b99ed38020c8de5ef
+ // /guava/src/com/google/common/hash/Murmur3_128HashFunction.java
+
+
+ private static final long C1 = 0x87c37b91114253d5L;
+ private static final long C2 = 0x4cf5ad432745937fL;
+
+ /**
+ * Returns the hash code for {@code len} continuous bytes of the given {@code input} object,
+ * starting from the given offset. The abstraction of input as ordered byte sequence and
+ * "offset within the input" is defined by the given {@code access} strategy.
+ *
+ *
This method doesn't promise to throw a {@code RuntimeException} if {@code
+ * [off, off + len - 1]} subsequence exceeds the bounds of the bytes sequence, defined by {@code
+ * access} strategy for the given {@code input}, so use this method with caution.
+ *
+ * @param input the object to read bytes from
+ * @param access access which defines the abstraction of the given input
+ * as ordered byte sequence
+ * @param offset offset to the first byte of the subsequence to hash
+ * @param length length of the subsequence to hash
+ * @param the type of the input
+ * @return hash code for the specified bytes subsequence
+ */
+ public static long hash(T input, Access access, long offset, long length) {
+ assert offset >= 0L && length >= 0L;
+
+ long h1 = 0L;
+ long h2 = 0L;
+ long remaining = length;
+ while (remaining >= 16L) {
+ long k1 = access.getLong(input, offset);
+ long k2 = access.getLong(input, offset);
+ offset += 16L;
+ remaining -= 16L;
+ h1 ^= mixK1(k1);
+
+ h1 = Long.rotateLeft(h1, 27);
+ h1 += h2;
+ h1 = h1 * 5L + 0x52dce729L;
+
+ h2 ^= mixK2(k2);
+
+ h2 = Long.rotateLeft(h2, 31);
+ h2 += h1;
+ h2 = h2 * 5L + 0x38495ab5L;
+ }
+
+ if (remaining > 0L) {
+ long k1 = 0L;
+ long k2 = 0L;
+ switch ((int) remaining) {
+ case 15:
+ k2 ^= ((long) access.getUnsignedByte(input, offset + 14L)) << 48;// fall through
+ case 14:
+ k2 ^= ((long) access.getUnsignedByte(input, offset + 13L)) << 40;// fall through
+ case 13:
+ k2 ^= ((long) access.getUnsignedByte(input, offset + 12L)) << 32;// fall through
+ case 12:
+ k2 ^= ((long) access.getUnsignedByte(input, offset + 11L)) << 24;// fall through
+ case 11:
+ k2 ^= ((long) access.getUnsignedByte(input, offset + 10L)) << 16;// fall through
+ case 10:
+ k2 ^= ((long) access.getUnsignedByte(input, offset + 9L)) << 8; // fall through
+ case 9:
+ k2 ^= ((long) access.getUnsignedByte(input, offset + 8L)); // fall through
+ case 8:
+ k1 ^= access.getLong(input, offset);
+ break;
+ case 7:
+ k1 ^= ((long) access.getUnsignedByte(input, offset + 6L)) << 48; // fall through
+ case 6:
+ k1 ^= ((long) access.getUnsignedByte(input, offset + 5L)) << 40; // fall through
+ case 5:
+ k1 ^= ((long) access.getUnsignedByte(input, offset + 4L)) << 32; // fall through
+ case 4:
+ k1 ^= access.getUnsignedInt(input, offset);
+ break;
+ case 3:
+ k1 ^= ((long) access.getUnsignedByte(input, offset + 2L)) << 16; // fall through
+ case 2:
+ k1 ^= ((long) access.getUnsignedByte(input, offset + 1L)) << 8; // fall through
+ case 1:
+ k1 ^= ((long) access.getUnsignedByte(input, offset));
+ case 0:
+ break;
+ default:
+ throw new AssertionError("Should never get here.");
+ }
+ h1 ^= mixK1(k1);
+ h2 ^= mixK2(k2);
+ }
+
+ return finalize(length, h1, h2);
+ }
+
+ private static long finalize(long length, long h1, long h2) {
+ h1 ^= length;
+ h2 ^= length;
+
+ h1 += h2;
+ h2 += h1;
+
+ h1 = fmix64(h1);
+ h2 = fmix64(h2);
+
+ h1 += h2;
+ return h1;
+ }
+
+
+ private static long fmix64(long k) {
+ k ^= k >>> 33;
+ k *= 0xff51afd7ed558ccdL;
+ k ^= k >>> 33;
+ k *= 0xc4ceb9fe1a85ec53L;
+ k ^= k >>> 33;
+ return k;
+ }
+
+ private static long mixK1(long k1) {
+ k1 *= C1;
+ k1 = Long.rotateLeft(k1, 31);
+ k1 *= C2;
+ return k1;
+ }
+
+ private static long mixK2(long k2) {
+ k2 *= C2;
+ k2 = Long.rotateLeft(k2, 33);
+ k2 *= C1;
+ return k2;
+ }
+
+ public static long hash(Bytes bytes) {
+ return hash(bytes, bytes.position(), bytes.limit());
+ }
+
+ public static long hash(Bytes bytes, long offset, long limit) {
+ return hash(bytes, Accesses.toBytes(), offset, limit - offset);
+ }
+
+ public static long hash(byte[] array) {
+ return hash(array, Accesses.unsafe(), BYTE_BASE, (long) array.length);
+ }
+
+ public static long hash(char[] array) {
+ return hash(array, Accesses.unsafe(), CHAR_BASE, array.length * 2L);
+ }
+
+ public static long hash(int value) {
+ long k1 = Primitives.unsignedInt(value);
+ long h1 = 0L ^ mixK1(k1);
+ long h2 = 0L;
+ return finalize(4L, h1, h2);
+ }
+
+ public static long hash(long value) {
+ long k1 = value;
+ long h1 = 0L ^ mixK1(k1);
+ long h2 = 0L;
+ return finalize(8L, h1, h2);
+ }
+
+ private Hasher() {}
+}
diff --git a/src/main/java/net/openhft/chronicle/hash/hashing/Primitives.java b/src/main/java/net/openhft/chronicle/hash/hashing/Primitives.java
new file mode 100644
index 000000000..ba1ac75a9
--- /dev/null
+++ b/src/main/java/net/openhft/chronicle/hash/hashing/Primitives.java
@@ -0,0 +1,34 @@
+/*
+ * Copyright 2014 Higher Frequency Trading http://www.higherfrequencytrading.com
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package net.openhft.chronicle.hash.hashing;
+
+final class Primitives {
+
+ private Primitives() {}
+
+ static long unsignedInt(int i) {
+ return i & 0xFFFFFFFFL;
+ }
+
+ static int unsignedShort(int s) {
+ return s & 0xFFFF;
+ }
+
+ static int unsignedByte(int b) {
+ return b & 0xFF;
+ }
+}
diff --git a/src/main/java/net/openhft/chronicle/hash/hashing/UnsafeAccess.java b/src/main/java/net/openhft/chronicle/hash/hashing/UnsafeAccess.java
new file mode 100644
index 000000000..36d54a1cb
--- /dev/null
+++ b/src/main/java/net/openhft/chronicle/hash/hashing/UnsafeAccess.java
@@ -0,0 +1,93 @@
+/*
+ * Copyright 2014 Higher Frequency Trading http://www.higherfrequencytrading.com
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package net.openhft.chronicle.hash.hashing;
+
+import sun.misc.Unsafe;
+
+import java.lang.reflect.Field;
+import java.nio.ByteOrder;
+
+import static net.openhft.chronicle.hash.hashing.Primitives.unsignedByte;
+import static net.openhft.chronicle.hash.hashing.Primitives.unsignedInt;
+import static net.openhft.chronicle.hash.hashing.Primitives.unsignedShort;
+
+enum UnsafeAccess implements Access
*
From {@code valuePos} offset value area in the given {@code entry}, serialized value size
* (using {@linkplain ChronicleMapBuilder#keySizeMarshaller(SizeMarshaller)}), then (optionally)
- * alignment (see {@link OffHeapUpdatableChronicleMapBuilder#entryAndValueAlignment(Alignment)}
+ * alignment (see {@link ChronicleMapBuilder#entryAndValueAlignment(Alignment)}
* ), followed by the serialized value itself.
*
*/
diff --git a/src/main/java/net/openhft/chronicle/map/ChannelProvider.java b/src/main/java/net/openhft/chronicle/map/ChannelProvider.java
index ea56231c2..b69f8c3bb 100644
--- a/src/main/java/net/openhft/chronicle/map/ChannelProvider.java
+++ b/src/main/java/net/openhft/chronicle/map/ChannelProvider.java
@@ -64,7 +64,7 @@ final class ChannelProvider implements Closeable {
final TcpReplicator tcpReplicator = new TcpReplicator(
channelProvider.asReplica,
channelProvider.asEntryExternalizable,
- tcpConfig, hub.maxEntrySize(), null, hub.remoteNodeValidator());
+ tcpConfig, null, hub.remoteNodeValidator());
channelProvider.add(tcpReplicator);
}
@@ -85,6 +85,30 @@ final class ChannelProvider implements Closeable {
private static final byte BOOTSTRAP_MESSAGE = 'B';
final EntryExternalizable asEntryExternalizable = new EntryExternalizable() {
+ @Override
+ public int sizeOfEntry(@NotNull Bytes entry, int chronicleChannel) {
+ channelDataLock.readLock().lock();
+ try {
+
+ return channelEntryExternalizables[chronicleChannel]
+ .sizeOfEntry(entry, chronicleChannel);
+ } finally {
+ channelDataLock.readLock().unlock();
+ }
+ }
+
+ @Override
+ public boolean identifierCheck(@NotNull Bytes entry, int chronicleChannel) {
+ channelDataLock.readLock().lock();
+ try {
+
+ return channelEntryExternalizables[chronicleChannel]
+ .identifierCheck(entry, chronicleChannel);
+ } finally {
+ channelDataLock.readLock().unlock();
+ }
+ }
+
/**
* writes the entry to the chronicle channel provided
*
@@ -116,7 +140,7 @@ final class ChannelProvider implements Closeable {
final int chronicleId = (int) source.readStopBit();
if (chronicleId < chronicleChannels.length) {
// this channel is has not currently been created so it updates will be ignored
- if (channelEntryExternalizables[chronicleId] != null)
+ if (channelEntryExternalizables[chronicleId] != null)
channelEntryExternalizables[chronicleId]
.readExternalEntry(copies, segmentState, source);
} else
@@ -398,8 +422,8 @@ final class ChannelProvider implements Closeable {
}
/**
- * used to send system messages such as bootstrap from one remote node to another, it also can be used in
- * a broadcast context
+ * used to send system messages such as bootstrap from one remote node to another, it also can
+ * be used in a broadcast context
*/
class SystemQueue {
@@ -467,6 +491,16 @@ final class ChannelProvider implements Closeable {
}
};
final EntryExternalizable asEntryExternalizable = new EntryExternalizable() {
+ @Override
+ public int sizeOfEntry(@NotNull Bytes entry, int chronicleId) {
+ return (int) entry.remaining();
+ }
+
+ @Override
+ public boolean identifierCheck(@NotNull Bytes entry, int chronicleId) {
+ return true;
+ }
+
@Override
public void writeExternalEntry(@NotNull Bytes entry, @NotNull Bytes destination,
int na) {
@@ -506,7 +540,7 @@ final class ChannelProvider implements Closeable {
}
@Override
- protected Closeable applyTo(AbstractChronicleMapBuilder builder,
+ protected Closeable applyTo(ChronicleMapBuilder builder,
Replica map, EntryExternalizable entryExternalizable,
final ChronicleMap chronicleMap) {
add(chronicleChannel, map, entryExternalizable);
diff --git a/src/main/java/net/openhft/chronicle/map/ChronicleMap.java b/src/main/java/net/openhft/chronicle/map/ChronicleMap.java
index db2acbc35..33f98fa45 100755
--- a/src/main/java/net/openhft/chronicle/map/ChronicleMap.java
+++ b/src/main/java/net/openhft/chronicle/map/ChronicleMap.java
@@ -36,7 +36,7 @@ import java.util.concurrent.ConcurrentMap;
*
*
Functionally this interface defines some methods supporting garbage-free off-heap programming:
* {@link #getUsing(Object, Object)}, {@link #acquireUsing(Object, Object)}.
@@ -98,7 +98,7 @@ public interface ChronicleMap extends ConcurrentMap, ChronicleHash {
* no mapping for the key
* @see #get(Object)
* @see #acquireUsing(Object, Object)
- * @see OnHeapUpdatableChronicleMapBuilder#valueMarshaller(BytesMarshaller)
+ * @see ChronicleMapBuilder#valueMarshaller(BytesMarshaller)
*/
V getUsing(K key, V usingValue);
@@ -141,7 +141,7 @@ public interface ChronicleMap extends ConcurrentMap, ChronicleHash {
* no mapping for the key
* @see #get(Object)
* @see #getUsing(Object, Object)
- * @see OnHeapUpdatableChronicleMapBuilder#valueMarshaller(BytesMarshaller)
+ * @see ChronicleMapBuilder#valueMarshaller(BytesMarshaller)
*/
@NotNull
ReadContext getUsingLocked(@NotNull K key, @NotNull V usingValue);
@@ -149,8 +149,8 @@ public interface ChronicleMap extends ConcurrentMap, ChronicleHash {
/**
* Acquire a value for a key, creating if absent.
*
- *
If the specified key is absent in the map, {@linkplain AbstractChronicleMapBuilder#defaultValue(Object)
- * default value} is taken or {@linkplain AbstractChronicleMapBuilder#defaultValueProvider(DefaultValueProvider)
+ *
If the specified key is absent in the map, {@linkplain ChronicleMapBuilder#defaultValue(Object)
+ * default value} is taken or {@linkplain ChronicleMapBuilder#defaultValueProvider(DefaultValueProvider)
* default value provider} is called. Then this object is put to this map for the specified
* key.
*
@@ -169,13 +169,13 @@ public interface ChronicleMap extends ConcurrentMap, ChronicleHash {
* }}
*
*
- * Where {@code defaultValue(key)} returns either {@linkplain AbstractChronicleMapBuilder#defaultValue(Object)
- * default value} or {@link AbstractChronicleMapBuilder#defaultValueProvider(DefaultValueProvider)
+ * Where {@code defaultValue(key)} returns either {@linkplain ChronicleMapBuilder#defaultValue(Object)
+ * default value} or {@link ChronicleMapBuilder#defaultValueProvider(DefaultValueProvider)
* defaultValueProvider.}
*
*
If the {@code ChronicleMap} is off-heap updatable, i. e. created via {@link
* ChronicleMapBuilder} builder (values are {@link Byteable}), there is one more option of what
- * to do if the key is absent in the map, see {@link ChronicleMapBuilder#prepareValueBytesOnAcquire(PrepareValueBytes)}.
+ * to do if the key is absent in the map.
* By default, value bytes are just zeroed out, no default value, either provided for key or
* constant, is put for the absent key.
*
diff --git a/src/main/java/net/openhft/chronicle/map/ChronicleMapBuilder.java b/src/main/java/net/openhft/chronicle/map/ChronicleMapBuilder.java
index 5dcee5383..4426d3718 100755
--- a/src/main/java/net/openhft/chronicle/map/ChronicleMapBuilder.java
+++ b/src/main/java/net/openhft/chronicle/map/ChronicleMapBuilder.java
@@ -1,48 +1,164 @@
+/*
+ * Copyright 2014 Higher Frequency Trading
+ *
+ * http://www.higherfrequencytrading.com
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
package net.openhft.chronicle.map;
-import net.openhft.chronicle.hash.ChronicleHashBuilder;
-import net.openhft.chronicle.hash.ChronicleHashErrorListener;
-import net.openhft.chronicle.hash.ChronicleHashInstanceConfig;
-import net.openhft.chronicle.hash.StatelessClientConfig;
-import net.openhft.chronicle.hash.replication.SingleChronicleHashReplication;
-import net.openhft.chronicle.hash.replication.TcpTransportAndNetworkConfig;
-import net.openhft.chronicle.hash.replication.TimeProvider;
-import net.openhft.chronicle.hash.serialization.BytesReader;
-import net.openhft.chronicle.hash.serialization.BytesWriter;
-import net.openhft.chronicle.hash.serialization.SizeMarshaller;
-import net.openhft.lang.io.serialization.BytesMarshaller;
-import net.openhft.lang.io.serialization.BytesMarshallerFactory;
-import net.openhft.lang.io.serialization.ObjectFactory;
-import net.openhft.lang.io.serialization.ObjectSerializer;
+import net.openhft.chronicle.hash.*;
+import net.openhft.chronicle.hash.replication.*;
+import net.openhft.chronicle.hash.serialization.*;
+import net.openhft.chronicle.hash.serialization.internal.MetaBytesInterop;
+import net.openhft.chronicle.hash.serialization.internal.MetaBytesWriter;
+import net.openhft.chronicle.hash.serialization.internal.MetaProvider;
+import net.openhft.chronicle.set.ChronicleSetBuilder;
+import net.openhft.lang.Maths;
+import net.openhft.lang.io.Bytes;
+import net.openhft.lang.io.BytesStore;
+import net.openhft.lang.io.DirectStore;
+import net.openhft.lang.io.serialization.*;
+import net.openhft.lang.io.serialization.impl.AllocateInstanceObjectFactory;
+import net.openhft.lang.io.serialization.impl.NewInstanceObjectFactory;
+import net.openhft.lang.io.serialization.impl.VanillaBytesMarshallerFactory;
import net.openhft.lang.model.Byteable;
-import net.openhft.lang.model.DataValueClasses;
+import net.openhft.lang.threadlocal.Provider;
+import net.openhft.lang.threadlocal.ThreadLocalCopies;
import org.jetbrains.annotations.NotNull;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
-import java.io.File;
-import java.io.IOException;
+import java.io.*;
+import java.lang.reflect.Constructor;
import java.net.InetSocketAddress;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+
+import static net.openhft.chronicle.map.Objects.builderEquals;
+import static net.openhft.chronicle.map.VanillaChronicleMap.MAX_ENTRY_OVERSIZE_FACTOR;
+import static net.openhft.lang.model.DataValueGenerator.firstPrimitiveFieldType;
/**
- * {@code ChronicleMapBuilder} is intended to be used to create {@link ChronicleMap}s
+ * {@code ChronicleMapBuilder} manages {@link ChronicleMap} configurations; could be used as a
+ * classic builder and/or factory. This means that in addition to the standard builder usage
+ * pattern:
+ * i. e. created {@code ChronicleMap} instances don't depend on the builder.
+ *
+ *
{@code ChronicleMapBuilder} is mutable, see a note in {@link ChronicleHashBuilder} interface
+ * documentation.
+ *
+ *
Later in this documentation, "ChronicleMap" means "ChronicleMaps, created by {@code
+ * ChronicleMapBuilder}", unless specified different, because theoretically someone might provide
+ * {@code ChronicleMap} implementations with completely different properties.
+ *
+ *
{@code ChronicleMap} ("ChronicleMaps, created by {@code ChronicleMapBuilder}") currently
+ * doesn't support resizing. That is why you should always configure {@linkplain
+ * #entries(long) number of entries} you are going to insert into the created map at most.
+ * See {@link #entries(long)} method documentation for more information on this.
+ *
+ *
{@code ChronicleMap} allocates memory by equally sized chunks. This size is called {@linkplain
+ * #entrySize(int) entry size}, you are strongly recommended to configure it to achieve least memory
+ * consumption and best speed. See {@link #entrySize(int)} method documentation for more information
+ * on this.
*
- * @param key type of the maps, created by this builder
- * @param value type of the maps, created by this builder
- * @see AbstractChronicleMapBuilder
- * @see net.openhft.chronicle.hash.ChronicleHashBuilder
+ * @param key type of the maps, produced by this builder
+ * @param value type of the maps, produced by this builder
+ * @see ChronicleMap
+ * @see ChronicleSetBuilder
*/
-public final class ChronicleMapBuilder implements ChronicleMapBuilderI, ChronicleHashBuilder, ChronicleMapBuilder> {
+public final class ChronicleMapBuilder implements Cloneable,
+ ChronicleHashBuilder, ChronicleMapBuilder>, Serializable {
- final ChronicleMapBuilderI delegate;
+ static final byte UDP_REPLICATION_MODIFICATION_ITERATOR_ID = (byte) 128;
+ private static final int DEFAULT_KEY_OR_VALUE_SIZE = 120;
+ private static final int MAX_SEGMENTS = (1 << 30);
+ private static final int MAX_SEGMENTS_TO_CHAISE_COMPACT_MULTI_MAPS = (1 << 20);
+ private static final Logger LOG =
+ LoggerFactory.getLogger(ChronicleMapBuilder.class.getName());
+ static final long DEFAULT_STATELESS_CLIENT_TIMEOUT = TimeUnit.SECONDS.toMillis(10);
+
+ private static final StringBuilder EMTRY_STRING_BUILDER = new StringBuilder();
+
+ SerializationBuilder keyBuilder;
+ SerializationBuilder valueBuilder;
+
+ private String name;
+
+ // used when configuring the number of segments.
+ private int minSegments = -1;
+ private int actualSegments = -1;
+ // used when reading the number of entries per
+ private long actualEntriesPerSegment = -1L;
+ private int keySize = 0;
+ private K sampleKey;
+ private int valueSize = 0;
+ private V sampleValue;
+ private int entrySize = 0;
+ private int maxEntryOversizeFactor = -1;
+ private Alignment alignment = null;
+ private long entries = 1 << 20; // 1 million by default
+ private long lockTimeOut = 20000;
+ private TimeUnit lockTimeOutUnit = TimeUnit.MILLISECONDS;
+ private int metaDataBytes = 0;
+ private ChronicleHashErrorListener errorListener = ChronicleHashErrorListeners.logging();
+ private boolean putReturnsNull = false;
+ private boolean removeReturnsNull = false;
+
+ // replication
+ private TimeProvider timeProvider = TimeProvider.SYSTEM;
+ private BytesMarshallerFactory bytesMarshallerFactory;
+ private ObjectSerializer objectSerializer;
+ private MapEventListener eventListener = null;
+ private BytesMapEventListener bytesEventListener = null;
+ private V defaultValue = null;
+ private DefaultValueProvider defaultValueProvider = null;
+ private PrepareValueBytes prepareValueBytes = null;
+
+ private SingleChronicleHashReplication singleHashReplication = null;
+ private InetSocketAddress[] pushToAddresses;
+
+ ChronicleMapBuilder(Class keyClass, Class valueClass) {
+ keyBuilder = new SerializationBuilder<>(keyClass, SerializationBuilder.Role.KEY);
+ valueBuilder = new SerializationBuilder<>(valueClass, SerializationBuilder.Role.VALUE);
+
+ if (CharSequence.class == valueClass)
+ defaultValue = (V) "";
+ if (StringBuilder.class == valueClass)
+ defaultValue = (V) EMTRY_STRING_BUILDER;
+ }
/**
* Returns a new {@code ChronicleMapBuilder} instance which is able to {@linkplain #create()
* create} maps with the specified key and value classes.
*
- *
{@code ChronicleMapBuilder} analyzes provided key and value classes and automatically
- * chooses the most specific internal builder.
- *
* @param keyClass class object used to infer key type and discover it's properties via
* reflection
* @param valueClass class object used to infer value type and discover it's properties via
@@ -53,159 +169,153 @@ public final class ChronicleMapBuilder implements ChronicleMapBuilderI ChronicleMapBuilder of(
@NotNull Class keyClass, @NotNull Class valueClass) {
- if (valueClass.isEnum())
- return new ChronicleMapBuilder(OnHeapUpdatableChronicleMapBuilder.of(keyClass, valueClass));
+ return new ChronicleMapBuilder<>(keyClass, valueClass);
+ }
- if (keyClass.isInterface() && !builtInType(keyClass))
- keyClass = DataValueClasses.directClassFor(keyClass);
+ private static long roundUpMapHeaderSize(long headerSize) {
+ long roundUp = (headerSize + 127L) & ~127L;
+ if (roundUp - headerSize < 64)
+ roundUp += 128;
+ return roundUp;
+ }
- if ((valueClass.isInterface() && !builtInType(valueClass))) {
- valueClass = DataValueClasses.directClassFor(valueClass);
- } else if (!offHeapReference(valueClass)) {
- return new ChronicleMapBuilder(OnHeapUpdatableChronicleMapBuilder.of(keyClass, valueClass));
+ private static void checkSegments(int segments) {
+ if (segments <= 0 || segments > MAX_SEGMENTS)
+ throw new IllegalArgumentException("segments should be positive, " +
+ segments + " given");
+ if (segments > MAX_SEGMENTS)
+ throw new IllegalArgumentException("Max segments is " + MAX_SEGMENTS + ", " +
+ segments + " given");
+ }
+
+ private static long divideUpper(long dividend, long divisor) {
+ return ((dividend - 1L) / divisor) + 1L;
+ }
+
+ private static String pretty(int value) {
+ return value > 0 ? value + "" : "not configured";
+ }
+
+ private static String pretty(Object obj) {
+ return obj != null ? obj + "" : "not configured";
+ }
+
+ public ChronicleMapBuilder pushTo(InetSocketAddress... addresses) {
+ this.pushToAddresses = addresses;
+ return this;
+ }
+
+ @Override
+ public ChronicleMapBuilder clone() {
+ try {
+ @SuppressWarnings("unchecked")
+ ChronicleMapBuilder result =
+ (ChronicleMapBuilder) super.clone();
+ result.keyBuilder = keyBuilder.clone();
+ result.valueBuilder = valueBuilder.clone();
+ return result;
+ } catch (CloneNotSupportedException e) {
+ throw new AssertionError(e);
}
-
- ChronicleMapBuilderI builder = new OffHeapUpdatableChronicleMapBuilder(keyClass, valueClass);
- return new ChronicleMapBuilder(builder);
- }
-
-
-
- static boolean builtInType(Class clazz) {
- return clazz.getClassLoader() == Class.class.getClassLoader();
- }
-
- ChronicleMapBuilder(ChronicleMapBuilderI delegate) {
- this.delegate = delegate;
- }
-
- static boolean offHeapReference(Class valueClass) {
- return Byteable.class.isAssignableFrom(valueClass);
}
/**
- * {@inheritDoc} With respect to {@linkplain #entryAndValueAlignment(Alignment) alignment}.
+ * {@inheritDoc}
+ *
+ *
Example: if keys in your map(s) are English words in {@link String} form, keys size 10
+ * (a bit more than average English word length) would be a good choice:
{@code
+ * ChronicleMap wordFrequencies = ChronicleMapBuilder
+ * .of(String.class, LongValue.class)
+ * .entries(50000)
+ * .keySize(10)
+ * // shouldn't specify valueSize(), because it is statically known
+ * .create();}
+ * (Note that 10 is chosen as key size in bytes despite strings in Java are UTF-16 encoded (and
+ * each character takes 2 bytes on-heap), because default off-heap {@link String} encoding is
+ * UTF-8 in {@code ChronicleMap}.)
*
- *
Note that the actual entrySize will be aligned to 4 (default {@linkplain
- * #entryAndValueAlignment(Alignment) entry alignment}). I. e. if you set entry size to 30, and
- * entry alignment is set to {@link Alignment#OF_4_BYTES}, the actual entry size will be 32 (30
- * aligned to 4 bytes).
- *
- * @see #entryAndValueAlignment(Alignment) // * @see #entries(long)
+ * @throws IllegalStateException {@inheritDoc}
+ * @throws IllegalArgumentException {@inheritDoc}
+ * @see #constantKeySizeBySample(Object)
+ * @see #valueSize(int)
+ * @see #entrySize(int)
*/
- @Override
- public ChronicleMapBuilder entrySize(int entrySize) {
- delegate.entrySize(entrySize);
- return this;
- }
-
- /**
- * Configures alignment strategy of address in memory of entries and independently of address in
- * memory of values within entries in ChronicleMaps, created by this builder.
- *
- *
Useful when values of the map are updated intensively, particularly fields with volatile
- * access, because it doesn't work well if the value crosses cache lines. Also, on some
- * (nowadays rare) architectures any misaligned memory access is more expensive than aligned.
- *
- *
Note that {@linkplain #entrySize(int) entry size} will be aligned according to this
- * alignment. I. e. if you set {@code entrySize(20)} and {@link Alignment#OF_8_BYTES}, actual
- * entry size will be 24 (20 aligned to 8 bytes).
- *
- *
Default is {@link Alignment#OF_4_BYTES} for Byteable values.
- *
- * @param alignment the new alignment of the maps constructed by this builder
- * @return this {@code ChronicleMapOnHeapUpdatableBuilder} back
- */
- @Override
- public ChronicleMapBuilder entryAndValueAlignment(Alignment alignment) {
- delegate.entryAndValueAlignment(alignment);
- return this;
- }
-
- /**
- * {@inheritDoc} Also, it overrides any previous {@link #prepareValueBytesOnAcquire}
- * configuration to this {@code ChronicleMapBuilder}.
- *
- *
By default, the default value is not specified, default {@linkplain
- * #prepareValueBytesOnAcquire prepare value bytes routine} is specified instead.
- *
- * @see #defaultValueProvider(DefaultValueProvider)
- * @see #prepareValueBytesOnAcquire(PrepareValueBytes)
- */
- @Override
- public ChronicleMapBuilder defaultValue(V defaultValue) {
- delegate.defaultValue(defaultValue);
- return this;
- }
-
- /**
- * Configures the procedure which is called on the bytes, which later the returned value is
- * pointing to, if the key is absent, on {@link ChronicleMap#acquireUsing(Object, Object)
- * acquireUsing()} call on maps, created by this builder. See {@link PrepareValueBytes} for more
- * information.
- *
- *
The default preparation callback zeroes out the value bytes.
- *
- * @param prepareValueBytes what to do with the value bytes before assigning them into the
- * {@link net.openhft.lang.model.Byteable} value to return from {@code
- * acquireUsing()} call
- * @return this builder back
- * @see PrepareValueBytes
- * @see #defaultValue(Object)
- * @see #defaultValueProvider(DefaultValueProvider)
- */
- @Override
- public ChronicleMapBuilder prepareValueBytesOnAcquire(@NotNull PrepareValueBytes prepareValueBytes) {
- delegate.prepareValueBytesOnAcquire(prepareValueBytes);
- return this;
- }
-
@Override
public ChronicleMapBuilder keySize(int keySize) {
- delegate.keySize(keySize);
+ checkSizeIsNotStaticallyKnown(keyBuilder);
+ if (keySize <= 0)
+ throw new IllegalArgumentException("Key size must be positive");
+ this.keySize = keySize;
return this;
}
+ /**
+ * {@inheritDoc}
+ *
+ *
+ * @see #keySize(int)
+ * @see #constantValueSizeBySample(Object)
+ */
@Override
public ChronicleMapBuilder constantKeySizeBySample(K sampleKey) {
- delegate.constantKeySizeBySample(sampleKey);
+ this.sampleKey = sampleKey;
return this;
}
+ private int keySize() {
+ return keyOrValueSize(keySize, keyBuilder);
+ }
+
/**
* Configures the optimal number of bytes, taken by serialized form of values, put into maps,
* created by this builder. If value size is always the same, call {@link
* #constantValueSizeBySample(Object)} method instead of this one.
- *
- *
If value is a boxed primitive type, i. e. if value size is known statically, it is
- * automatically accounted and shouldn't be specified by user.
- *
+ *
+ *
If values are of boxed primitive type or {@link Byteable} subclass, i. e. if value size is
+ * known statically, it is automatically accounted and shouldn't be specified by user.
+ *
*
If value size varies moderately, specify the size higher than average, but lower than the
* maximum possible, to minimize average memory overuse. If value size varies in a wide range,
* it's better to use {@linkplain #entrySize(int) entry size} in "chunk" mode and configure it
* directly.
*
* @param valueSize number of bytes, taken by serialized form of values
- * @return this {@code ChronicleMapOnHeapUpdatableBuilder} back
+ * @return this builder back
+ * @throws IllegalStateException if value size is known statically and shouldn't be
+ * configured by user
+ * @throws IllegalArgumentException if the given {@code valueSize} is non-positive
* @see #constantValueSizeBySample(Object)
* @see #keySize(int)
* @see #entrySize(int)
*/
- @Override
public ChronicleMapBuilder valueSize(int valueSize) {
- delegate.valueSize(valueSize);
+ checkSizeIsNotStaticallyKnown(valueBuilder);
+ if (valueSize <= 0)
+ throw new IllegalArgumentException("Value size must be positive");
+ this.valueSize = valueSize;
return this;
}
+ private static void checkSizeIsNotStaticallyKnown(SerializationBuilder builder) {
+ if (builder.sizeIsStaticallyKnown)
+ throw new IllegalStateException("Size of type " + builder.eClass +
+ " is statically known and shouldn't be specified manually");
+ }
+
/**
* Configures the constant number of bytes, taken by serialized form of values, put into maps,
* created by this builder. This is done by providing the {@code sampleValue}, all values should
* take the same number of bytes in serialized form, as this sample object.
- *
- *
If values are of boxed primitive type or {@link net.openhft.lang.model.Byteable} subclass,
- * i. e. if value size is known statically, it is automatically accounted and this method
- * shouldn't be called.
- *
+ *
+ *
If values are of boxed primitive type or {@link Byteable} subclass, i. e. if value size is
+ * known statically, it is automatically accounted and this method shouldn't be called.
+ *
*
If value size varies, method {@link #valueSize(int)} or {@link #entrySize(int)} should be
* called instead of this one.
*
@@ -214,15 +324,481 @@ public final class ChronicleMapBuilder implements ChronicleMapBuilderI constantValueSizeBySample(@NotNull V sampleValue) {
- delegate.constantValueSizeBySample(sampleValue);
+ public ChronicleMapBuilder constantValueSizeBySample(V sampleValue) {
+ this.sampleValue = sampleValue;
return this;
}
+ int valueSize() {
+ return keyOrValueSize(valueSize, valueBuilder);
+ }
+
+ private int keyOrValueSize(int configuredSize, SerializationBuilder builder) {
+ if (configuredSize > 0)
+ return configuredSize;
+ if (builder.constantSizeMarshaller())
+ return builder.pseudoReadConstantSize();
+ return DEFAULT_KEY_OR_VALUE_SIZE;
+ }
+
/**
* {@inheritDoc}
+ *
+ *
In fully default case you can expect entry size to be about 256 bytes. But it is strongly
+ * recommended always to configure {@linkplain #keySize(int) key size} and {@linkplain
+ * #valueSize(int) value size}, if they couldn't be derived statically.
+ *
+ *
If entry size is not configured explicitly by calling this method, it is computed based on
+ * {@linkplain #metaDataBytes(int) meta data bytes}, plus {@linkplain #keySize(int) key size},
+ * plus {@linkplain #valueSize(int) value size}, plus a few bytes required by implementations,
+ * with respect to {@linkplain #entryAndValueAlignment(Alignment) alignment}.
+ *
+ *
Note that the actual entrySize will be aligned by the configured {@linkplain
+ * #entryAndValueAlignment(Alignment) entry and value alignment}. I. e. if you set entry size to
+ * 30, and entry alignment is set to {@link Alignment#OF_4_BYTES}, the actual entry size will be
+ * 32 (30 aligned to 4 bytes).
*
+ * @throws IllegalStateException is sizes of both keys and values of maps created by this
+ * builder are statically known, hence entry size shouldn't be
+ * configured by user
+ * @see #entryAndValueAlignment(Alignment)
+ * @see #entries(long)
+ * @see #maxEntryOversizeFactor(int)
+ */
+ @Override
+ public ChronicleMapBuilder entrySize(int entrySize) {
+ if (keyBuilder.sizeIsStaticallyKnown && valueBuilder.sizeIsStaticallyKnown) {
+ throw new IllegalStateException("Sizes of key type: " + keyBuilder.eClass + " and " +
+ "value type: " + valueBuilder.eClass + " are both statically known, " +
+ "so entry size shouldn't be specified manually");
+ }
+ if (entrySize <= 0)
+ throw new IllegalArgumentException("Entry Size must be positive");
+ this.entrySize = entrySize;
+ return this;
+ }
+
+ int entrySize(boolean replicated) {
+ if (entrySize > 0)
+ return entryAndValueAlignment().alignSize(entrySize);
+ int size = metaDataBytes;
+ int keySize = keySize();
+ size += keyBuilder.sizeMarshaller().sizeEncodingSize(keySize);
+ size += keySize;
+ if (replicated)
+ size += ReplicatedChronicleMap.ADDITIONAL_ENTRY_BYTES;
+ int valueSize = valueSize();
+ size += valueBuilder.sizeMarshaller().sizeEncodingSize(valueSize);
+ size = entryAndValueAlignment().alignSize(size);
+ size += valueSize;
+ // Some cache line heuristics
+ for (int i = 1; i <= 4; i++) {
+ int bound = i * 64;
+ // Not more than 5% oversize.
+ // DEFAULT_KEY_OR_VALUE_SIZE and this heuristic are specially adjusted to produce
+ // entry size 256 -- the default value prior to keySize and valueSize -- when key and
+ // value sizes are both not static nor configured, in both vanilla and replicated modes.
+ if (size < bound && (bound - size) <= bound / 20) {
+ size = bound;
+ break;
+ }
+ }
+ return entryAndValueAlignment().alignSize(size);
+ }
+
+ @Override
+ public ChronicleMapBuilder maxEntryOversizeFactor(int maxEntryOversizeFactor) {
+ if (maxEntryOversizeFactor < 1)
+ throw new IllegalArgumentException("maxEntryOversizeFactor should be more than 1, " +
+ maxEntryOversizeFactor + " given");
+ this.maxEntryOversizeFactor = maxEntryOversizeFactor;
+ return this;
+ }
+
+ private void checkMaxEntryOversizeFactorIfReplicated(boolean replicated) {
+ if (replicated && maxEntryOversizeFactor > MAX_ENTRY_OVERSIZE_FACTOR)
+ throw new IllegalStateException("As of now Replicated Chornicle Map doesn't support " +
+ "more than " + MAX_ENTRY_OVERSIZE_FACTOR + " entry oversize factor, " +
+ maxEntryOversizeFactor + " is configured");
+ }
+
+ int maxEntryOversizeFactor(boolean replicated) {
+ if (maxEntryOversizeFactor < 0) {
+ if (keyBuilder.constantSizeMarshaller() && valueBuilder.constantSizeMarshaller()) {
+ return 1;
+ } else {
+ if (replicated) {
+ return MAX_ENTRY_OVERSIZE_FACTOR;
+ } else {
+ return Integer.MAX_VALUE;
+ }
+ }
+ }
+ return maxEntryOversizeFactor;
+ }
+
+ /**
+ * Configures alignment strategy of address in memory of entries and independently of address in
+ * memory of values within entries in ChronicleMaps, created by this builder.
+ *
+ *
Useful when values of the map are updated intensively, particularly fields with volatile
+ * access, because it doesn't work well if the value crosses cache lines. Also, on some
+ * (nowadays rare) architectures any misaligned memory access is more expensive than aligned.
+ *
+ *
Note that {@linkplain #entrySize(int) entry size} will be aligned according to this
+ * alignment. I. e. if you set {@code entrySize(20)} and {@link Alignment#OF_8_BYTES}, actual
+ * entry size will be 24 (20 aligned to 8 bytes).
+ *
+ *
If values couldn't reference off-heap memory (i. e. it is not {@link Byteable} or "data
+ * value generated"), alignment configuration makes no sense and forbidden.
+ *
+ *
Default is {@link Alignment#NO_ALIGNMENT} if values couldn't reference off-heap memory,
+ * otherwise chosen heuristically (configure explicitly for being sure and to compare
+ * performance in your case).
+ *
+ * @param alignment the new alignment of the maps constructed by this builder
+ * @return this {@code ChronicleMapOnHeapUpdatableBuilder} back
+ * @throws IllegalStateException if values of maps, created by this builder, couldn't reference
+ * off-heap memory
+ */
+ public ChronicleMapBuilder entryAndValueAlignment(Alignment alignment) {
+ this.alignment = alignment;
+ checkAlignmentOnlyIfValuesPossiblyReferenceOffHeap();
+ return this;
+ }
+
+ private void checkAlignmentOnlyIfValuesPossiblyReferenceOffHeap() {
+ if (!valueBuilder.possibleOffHeapReferences() &&
+ (alignment == Alignment.OF_4_BYTES || alignment == Alignment.OF_8_BYTES))
+ throw new IllegalStateException("Entry and value alignment should be configured only " +
+ "if values might point to off-heap memory");
+ }
+
+ Alignment entryAndValueAlignment() {
+ if (alignment != null)
+ return alignment;
+ Class firstPrimitiveFieldType = firstPrimitiveFieldType(valueBuilder.eClass);
+ if (firstPrimitiveFieldType == long.class || firstPrimitiveFieldType == double.class)
+ return Alignment.OF_8_BYTES;
+ if (firstPrimitiveFieldType == int.class || firstPrimitiveFieldType == float.class)
+ return Alignment.OF_4_BYTES;
+ return Alignment.NO_ALIGNMENT;
+ }
+
+ /**
+ * Package-private for tests
+ */
+
+
+ @Override
+ public ChronicleMapBuilder entries(long entries) {
+ if (entries <= 0L)
+ throw new IllegalArgumentException("Entries should be positive, " + entries + " given");
+ this.entries = entries;
+ return this;
+ }
+
+ long entries() {
+ return entries;
+ }
+
+ @Override
+ public ChronicleMapBuilder actualEntriesPerSegment(long actualEntriesPerSegment) {
+ if (actualEntriesPerSegment <= 0L)
+ throw new IllegalArgumentException("entries per segment should be positive, " +
+ actualEntriesPerSegment + " given");
+ if (tooManyEntriesPerSegment(actualEntriesPerSegment))
+ throw new IllegalArgumentException("max entries per segment is " +
+ MultiMapFactory.MAX_CAPACITY + ", " + actualEntriesPerSegment + " given");
+ this.actualEntriesPerSegment = actualEntriesPerSegment;
+ return this;
+ }
+
+ private boolean tooManyEntriesPerSegment(long entriesPerSegment) {
+ return entriesPerSegment > MultiMapFactory.MAX_CAPACITY;
+ }
+
+ long actualEntriesPerSegment() {
+ if (actualEntriesPerSegment > 0L)
+ return actualEntriesPerSegment;
+ int actualSegments = actualSegments();
+ long actualEntries = totalEntriesIfPoorDistribution(actualSegments);
+ long actualEntriesPerSegment = divideUpper(actualEntries, actualSegments);
+ if (tooManyEntriesPerSegment(actualEntriesPerSegment))
+ throw new IllegalStateException("max entries per segment is " +
+ MultiMapFactory.MAX_CAPACITY + " configured entries() and " +
+ "actualSegments() so that there should be " + actualEntriesPerSegment +
+ " entries per segment");
+ return actualEntriesPerSegment;
+ }
+
+ private long totalEntriesIfPoorDistribution(int segments) {
+ if (segments == 1)
+ return entries;
+ long entries = this.entries;
+ // check if the number of entries is small compared with the number of segments.
+ long s3 = (long) Math.min(8, segments) * Math.min(32, segments) * Math.min(128, segments);
+ if (entries * 4 <= s3)
+ entries *= 1.8;
+ else if (entries <= s3)
+ entries *= 1.45;
+ else if (entries <= s3 * 4)
+ entries *= 1.33;
+ else if (entries <= s3 * 8)
+ entries *= 1.22;
+ else if (entries <= s3 * 16)
+ entries *= 1.15;
+ else
+ entries *= 1.1;
+ return Math.min(segments * this.entries,
+ entries + 4 * segments + 8);
+ }
+
+ @Override
+ public ChronicleMapBuilder minSegments(int minSegments) {
+ checkSegments(minSegments);
+ this.minSegments = minSegments;
+ return this;
+ }
+
+ int minSegments() {
+ return Math.max(estimateSegments(), minSegments);
+ }
+
+ private int estimateSegments() {
+ return (int) Math.min(Maths.nextPower2(entries / 32, 1), estimateSegmentsBasedOnSize());
+ }
+
+ private int estimateSegmentsBasedOnSize() {
+ // based on entries with multimap of 100 bytes.
+ int segmentsForEntries = estimateSegmentsForEntries(entries);
+ return entrySize >= 1000000
+ ? segmentsForEntries * 16
+ : entrySize >= 100000
+ ? segmentsForEntries * 8
+ : entrySize >= 10000
+ ? segmentsForEntries * 4
+ : entrySize >= 1000
+ ? segmentsForEntries * 2
+ : segmentsForEntries;
+ }
+
+ private static int estimateSegmentsForEntries(long size) {
+ if (size > 200 << 20)
+ return 256;
+ if (size >= 1 << 20)
+ return 128;
+ if (size >= 128 << 10)
+ return 64;
+ if (size >= 16 << 10)
+ return 32;
+ if (size >= 4 << 10)
+ return 16;
+ if (size >= 1 << 10)
+ return 8;
+ return 1;
+ }
+
+ @Override
+ public ChronicleMapBuilder actualSegments(int actualSegments) {
+ checkSegments(actualSegments);
+ this.actualSegments = actualSegments;
+ return this;
+ }
+
+ int actualSegments() {
+ if (actualSegments > 0)
+ return actualSegments;
+ long shortMMapSegments = trySegments(MultiMapFactory.I16_MAX_CAPACITY,
+ MAX_SEGMENTS_TO_CHAISE_COMPACT_MULTI_MAPS);
+ if (shortMMapSegments > 0L)
+ return (int) shortMMapSegments;
+ long intMMapSegments = trySegments(MultiMapFactory.MAX_CAPACITY, MAX_SEGMENTS);
+ if (intMMapSegments > 0L)
+ return (int) intMMapSegments;
+ throw new IllegalStateException("Max segments is " + MAX_SEGMENTS + ", configured so much" +
+ " entries() that builder automatically decided to use " +
+ (-intMMapSegments) + " segments");
+ }
+
+ private long trySegments(long maxSegmentCapacity, int maxSegments) {
+ long segments = divideUpper(totalEntriesIfPoorDistribution(minSegments()),
+ maxSegmentCapacity);
+ segments = Maths.nextPower2(Math.max(segments, minSegments()), 1L);
+ return segments <= maxSegments ? segments : -segments;
+ }
+
+ int segmentHeaderSize() {
+ int segments = actualSegments();
+ // reduce false sharing unless we have a lot of segments.
+ return segments <= 16 * 1024 ? 64 : 32;
+ }
+
+ MultiMapFactory multiMapFactory() {
+ return MultiMapFactory.forCapacity(actualEntriesPerSegment());
+ }
+
+ public ChronicleMapBuilder lockTimeOut(long lockTimeOut, TimeUnit unit) {
+ this.lockTimeOut = lockTimeOut;
+ lockTimeOutUnit = unit;
+ return this;
+ }
+
+ long lockTimeOut(TimeUnit unit) {
+ return unit.convert(lockTimeOut, lockTimeOutUnit);
+ }
+
+ @Override
+ public ChronicleMapBuilder errorListener(ChronicleHashErrorListener errorListener) {
+ this.errorListener = errorListener;
+ return this;
+ }
+
+ ChronicleHashErrorListener errorListener() {
+ return errorListener;
+ }
+
+ /**
+ * Configures if the maps created by this builder should return {@code null} instead of previous
+ * mapped values on {@link ChronicleMap#put(Object, Object) ChornicleMap.put(key, value)}
+ * calls.
+ *
+ *
{@link Map#put(Object, Object) Map.put()} returns the previous value, functionality which
+ * is rarely used but fairly cheap for {@link HashMap}. In the case, for an off heap collection,
+ * it has to create a new object and deserialize the data from off-heap memory. It's expensive
+ * for something you probably don't use.
+ *
+ *
By default, of cause, {@code ChronicleMap} conforms the general {@code Map} contract and
+ * returns the previous mapped value on {@code put()} calls.
+ *
+ * @param putReturnsNull {@code true} if you want {@link ChronicleMap#put(Object, Object)
+ * ChronicleMap.put()} to not return the value that was replaced but
+ * instead return {@code null}
+ * @return this builder back
+ * @see #removeReturnsNull(boolean)
+ */
+ public ChronicleMapBuilder putReturnsNull(boolean putReturnsNull) {
+ this.putReturnsNull = putReturnsNull;
+ return this;
+ }
+
+ boolean putReturnsNull() {
+ return putReturnsNull;
+ }
+
+ /**
+ * Configures if the maps created by this builder should return {@code null} instead of the last
+ * mapped value on {@link ChronicleMap#remove(Object) ChronicleMap.remove(key)} calls.
+ *
+ *
{@link Map#remove(Object) Map.remove()} returns the previous value, functionality which is
+ * rarely used but fairly cheap for {@link HashMap}. In the case, for an off heap collection, it
+ * has to create a new object and deserialize the data from off-heap memory. It's expensive for
+ * something you probably don't use.
+ *
{@code Map map =
* ChronicleMapBuilder.of(Key.class, Value.class)
* .entries(1_000_000)
@@ -230,37 +806,52 @@ public final class ChronicleMapBuilder implements ChronicleMapBuilderI
+ *
+ *
This serializer is used to serialize both keys and values, if they both require this:
+ * loosely typed, nullable, and custom {@linkplain #keyMarshaller(BytesMarshaller) key} and
+ * {@linkplain #valueMarshaller(BytesMarshaller) value} marshallers are not configured.
*/
@Override
- public ChronicleMapBuilder objectSerializer(@NotNull ObjectSerializer objectSerializer) {
- delegate.objectSerializer(objectSerializer);
+ public ChronicleMapBuilder objectSerializer(ObjectSerializer objectSerializer) {
+ this.objectSerializer = objectSerializer;
+ return this;
+ }
+
+ @Override
+ public ChronicleMapBuilder keyMarshaller(
+ @NotNull BytesMarshaller keyMarshaller) {
+ keyBuilder.marshaller(keyMarshaller);
+ return this;
+ }
+
+ @Override
+ public ChronicleMapBuilder keyMarshallers(
+ @NotNull BytesWriter keyWriter, @NotNull BytesReader keyReader) {
+ keyBuilder.writer(keyWriter);
+ keyBuilder.reader(keyReader);
+ return this;
+ }
+
+ @Override
+ public ChronicleMapBuilder keySizeMarshaller(@NotNull SizeMarshaller keySizeMarshaller) {
+ keyBuilder.sizeMarshaller(keySizeMarshaller);
return this;
}
/**
- * {@inheritDoc}
- *
- * @throws IllegalStateException if custom value marshaller is specified or value class is not
- * either {@code BytesMarshallable} or {@code Externalizable}
+ * @throws IllegalStateException {@inheritDoc}
+ * @see #valueDeserializationFactory(ObjectFactory)
*/
-
@Override
- public ChronicleMapBuilder valueDeserializationFactory(@NotNull ObjectFactory valueDeserializationFactory) {
- delegate.valueDeserializationFactory(valueDeserializationFactory);
+ public ChronicleMapBuilder keyDeserializationFactory(
+ @NotNull ObjectFactory keyDeserializationFactory) {
+ keyBuilder.factory(keyDeserializationFactory);
return this;
}
- /**
- * {@inheritDoc}
- *
- *
By default, default value provider is not specified, {@link #defaultValue(Object) default
- * value} is specified instead.
- *
- * @see #defaultValue(Object)
- */
@Override
- public ChronicleMapBuilder defaultValueProvider(@NotNull DefaultValueProvider defaultValueProvider) {
- delegate.defaultValueProvider(defaultValueProvider);
+ public ChronicleMapBuilder immutableKeys() {
+ keyBuilder.instancesAreMutable(false);
return this;
}
@@ -271,15 +862,13 @@ public final class ChronicleMapBuilder implements ChronicleMapBuilderI valueMarshaller(
@NotNull BytesMarshaller valueMarshaller) {
- delegate.valueMarshaller(valueMarshaller);
+ valueBuilder.marshaller(valueMarshaller);
return this;
}
@@ -287,243 +876,440 @@ public final class ChronicleMapBuilder implements ChronicleMapBuilderIthe
* section about serialization in ChronicleMap manual for more information.
- *
+ *
*
Configuring marshalling this way results to a little bit more compact in-memory layout of
- * the map, comparing to a single interface configuration: {@link #valueMarshaller(BytesMarshaller)}.
+ * the map, comparing to a single interface configuration: {@link #valueMarshaller(
+ *BytesMarshaller)}.
+ *
+ *
Passing {@link BytesInterop} instead of plain {@link BytesWriter} is, of cause, possible,
+ * but currently pointless for values.
*
- *
Passing {@link net.openhft.chronicle.hash.serialization.BytesInterop} instead of plain
- * {@link net.openhft.chronicle.hash.serialization.BytesWriter} is, of cause, possible, but
- * currently pointless for values.
- *
- * @param valueWriter the new value object → {@link net.openhft.lang.io.Bytes} writer
- * (interop) strategy
- * @param valueReader the new {@link net.openhft.lang.io.Bytes} → value object reader
- * strategy
+ * @param valueWriter the new value object → {@link Bytes} writer (interop) strategy
+ * @param valueReader the new {@link Bytes} → value object reader strategy
* @return this builder back
* @see #valueMarshaller(BytesMarshaller)
+ * @see #valueSizeMarshaller(SizeMarshaller)
+ * @see #keyMarshallers(BytesWriter, BytesReader)
*/
- @Override
- public ChronicleMapBuilder valueMarshallers(@NotNull BytesWriter valueWriter, @NotNull BytesReader valueReader) {
- delegate.valueMarshallers(valueWriter, valueReader);
+ public ChronicleMapBuilder valueMarshallers(
+ @NotNull BytesWriter valueWriter, @NotNull BytesReader valueReader) {
+ valueBuilder.writer(valueWriter);
+ valueBuilder.reader(valueReader);
return this;
}
/**
* Configures the marshaller used to serialize actual value sizes to off-heap memory in maps,
* created by this builder.
- *
- *
Default value size marshaller is so-called {@linkplain net.openhft.chronicle.hash.serialization.SizeMarshallers#stopBit()
- * stop bit encoding marshalling}. If {@linkplain #constantValueSizeBySample(Object) constant
- * value size} is configured, or defaulted if the value type is always constant and {@code
- * ChronicleHashBuilder} implementation knows about it, this configuration takes no effect,
- * because a special {@link net.openhft.chronicle.hash.serialization.SizeMarshaller}
- * implementation, which doesn't actually do any marshalling, and just returns the known
- * constant size on {@link net.openhft.chronicle.hash.serialization.SizeMarshaller#readSize(
- *net.openhft.lang.io.Bytes)} calls, is used instead of any {@code SizeMarshaller} configured
- * using this method.
+ *
+ *
Default value size marshaller is so-called {@linkplain SizeMarshallers#stopBit() stop bit
+ * encoding marshalling}, unless {@link #constantValueSizeBySample(Object)} or the builder
+ * statically knows the value size is constant -- {@link SizeMarshallers#constant(long)} or
+ * equivalents are used by default in these cases.
*
* @param valueSizeMarshaller the new marshaller, used to serialize actual value sizes to
* off-heap memory
* @return this builder back
+ * @see #keySizeMarshaller(SizeMarshaller)
*/
- @Override
- public ChronicleMapBuilder valueSizeMarshaller(@NotNull SizeMarshaller valueSizeMarshaller) {
- delegate.valueSizeMarshaller(valueSizeMarshaller);
- return this;
- }
-
- @Override
- public ChronicleMapBuilder entries(long l) {
- delegate.entries(l);
+ public ChronicleMapBuilder valueSizeMarshaller(
+ @NotNull SizeMarshaller valueSizeMarshaller) {
+ valueBuilder.sizeMarshaller(valueSizeMarshaller);
return this;
}
/**
- * each map can be identified by a name
+ * Configures factory which is used to create a new value instance, if value class is either
+ * {@link Byteable}, {@link BytesMarshallable} or {@link Externalizable} subclass, or value type
+ * is eligible for data value generation, or configured custom value reader is an instance of
+ * {@link DeserializationFactoryConfigurableBytesReader}, in maps, created by this builder.
+ *
+ *
Default value deserialization factory is {@link NewInstanceObjectFactory}, which creates a
+ * new value instance using {@link Class#newInstance()} default constructor. You could provide
+ * an {@link AllocateInstanceObjectFactory}, which uses {@code Unsafe.allocateInstance(Class)}
+ * (you might want to do this for better performance or if you don't want to initialize fields),
+ * or a factory which calls a value class constructor with some arguments, or a factory which
+ * internally delegates to instance pool or {@link ThreadLocal}, to reduce allocations.
*
- * @param name the name of the map
- * @return self
+ * @param valueDeserializationFactory the value factory used to produce instances to deserialize
+ * data in
+ * @return this builder back
+ * @throws IllegalStateException if it is not possible to apply deserialization factory to value
+ * deserializers, currently configured for this builder
+ * @see #keyDeserializationFactory(ObjectFactory)
*/
- @Override
- public ChronicleMapBuilder name(String name) {
- delegate.name(name);
+ public ChronicleMapBuilder valueDeserializationFactory(
+ @NotNull ObjectFactory valueDeserializationFactory) {
+ valueBuilder.factory(valueDeserializationFactory);
return this;
}
- @Override
- public String name() {
- return delegate.name();
- }
-
- @Override
- public ChronicleMapBuilder replication(SingleChronicleHashReplication withId) {
- delegate.replication(withId);
+ public ChronicleMapBuilder eventListener(MapEventListener eventListener) {
+ this.eventListener = eventListener;
return this;
}
- @Override
- public ChronicleMapBuilder pushTo(InetSocketAddress... addresses) {
- delegate.pushTo(addresses);
+ MapEventListener eventListener() {
+ return eventListener;
+ }
+
+ public ChronicleMapBuilder bytesEventListener(BytesMapEventListener eventListener) {
+ this.bytesEventListener = eventListener;
return this;
}
- @Override
- public ChronicleMap create() {
- return delegate.create();
+ BytesMapEventListener bytesEventListener() {
+ return bytesEventListener;
}
- @Override
- public ChronicleMapBuilder actualSegments(int actualSegments) {
- delegate.actualSegments(actualSegments);
+ /**
+ * Specifies the value to be put for each key queried in {@link ChronicleMap#acquireUsing
+ * acquireUsing()} method, if the key is absent in the map, created by this builder.
+ *
+ *
This configuration overrides any previous {@link #defaultValueProvider(
+ *DefaultValueProvider)} and {@link #prepareDefaultValueBytes(PrepareValueBytes)}
+ * configurations to this builder.
+ *
+ * @param defaultValue the default value to be put to the map for absent keys during {@code
+ * acquireUsing()} calls
+ * @return this builder object back
+ * @see #defaultValueProvider(DefaultValueProvider)
+ * @see #prepareDefaultValueBytes(PrepareValueBytes)
+ */
+ public ChronicleMapBuilder defaultValue(V defaultValue) {
+ if (defaultValue == null)
+ throw new IllegalArgumentException("default ChronicleMap value couldn't be null");
+ this.defaultValue = defaultValue;
+ this.defaultValueProvider = null;
+ this.prepareValueBytes = null;
return this;
}
- @Override
- public ChronicleMapBuilder minSegments(int minSegments) {
- delegate.minSegments(minSegments);
+ /**
+ * Specifies the function to obtain a value for the key during {@link ChronicleMap#acquireUsing
+ * acquireUsing()} calls, if the key is absent in the map, created by this builder.
+ *
+ *
This configuration overrides any previous {@link #defaultValue(Object)} and {@link
+ * #prepareDefaultValueBytes(PrepareValueBytes)} configurations to this builder.
+ *
+ * @param defaultValueProvider the strategy to obtain a default value by the absent key
+ * @return this builder object back
+ * @see #defaultValue(Object)
+ * @see #prepareDefaultValueBytes(PrepareValueBytes)
+ */
+ public ChronicleMapBuilder defaultValueProvider(
+ @NotNull DefaultValueProvider defaultValueProvider) {
+ this.defaultValueProvider = defaultValueProvider;
+ this.defaultValue = null;
+ this.prepareValueBytes = null;
return this;
}
- @Override
- public ChronicleMapBuilder actualEntriesPerSegment(long actualEntriesPerSegment) {
- delegate.actualEntriesPerSegment(actualEntriesPerSegment);
+ /**
+ * Configures the procedure which is called on the bytes, which later the returned value is
+ * pointing to or deserialized from, if the key is absent, on {@link ChronicleMap#acquireUsing
+ * acquireUsing()} call on maps, created by this builder. See {@link PrepareValueBytes} for more
+ * information.
+ *
+ *
This method of value initialization on {@code acquireUsing()} calls is allowed only if
+ * value size is constant. Otherwise you should use either {@link #defaultValue(Object)} or
+ * {@link #defaultValueProvider(DefaultValueProvider)} methods.
+ *
+ *
This configuration overrides any previous {@link #defaultValue(Object)} and {@link
+ * #defaultValueProvider(DefaultValueProvider)} configurations to this builder.
+ *
+ *
The default preparation callback zeroes out the value bytes.
+ *
+ * @param prepareValueBytes what to do with the value bytes before assigning them into the
+ * {@link Byteable} value or deserializing the value from, to return
+ * from {@code acquireUsing()} call
+ * @return this builder back
+ * @throws IllegalStateException is value size is not constant
+ * @see PrepareValueBytes
+ * @see #defaultValue(Object)
+ * @see #defaultValueProvider(DefaultValueProvider)
+ */
+ public ChronicleMapBuilder prepareDefaultValueBytes(
+ @NotNull PrepareValueBytes prepareValueBytes) {
+ this.prepareValueBytes = prepareValueBytes;
+ this.defaultValue = null;
+ this.defaultValueProvider = null;
+ checkPrepareValueBytesOnlyIfConstantValueSize();
return this;
}
+ private void checkPrepareValueBytesOnlyIfConstantValueSize() {
+ if (prepareValueBytes != null && !valueBuilder.constantSizeMarshaller())
+ throw new IllegalStateException("Prepare value bytes could be used only if " +
+ "value size is constant");
+ }
+
+ PrepareValueBytesAsWriter prepareValueBytesAsWriter() {
+ PrepareValueBytes prepareValueBytes = this.prepareValueBytes;
+ if ((prepareValueBytes == null && defaultValueProvider() != null) ||
+ !valueBuilder.constantSizeMarshaller())
+ return null;
+ if (prepareValueBytes == null)
+ prepareValueBytes = new ZeroOutValueBytes<>(valueSize());
+ return new PrepareValueBytesAsWriter<>(prepareValueBytes, valueSize());
+ }
+
+ /**
+ * Non-public because should be called only after {@link #preMapConstruction(boolean)}
+ */
+ DefaultValueProvider defaultValueProvider() {
+ if (defaultValueProvider != null)
+ return defaultValueProvider;
+ if (defaultValue == null)
+ return null;
+ Object originalValueWriter = valueBuilder.interop();
+ Provider writerProvider = Provider.of(originalValueWriter.getClass());
+ ThreadLocalCopies copies = writerProvider.getCopies(null);
+ Object valueWriter = writerProvider.get(copies, originalValueWriter);
+ MetaProvider metaWriterProvider = valueBuilder.metaInteropProvider();
+ copies = metaWriterProvider.getCopies(copies);
+ MetaBytesWriter metaValueWriter = metaWriterProvider.get(copies,
+ valueBuilder.metaInterop(), valueWriter, defaultValue);
+ return new ConstantValueProvider(defaultValue, metaValueWriter, valueWriter);
+ }
+
@Override
- public ChronicleMapBuilder lockTimeOut(long lockTimeOut, TimeUnit unit) {
- delegate.lockTimeOut(lockTimeOut, unit);
+ public ChronicleMapBuilder replication(SingleChronicleHashReplication replication) {
+ this.singleHashReplication = replication;
return this;
}
- @Override
- public ChronicleMapBuilder errorListener(ChronicleHashErrorListener errorListener) {
- delegate.errorListener(errorListener);
- return this;
- }
-
- @Override
- public ChronicleMapBuilder metaDataBytes(int metaDataBytes) {
- delegate.metaDataBytes(metaDataBytes);
- return this;
- }
-
- @Override
- public ChronicleMapBuilder timeProvider(TimeProvider timeProvider) {
- delegate.timeProvider(timeProvider);
- return this;
- }
-
- @Override
- public ChronicleMapBuilder bytesMarshallerFactory(BytesMarshallerFactory bytesMarshallerFactory) {
- delegate.bytesMarshallerFactory(bytesMarshallerFactory);
- return this;
- }
-
- @Override
- public ChronicleMapBuilder keyMarshaller(BytesMarshaller keyMarshaller) {
- delegate.keyMarshaller(keyMarshaller);
- return this;
- }
-
- @Override
- public ChronicleMapBuilder keyMarshallers(BytesWriter keyWriter, BytesReader keyReader) {
- delegate.keyMarshallers(keyWriter, keyReader);
- return this;
- }
-
- @Override
- public ChronicleMapBuilder keySizeMarshaller(SizeMarshaller keySizeMarshaller) {
- delegate.keySizeMarshaller(keySizeMarshaller);
- return this;
- }
-
- @Override
- public ChronicleMapBuilder keyDeserializationFactory(ObjectFactory keyDeserializationFactory) {
- delegate.keyDeserializationFactory(keyDeserializationFactory);
- return this;
- }
-
- @Override
- public ChronicleMapBuilder immutableKeys() {
- delegate.immutableKeys();
- return this;
- }
-
- @Override
- public StatelessClientConfig> statelessClient(InetSocketAddress remoteAddress) {
- return delegate.statelessClient(remoteAddress);
- }
-
- @Override
- public ChronicleHashInstanceConfig> instance() {
- return delegate.instance();
- }
-
@Override
public ChronicleMapBuilder replication(byte identifier) {
- delegate.replication(identifier);
- return this;
+ return replication(SingleChronicleHashReplication.builder().createWithId(identifier));
}
@Override
- public ChronicleMapBuilder replication(byte identifier, TcpTransportAndNetworkConfig tcpTransportAndNetwork) {
- delegate.replication(identifier, tcpTransportAndNetwork);
- return this;
+ public ChronicleMapBuilder replication(
+ byte identifier, TcpTransportAndNetworkConfig tcpTransportAndNetwork) {
+ return replication(SingleChronicleHashReplication.builder()
+ .tcpTransportAndNetwork(tcpTransportAndNetwork).createWithId(identifier));
}
@Override
- public ChronicleMapBuilder clone() {
- delegate.clone();
- return this;
+ public ChronicleHashInstanceConfig> instance() {
+ return new MapInstanceConfig<>(this.clone(), singleHashReplication, null, null, null,
+ new AtomicBoolean(false));
}
@Override
- public ChronicleMapBuilder putReturnsNull(boolean b) {
- delegate.putReturnsNull(b);
- return this;
+ public ChronicleMap createStatelessClient(InetSocketAddress serverAddress)
+ throws IOException {
+ return statelessClient(serverAddress).create();
}
- @Override
- public ChronicleMapBuilder removeReturnsNull(boolean b) {
- delegate.removeReturnsNull(b);
- return this;
- }
-
- @Override
- public ChronicleMapBuilder eventListener(MapEventListener eventListener) {
- delegate.eventListener(eventListener);
- return this;
- }
-
- @Override
- public ChronicleMapBuilderI bytesEventListener(BytesMapEventListener eventListener) {
- delegate.bytesEventListener(eventListener);
- return this;
+ ChronicleMap createStatelessClient(StatelessMapConfig statelessClientConfig)
+ throws IOException {
+ pushingToMapEventListener();
+ preMapConstruction(false);
+ return new StatelessChronicleMap<>(statelessClientConfig, this);
}
@Override
public ChronicleMap createPersistedTo(File file) throws IOException {
- return delegate.createPersistedTo(file);
+ // clone() to make this builder instance thread-safe, because createWithFile() method
+ // computes some state based on configurations, but doesn't synchronize on configuration
+ // changes.
+ return clone().createWithFile(file, singleHashReplication, null);
}
- /**
- * @param bootstrapOnlyLocalEntries if set to true - when a new node joins a TCP replication
- * grid, the new node will be populated with data, only for the
- * nodes that created that data. Otherwise, all the nodes will
- * publish all the data they have ( potentially swamping the
- * new node with duplicates ) however this does guarantee that
- * all the data is replicated over to the new node, and is
- * useful especially in the case that the originating node is
- * not currently running.
- */
- public ChronicleMapBuilder bootstrapOnlyLocalEntries(boolean bootstrapOnlyLocalEntries) {
- delegate.bootstrapOnlyLocalEntries(bootstrapOnlyLocalEntries);
- return this;
+ @Override
+ public StatelessClientConfig> statelessClient(
+ InetSocketAddress remoteAddress) {
+ return new StatelessMapConfig<>(clone(), remoteAddress, DEFAULT_STATELESS_CLIENT_TIMEOUT,
+ null, new AtomicBoolean(false));
}
- public ChronicleMapBuilder disableOversizedEntries() {
- delegate.disableOversizedEntries();
- return this;
+ @Override
+ public ChronicleMap create() {
+ // clone() to make this builder instance thread-safe, because createWithoutFile() method
+ // computes some state based on configurations, but doesn't synchronize on configuration
+ // changes.
+ return clone().createWithoutFile(singleHashReplication, null);
+ }
+
+ ChronicleMap create(MapInstanceConfig ib) throws IOException {
+ if (ib.file != null) {
+ return createWithFile(ib.file, ib.singleHashReplication, ib.channel);
+ } else {
+ return createWithoutFile(ib.singleHashReplication, ib.channel);
+ }
+ }
+
+ ChronicleMap createWithFile(File file, SingleChronicleHashReplication singleHashReplication,
+ ReplicationChannel channel) throws IOException {
+ pushingToMapEventListener();
+ for (int i = 0; i < 10; i++) {
+ if (file.exists() && file.length() > 0) {
+ try (FileInputStream fis = new FileInputStream(file);
+ ObjectInputStream ois = new ObjectInputStream(fis)) {
+ VanillaChronicleMap map =
+ (VanillaChronicleMap) ois.readObject();
+ map.headerSize = roundUpMapHeaderSize(fis.getChannel().position());
+ map.createMappedStoreAndSegments(file);
+ return establishReplication(map, singleHashReplication, channel);
+ } catch (ClassNotFoundException e) {
+ throw new IOException(e);
+ }
+ }
+ if (file.createNewFile() || file.length() == 0) {
+ break;
+ }
+ try {
+ Thread.sleep(100);
+ } catch (InterruptedException e) {
+ throw new IOException(e);
+ }
+ }
+ // new file
+ if (!file.exists())
+ throw new FileNotFoundException("Unable to create " + file);
+
+ VanillaChronicleMap map = newMap(singleHashReplication, channel);
+
+ try (FileOutputStream fos = new FileOutputStream(file);
+ ObjectOutputStream oos = new ObjectOutputStream(fos)) {
+ oos.writeObject(map);
+ oos.flush();
+ map.headerSize = roundUpMapHeaderSize(fos.getChannel().position());
+ map.createMappedStoreAndSegments(file);
+ }
+
+ return establishReplication(map, singleHashReplication, channel);
+ }
+
+ ChronicleMap createWithoutFile(
+ SingleChronicleHashReplication singleHashReplication, ReplicationChannel channel) {
+ try {
+ pushingToMapEventListener();
+ VanillaChronicleMap map = newMap(singleHashReplication, channel);
+ map.warnOnWindows();
+ BytesStore bytesStore = new DirectStore(JDKObjectSerializer.INSTANCE,
+ map.sizeInBytes(), true);
+ map.createMappedStoreAndSegments(bytesStore);
+ return establishReplication(map, singleHashReplication, channel);
+ } catch (IOException e) {
+ // file-less version should never trigger an IOException.
+ throw new AssertionError(e);
+ }
+ }
+
+ private void pushingToMapEventListener() {
+ if (pushToAddresses == null || pushToAddresses.length == 0) {
+ return;
+ }
+ try {
+ Class pmel = Class.forName(
+ "com.higherfrequencytrading.chronicle.enterprise.map.PushingMapEventListener");
+ Constructor constructor = pmel.getConstructor(ChronicleMap[].class);
+ // create a stateless client for each address
+ ChronicleMap[] statelessClients = new ChronicleMap[pushToAddresses.length];
+ ChronicleMapBuilder cmb = clone();
+ cmb.pushTo((InetSocketAddress[]) null);
+ for (int i = 0; i < pushToAddresses.length; i++) {
+ statelessClients[i] = cmb.createStatelessClient(pushToAddresses[i]);
+ }
+ eventListener =
+ (MapEventListener) constructor.newInstance((Object) statelessClients);
+ } catch (ClassNotFoundException e) {
+ LoggerFactory.getLogger(getClass().getName())
+ .warn("Chronicle Enterprise not found in the class path");
+ } catch (Exception e) {
+ LoggerFactory.getLogger(getClass().getName())
+ .error("PushingMapEventListener failed to load", e);
+ }
+ }
+
+ private VanillaChronicleMap newMap(
+ SingleChronicleHashReplication singleHashReplication, ReplicationChannel channel)
+ throws IOException {
+ boolean replicated = singleHashReplication != null || channel != null;
+ preMapConstruction(replicated);
+ if (replicated) {
+ AbstractReplication replication;
+ if (singleHashReplication != null) {
+ replication = singleHashReplication;
+ } else {
+ replication = channel.hub();
+ }
+ return new ReplicatedChronicleMap,
+ V, Object, MetaBytesInterop>(this, replication);
+ } else {
+ return new VanillaChronicleMap,
+ V, Object, MetaBytesInterop>(this);
+ }
+ }
+
+ void preMapConstruction(boolean replicated) {
+ keyBuilder.objectSerializer(acquireObjectSerializer(JDKObjectSerializer.INSTANCE));
+ valueBuilder.objectSerializer(acquireObjectSerializer(JDKObjectSerializer.INSTANCE));
+
+ long maxSize = (long) entrySize(replicated) * figureBufferAllocationFactor(replicated);
+ keyBuilder.maxSize(maxSize);
+ valueBuilder.maxSize(maxSize);
+
+ if (sampleKey != null)
+ keyBuilder.constantSizeBySample(sampleKey);
+ if (sampleValue != null)
+ valueBuilder.constantSizeBySample(sampleValue);
+ stateChecks(replicated);
+ }
+
+ private void stateChecks(boolean replicated) {
+ checkAlignmentOnlyIfValuesPossiblyReferenceOffHeap();
+ checkPrepareValueBytesOnlyIfConstantValueSize();
+ checkMaxEntryOversizeFactorIfReplicated(replicated);
+ }
+
+ private ChronicleMap establishReplication(
+ VanillaChronicleMap map,
+ SingleChronicleHashReplication singleHashReplication,
+ ReplicationChannel channel) throws IOException {
+ if (map instanceof ReplicatedChronicleMap) {
+ if (singleHashReplication != null && channel != null) {
+ throw new AssertionError("Only one non-null replication should be passed");
+ }
+ ReplicatedChronicleMap result = (ReplicatedChronicleMap) map;
+ List replicators = new ArrayList<>(2);
+ if (singleHashReplication != null) {
+ if (singleHashReplication.tcpTransportAndNetwork() != null)
+ replicators.add(Replicators.tcp(singleHashReplication));
+ if (singleHashReplication.udpTransport() != null)
+ replicators.add(Replicators.udp(singleHashReplication.udpTransport()));
+ } else {
+ ReplicationHub hub = channel.hub();
+ int entrySize = entrySize(true);
+ if (entrySize > hub.maxEntrySize()) {
+ throw new IllegalArgumentException("During ChannelProviderBuilder setup, " +
+ "maxEntrySize=" + hub.maxEntrySize() +
+ " was specified, but map with " +
+ "entrySize=" + entrySize + " is attempted to apply to the replicator");
+ }
+ ChannelProvider provider = ChannelProvider.getProvider(hub);
+ ChannelProvider.ChronicleChannel ch = provider.createChannel(channel.channelId());
+ replicators.add(ch);
+ }
+ for (Replicator replicator : replicators) {
+ Closeable token = replicator.applyTo(this, result, result, map);
+ if (replicators.size() == 1 && token.getClass() == UdpReplicator.class) {
+ LOG.warn(Replicators.ONLY_UDP_WARN_MESSAGE);
+ }
+ result.addCloseable(token);
+ }
+ }
+ return map;
+ }
+
+ private int figureBufferAllocationFactor(boolean replicated) {
+ // if expected map size is about 1000, seems rather wasteful to allocate
+ // key and value serialization buffers each x64 of expected entry size..
+ return (int) Math.min(Math.max(2L, entries() >> 10),
+ maxEntryOversizeFactor(replicated));
}
}
+
diff --git a/src/main/java/net/openhft/chronicle/map/ChronicleMapBuilderI.java b/src/main/java/net/openhft/chronicle/map/ChronicleMapBuilderI.java
deleted file mode 100755
index 99cc3455b..000000000
--- a/src/main/java/net/openhft/chronicle/map/ChronicleMapBuilderI.java
+++ /dev/null
@@ -1,319 +0,0 @@
-package net.openhft.chronicle.map;
-
-import net.openhft.chronicle.hash.ChronicleHashErrorListener;
-import net.openhft.chronicle.hash.ChronicleHashInstanceConfig;
-import net.openhft.chronicle.hash.StatelessClientConfig;
-import net.openhft.chronicle.hash.replication.SingleChronicleHashReplication;
-import net.openhft.chronicle.hash.replication.TcpTransportAndNetworkConfig;
-import net.openhft.chronicle.hash.replication.TimeProvider;
-import net.openhft.chronicle.hash.serialization.BytesReader;
-import net.openhft.chronicle.hash.serialization.BytesWriter;
-import net.openhft.chronicle.hash.serialization.SizeMarshaller;
-import net.openhft.lang.io.serialization.BytesMarshaller;
-import net.openhft.lang.io.serialization.BytesMarshallerFactory;
-import net.openhft.lang.io.serialization.ObjectFactory;
-import net.openhft.lang.io.serialization.ObjectSerializer;
-import org.jetbrains.annotations.NotNull;
-
-import java.io.File;
-import java.io.IOException;
-import java.io.Serializable;
-import java.net.InetSocketAddress;
-import java.util.concurrent.TimeUnit;
-
-/**
- * @author Rob Austin.
- */
-public interface ChronicleMapBuilderI extends Serializable {
-
- /**
- * {@inheritDoc} With respect to {@linkplain #entryAndValueAlignment(Alignment) alignment}.
- *
- *
Note that the actual entrySize will be aligned to 4 (default {@linkplain
- * #entryAndValueAlignment(Alignment) entry alignment}). I. e. if you set entry size to 30, and
- * entry alignment is set to {@link Alignment#OF_4_BYTES}, the actual entry size will be 32 (30
- * aligned to 4 bytes).
- *
- * @see #entryAndValueAlignment(Alignment) // * @see #entries(long)
- */
- ChronicleMapBuilderI entrySize(int entrySize);
-
- /**
- * Configures alignment strategy of address in memory of entries and independently of address in
- * memory of values within entries in ChronicleMaps, created by this builder.
- *
- *
Useful when values of the map are updated intensively, particularly fields with volatile
- * access, because it doesn't work well if the value crosses cache lines. Also, on some
- * (nowadays rare) architectures any misaligned memory access is more expensive than aligned.
- *
- *
Note that {@linkplain #entrySize(int) entry size} will be aligned according to this
- * alignment. I. e. if you set {@code entrySize(20)} and {@link Alignment#OF_8_BYTES}, actual
- * entry size will be 24 (20 aligned to 8 bytes).
- *
- *
Default is {@link Alignment#OF_4_BYTES} for Byteable values.
- *
- * @param alignment the new alignment of the maps constructed by this builder
- * @return this {@code ChronicleMapOnHeapUpdatableBuilder} back
- */
- ChronicleMapBuilderI entryAndValueAlignment(Alignment alignment);
-
- /**
- * {@inheritDoc} Also, it overrides any previous {@link #prepareValueBytesOnAcquire}
- * configuration to this {@code ChronicleMapBuilder}.
- *
- *
By default, the default value is not specified, default {@linkplain
- * #prepareValueBytesOnAcquire prepare value bytes routine} is specified instead.
- *
- * @see #defaultValueProvider(DefaultValueProvider)
- * @see #prepareValueBytesOnAcquire(PrepareValueBytes)
- */
- ChronicleMapBuilderI defaultValue(V defaultValue);
-
- /**
- * Configures the procedure which is called on the bytes, which later the returned value is
- * pointing to, if the key is absent, on {@link ChronicleMap#acquireUsing(Object, Object)
- * acquireUsing()} call on maps, created by this builder. See {@link PrepareValueBytes} for more
- * information.
- *
- *
The default preparation callback zeroes out the value bytes.
- *
- * @param prepareValueBytes what to do with the value bytes before assigning them into the
- * {@link net.openhft.lang.model.Byteable} value to return from {@code
- * acquireUsing()} call
- * @return this builder back
- * @see PrepareValueBytes
- * @see #defaultValue(Object)
- * @see #defaultValueProvider(DefaultValueProvider)
- */
- ChronicleMapBuilderI prepareValueBytesOnAcquire(
- @NotNull PrepareValueBytes prepareValueBytes);
-
- /**
- * {@inheritDoc}
- *
- * @see #constantKeySizeBySample(Object)
- * @see #valueSize(int)
- * @see #entrySize(int)
- */
- ChronicleMapBuilderI keySize(int keySize);
-
- /**
- * {@inheritDoc}
- *
- * @see #keySize(int)
- * @see #constantValueSizeBySample(Object)
- */
-
- ChronicleMapBuilderI constantKeySizeBySample(K sampleKey);
-
- /**
- * Configures the optimal number of bytes, taken by serialized form of values, put into maps,
- * created by this builder. If value size is always the same, call {@link
- * #constantValueSizeBySample(Object)} method instead of this one.
- *
- *
If value is a boxed primitive type, i. e. if value size is known statically, it is
- * automatically accounted and shouldn't be specified by user.
- *
- *
If value size varies moderately, specify the size higher than average, but lower than the
- * maximum possible, to minimize average memory overuse. If value size varies in a wide range,
- * it's better to use {@linkplain #entrySize(int) entry size} in "chunk" mode and configure it
- * directly.
- *
- * @param valueSize number of bytes, taken by serialized form of values
- * @return this {@code ChronicleMapOnHeapUpdatableBuilder} back
- * @see #constantValueSizeBySample(Object)
- * @see #keySize(int)
- * @see #entrySize(int)
- */
-
- ChronicleMapBuilderI valueSize(int valueSize);
-
- /**
- * Configures the constant number of bytes, taken by serialized form of values, put into maps,
- * created by this builder. This is done by providing the {@code sampleValue}, all values should
- * take the same number of bytes in serialized form, as this sample object.
- *
- *
If values are of boxed primitive type or {@link net.openhft.lang.model.Byteable} subclass,
- * i. e. if value size is known statically, it is automatically accounted and this method
- * shouldn't be called.
- *
- *
If value size varies, method {@link #valueSize(int)} or {@link #entrySize(int)} should be
- * called instead of this one.
- *
- * @param sampleValue the sample value
- * @return this builder back
- * @see #valueSize(int)
- * @see #constantKeySizeBySample(Object)
- */
-
- ChronicleMapBuilderI constantValueSizeBySample(
- @NotNull V sampleValue);
-
- /**
- * {@inheritDoc}
- *
- *
Example:
{@code Map map =
- * ChronicleMapBuilder.of(Key.class, Value.class)
- * .entries(1_000_000)
- * .keySize(50).valueSize(200)
- * // this class hasn't implemented yet, just for example
- * .objectSerializer(new KryoObjectSerializer())
- * .create();}
- */
-
- ChronicleMapBuilderI objectSerializer(
- @NotNull ObjectSerializer objectSerializer);
-
- /**
- * {@inheritDoc}
- *
- * @throws IllegalStateException if custom value marshaller is specified or value class is not
- * either {@code BytesMarshallable} or {@code Externalizable}
- */
-
- ChronicleMapBuilderI valueDeserializationFactory(
- @NotNull ObjectFactory valueDeserializationFactory);
-
- /**
- * {@inheritDoc}
- *
- *
By default, default value provider is not specified, {@link #defaultValue(Object) default
- * value} is specified instead.
- *
- * @see #defaultValue(Object)
- */
-
- ChronicleMapBuilderI defaultValueProvider(
- @NotNull DefaultValueProvider defaultValueProvider);
-
- /**
- * Configures the {@code BytesMarshaller} used to serialize/deserialize values to/from off-heap
- * memory in maps, created by this builder. See the
- * section about serialization in ChronicleMap manual for more information.
- *
- * @param valueMarshaller the marshaller used to serialize values
- * @return this builder back
- * @see #valueMarshallers(net.openhft.chronicle.hash.serialization.BytesWriter,
- * net.openhft.chronicle.hash.serialization.BytesReader)
- * @see ChronicleMapBuilder
- * @see #objectSerializer(ObjectSerializer)
- */
- ChronicleMapBuilderI valueMarshaller(
- @NotNull BytesMarshaller valueMarshaller);
-
- /**
- * Configures the marshallers, used to serialize/deserialize values to/from off-heap memory in
- * maps, created by this builder. See the
- * section about serialization in ChronicleMap manual for more information.
- *
- *
Configuring marshalling this way results to a little bit more compact in-memory layout of
- * the map, comparing to a single interface configuration: {@link #valueMarshaller(BytesMarshaller)}.
- *
- *
Passing {@link net.openhft.chronicle.hash.serialization.BytesInterop} instead of plain
- * {@link net.openhft.chronicle.hash.serialization.BytesWriter} is, of cause, possible, but
- * currently pointless for values.
- *
- * @param valueWriter the new value object → {@link net.openhft.lang.io.Bytes} writer
- * (interop) strategy
- * @param valueReader the new {@link net.openhft.lang.io.Bytes} → value object reader
- * strategy
- * @return this builder back
- * @see #valueMarshaller(BytesMarshaller)
- */
- ChronicleMapBuilderI valueMarshallers(
- @NotNull BytesWriter valueWriter, @NotNull BytesReader valueReader);
-
- /**
- * Configures the marshaller used to serialize actual value sizes to off-heap memory in maps,
- * created by this builder.
- *
- *
Default value size marshaller is so-called {@linkplain net.openhft.chronicle.hash.serialization.SizeMarshallers#stopBit()
- * stop bit encoding marshalling}. If {@linkplain #constantValueSizeBySample(Object) constant
- * value size} is configured, or defaulted if the value type is always constant and {@code
- * ChronicleHashBuilder} implementation knows about it, this configuration takes no effect,
- * because a special {@link net.openhft.chronicle.hash.serialization.SizeMarshaller}
- * implementation, which doesn't actually do any marshalling, and just returns the known
- * constant size on {@link net.openhft.chronicle.hash.serialization.SizeMarshaller#readSize(
- *net.openhft.lang.io.Bytes)} calls, is used instead of any {@code SizeMarshaller} configured
- * using this method.
- *
- * @param valueSizeMarshaller the new marshaller, used to serialize actual value sizes to
- * off-heap memory
- * @return this builder back
- */
- ChronicleMapBuilderI valueSizeMarshaller(
- @NotNull SizeMarshaller valueSizeMarshaller);
-
- ChronicleMapBuilderI entries(long l);
-
- ChronicleMapBuilderI replication(SingleChronicleHashReplication withId);
-
- ChronicleMapBuilderI pushTo(InetSocketAddress... addresses);
-
- ChronicleMap create();
-
- ChronicleMapBuilderI actualSegments(int actualSegments);
-
- ChronicleMapBuilderI minSegments(int minSegments);
-
- ChronicleMapBuilderI actualEntriesPerSegment(long actualEntriesPerSegment);
-
- ChronicleMapBuilderI lockTimeOut(long lockTimeOut, TimeUnit unit);
-
- ChronicleMapBuilderI errorListener(ChronicleHashErrorListener errorListener);
-
- ChronicleMapBuilderI metaDataBytes(int metaDataBytes);
-
- ChronicleMapBuilderI timeProvider(TimeProvider timeProvider);
-
- ChronicleMapBuilderI bytesMarshallerFactory(BytesMarshallerFactory bytesMarshallerFactory);
-
- ChronicleMapBuilderI