Alawn/Chronicle-Map
1
0
Fork 0
mirror of https://github.com/OpenHFT/Chronicle-Map.git synced 2021-09-13 01:26:12 +00:00
Code Releases Activity

Merge branch 'master' of github.com:OpenHFT/Chronicle-Map

Browse Source 
Conflicts:
	src/main/java/net/openhft/chronicle/map/AbstractChronicleMapBuilder.java
This commit is contained in:
leventov
2014-11-10 18:15:58 +03:00
parent daac52ba35 ec619f4dbd
commit 2be60c999a
4 changed files with 57 additions and 54 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+21 -15
View File
@@ -340,7 +340,7 @@ a key, but unlike `map.getUsing()` if there is not an entry in the map for this
will be added and the value return will we the same value which you provided. ( The section below
covers both `map.getUsing()` and `map.acquireUsing()` in more detail )
#### Off Heap Storage and how using a Proxy Object can Improve Performance
#### Off heap storage and how using a Proxy Object can improve performance
Chronicle Map stores its data off heap. There are some distinct advantages in using off heap data storage
@@ -350,7 +350,7 @@ Chronicle Map stores its data off heap. There are some distinct advantages in us
heap memory. When this memory is shared between processes on the same server,
the chronicle map is able to distribute entries between processes with extremely low overhead,
as both processes are sharing the same off heap memory space. Since your objects are already stored off
heap ( as a series of bytes ), replicating entries over the network add relatively low over head.
heap ( as a series of bytes ), replicating entries over the network adds relatively low over head.
One of the downsides of an Off Heap Map is that whenever you wish to get a value ( on heap ) from an entry
which is off heap, for example calling :
@@ -364,6 +364,7 @@ that entry has to be deserialised onto the java heap so that you can use its val
``` java
for(int i=1;i<=10;i++) {
Value v = get(key)
}
```
this would create 10 separate instances of the value. As each time get() is called the map has to
@@ -380,7 +381,8 @@ and existing object ( in this case the ‘using’ value ), you can instead call
Value using = new Value();
for(int i=1;i<=10;i++) {
Value v = map.getUsing(key,using); // this won’t create a new value each time.
Value bond = map.getUsing(key,using); // this won’t create a new value each time.
assert using == bond; // this will always be the same instance and the Vaule type can not be immutable
}
```
We can get a further performance improvement if we don't deserialize the whole object,
@@ -424,7 +426,10 @@ BondVOInterface bond = map.get(key);
bond.getCoupon()
```
lets say that it is only the `coupon` field that we are interested in, then its better not to have to deserialize the whole object that implements the `BondVOInterface`. It would be more efficient to just deserialize the field(s) that we require, in this example just the coupon. This is why we created the`OffHeapUpdatableChronicleMapBuilder`, when you use this builder chronicle will not create on heap copies of your whole DataStructure. You can instead use proxy objects that can read and write into the off heap data structures, this reduced serialisation can give you a big performance boost. Below we show you how you can work directly with the off heap entries.
lets say that it is only the `coupon` field that we are interested in, then its better not to have to
deserialize the whole object that implements the `BondVOInterface`. It would be more efficient to just
deserialize the field(s) that we require, in this example just the coupon. This is why we created
the`OffHeapUpdatableChronicleMapBuilder`, when you use this builder chronicle will not create on heap copies of your whole data structure. You can instead use proxy objects that can read and write into the off heap data structures, this reduced serialisation can give you a big performance boost. Below we show you how you can work directly with the off heap entries.
``` java
ChronicleMap<String, BondVOInterface> chm = OffHeapUpdatableChronicleMapBuilder
@@ -443,22 +448,24 @@ value class, in our case `BondVOInterface.class` is an `interface` rather than a
class that class has to extend `Byteable`), now
like before, we can
use the `getUsing(key,using)` method, but this time we have to create the ‘using’ instance slightly
differently
differently, we have to call the `newDirectReference(..)` method.
``` java
BondVOInterface using = DataValueClasses.newDirectReference(BondVOInterface.class);
```
now we can call
the call to `getUsing(key,value)` is the same as we had in the earlier example
``` java
BondVOInterface bond = map.getUsing(key,value);
BondVOInterface using = DataValueClasses.newDirectReference(BondVOInterface.class);
BondVOInterface bond = map.getUsing(key,using);
assert using == bond; // this will always be the same instance
```
this won’t create any on heap objects, and it won’t deserialize the ‘bond’ entry from off heap to on
heap, all it does is sets the bond as a proxy to the off heap memory, this proxy object was created by
`DataValueClasses.newDirectReference(BondVOInterface.class)` it allows us access the fileds for our entry
directly from the off heap storage,
`DataValueClasses.newDirectReference(BondVOInterface.class)`, it allows us access to the fields of our
entry, directly into the off heap storage.
so when you call :
@@ -468,13 +475,12 @@ bond.getCoupon()
its only the coupon that gets deserialized.
Just like any other concurrent map, chronicle map uses segment locking, if you wish to obtain a read lock
when calling when calling getUsing(key,using) you can do this :
when calling getUsing(key,using) you can do this :
``` java
try (ReadContext<?, BondVOInterface> context = map.getUsingLocked(key,using)) {
BondVOInterface bond = context.value().getValue();
BondVOInterface bond = map.getUsing(key,value);
try (ReadContext<?, BondVOInterface> context = map.getUsingLocked(key,bond)) {
long issueDate = bond.getIssueDate();
String symbol = bond.getSymbol();
@@ -497,7 +503,7 @@ If you wish not to use a try block you must manually release the segment lock by
context.close() // the lock will get released when this is called
```
#### what acquireUsingLocked()
#### acquireUsingLocked()
just like getUsing(), acquireUsing() will also recycle the value you pass it, the following
code is a pattern that you will often come across, acquireUsing(key,value) offers this
@@ -542,7 +548,7 @@ try (WriteContext<?, BondVOInterface> context = map.acquireUsingLocked("one", bo
}
```
if after you have read the 'issueDate' and 'symbol' you wish to remove the entry based on some
if after you have read the 'issueDate' and 'symbol' and you wish to remove the entry based on some
business logic, it more efficient to use the 'context' to remove the entry, as the contents is
already aware when the entry is in memory.
src/main/java/net/openhft/chronicle/map/AbstractChronicleMapBuilder.java
+16 -20
View File
@@ -372,10 +372,8 @@ public abstract class AbstractChronicleMapBuilder<K, V,
if (segments == 1)
return entries;
double poorDistEntriesScale = Math.log(segments) * entries;
if (segments <= 8)
return Math.min(entries * segments,
(long) (entries + poorDistEntriesScale * 0.11 + segments * 8)); // 11% was min for tests
return (long) (entries + poorDistEntriesScale * 0.11 + 80); // 11% was min for tests
return Math.min(entries * segments,
(long) (entries + poorDistEntriesScale * 0.14 + 32)); // 12% was min for tests
}
@Override
@@ -390,24 +388,22 @@ public abstract class AbstractChronicleMapBuilder<K, V,
}
private int estimateSegments() {
for (int power = 1; power < 64; power *= 2) {
if (entries <= power * power * power)
return power;
}
long size = (entrySize + 32) * entries;
if (size < 10L * 1000 * 1000) // 10 MB
return 64;
if (size < 100L * 1000 * 1000) // 100 MB
return 128;
if (size < 400L * 1000 * 1000) // 400 MB
return 256;
if (size < 1L * 1000 * 1000 * 1000) // 1 GB
return (int) Math.min(Maths.nextPower2(entries / 32, 1), estimateSegementBasedOnSize());
}
private int estimateSegementBasedOnSize() {
// based on entries with multimap of 100 bytes.
long size = (long) (Math.log10(entrySize + 32) / 2 * entries);
if (size < 10 * 1000)
return 512;
if (size < 10L * 1000 * 1000 * 1000) // 10 GB
if (size < 100 * 1000)
return 1024;
if (size < 100L * 1000 * 1000 * 1000) // 100 GB
if (size < 1000 * 1000)
return 2048;
return 4096;
if (size < 10 * 1000 * 1000)
return 4096;
return 8192;
}
@Override
@@ -443,7 +439,7 @@ public abstract class AbstractChronicleMapBuilder<K, V,
int segmentHeaderSize() {
int segments = actualSegments();
// reduce false sharing unless we have a lot of segments.
return segments <= 8192 ? 64 : 16;
return segments <= 8192 ? 128 : 64;
}
MultiMapFactory multiMapFactory() {
src/test/java/net/openhft/chronicle/map/ChronicleMapTest.java
+2 -1
View File
@@ -576,7 +576,7 @@ public class ChronicleMapTest {
InstantiationException, InterruptedException, ExecutionException {
// int runs = Integer.getInteger("runs", 10);
int procs = Runtime.getRuntime().availableProcessors();
int threads = procs * 2; // runs > 100 ? procs / 2 : procs;
int threads = procs * 3;
ExecutorService es = Executors.newFixedThreadPool(procs);
for (int runs : new int[]{10, 50, 250, 500, 1000, 2500}) {
// JAVA 8 produces more garbage than previous versions for internal work.
@@ -585,6 +585,7 @@ public class ChronicleMapTest {
OffHeapUpdatableChronicleMapBuilder<CharSequence, LongValue> builder = OffHeapUpdatableChronicleMapBuilder
.of(CharSequence.class, LongValue.class)
.entries(entries)
.actualSegments(8 * 1024)
.entryAndValueAlignment(OF_8_BYTES)
.entrySize(24);
src/test/java/net/openhft/chronicle/map/EntryCountMapTest.java
+18 -18
View File
@@ -53,7 +53,7 @@ public class EntryCountMapTest {
throws IOException {
OffHeapUpdatableChronicleMapBuilder<CharSequence, LongValue> mapBuilder = OffHeapUpdatableChronicleMapBuilder.of(CharSequence.class, LongValue.class)
.entries(entries)
.minSegments(segments)
.actualSegments(segments)
.entrySize(entrySize)
.entryAndValueAlignment(alignment);
return mapBuilder.createPersistedTo(getPersistenceFile());
@@ -103,33 +103,33 @@ public class EntryCountMapTest {
testEntriesMaxSize(s, 2000, 3100, i);
testEntriesMaxSize(s, 4000, 5100, i);
testEntriesMaxSize(s, 5000, 7100, i);
testEntriesMaxSize(s, 8000, 10200, i);
testEntriesMaxSize(s, 12000, 16000, i);
testEntriesMaxSize(s, 16000, 21300, i);
testEntriesMaxSize(s, 8000, 11000, i);
testEntriesMaxSize(s, 12000, 17000, i);
testEntriesMaxSize(s, 16000, 22000, i);
s = 32;
testEntriesMaxSize(s, 2000, 3900, i);
testEntriesMaxSize(s, 4000, 5900, i);
testEntriesMaxSize(s, 5000, 7900, i);
testEntriesMaxSize(s, 8000, 12000, i);
testEntriesMaxSize(s, 12000, 15900, i);
testEntriesMaxSize(s, 16000, 21300, i);
testEntriesMaxSize(s, 32000, 42200, i);
testEntriesMaxSize(s, 12000, 17000, i);
testEntriesMaxSize(s, 16000, 23000, i);
testEntriesMaxSize(s, 32000, 46000, i);
s = 64;
testEntriesMaxSize(s, 5000, 7550, i);
testEntriesMaxSize(s, 8000, 11400, i);
testEntriesMaxSize(s, 12000, 16000, i);
testEntriesMaxSize(s, 16000, 22000, i);
testEntriesMaxSize(s, 32000, 43000, i);
testEntriesMaxSize(s, 40000, 54000, i);
testEntriesMaxSize(s, 12000, 18000, i);
testEntriesMaxSize(s, 16000, 24000, i);
testEntriesMaxSize(s, 32000, 47000, i);
testEntriesMaxSize(s, 40000, 60000, i);
s = 128;
testEntriesMaxSize(s, 16000, 23000, i);
testEntriesMaxSize(s, 32000, 44200, i);
testEntriesMaxSize(s, 40000, 56000, i);
testEntriesMaxSize(s, 64000, 92000, i);
testEntriesMaxSize(s, 129000, 190000, i);
testEntriesMaxSize(s, 16000, 24000, i);
testEntriesMaxSize(s, 32000, 50000, i);
testEntriesMaxSize(s, 40000, 62000, i);
testEntriesMaxSize(s, 64000, 98000, i);
testEntriesMaxSize(s, 129000, 204000, i);
s = 256;
testEntriesMaxSize(s, 64000, 92000, i);
testEntriesMaxSize(s, 129000, 192000, i);
testEntriesMaxSize(s, 64000, 100000, i);
testEntriesMaxSize(s, 129000, 213000, i);
}
// hyperbolic average gives more weight to small numbers.
System.out.printf("Score: %.2f%n", scoreCount / score);