org.apache.mahout.math.map.package.html Maven / Gradle / Ivy
Show all versions of mahout-collections Show documentation
Automatically growing and shrinking maps holding objects or primitive
data types such as int, double, etc. Currently all maps are
based upon hashing.
1. Overview
The map package offers flexible object oriented abstractions modelling automatically
resizing maps. It is designed to be scalable in terms of performance and memory
requirements.
Features include:
- Maps operating on objects as well as all primitive data types such as
int,
double, etc.
- Compact representations
- Support for quick access to associations
- A number of general purpose map operations
File-based I/O can be achieved through the standard Java built-in serialization
mechanism. All classes implement the {@link java.io.Serializable} interface.
However, the toolkit is entirely decoupled from advanced I/O. It provides data
structures and algorithms only.
This toolkit borrows some terminology from the Javasoft
Collections framework written by Josh Bloch and introduced in JDK 1.2.
2. Introduction
A map is an associative container that manages a set of (key,value) pairs.
It is useful for implementing a collection of one-to-one mappings. A (key,value)
pair is called an association. A value can be looked up up via its key.
Associations can quickly be set, removed and retrieved. They are stored in a
hashing structure based on the hash code of their keys, which is obtained by
using a hash function.
A map can, for example, contain Name-->Location associations like
{("Pete", "Geneva"), ("Steve", "Paris"), ("Robert", "New York")} used
in address books or Index-->Value mappings like {(0, 100), (3,
1000), (100000, 70)} representing sparse lists or matrices. For example
this could mean at index 0 we have a value of 100, at index 3 we have a value
of 1000, at index 1000000 we have a value of 70, and at all other indexes we
have a value of, say, zero. Another example is a map of IP addresses to domain
names (DNS). Maps can also be useful to represent multi sets, that is,
sets where elements can occur more than once. For multi sets one would have
Value-->Frequency mappings like {(100, 1), (50, 1000), (101, 3))}
meaning element 100 occurs 1 time, element 50 occurs 1000 times, element 101
occurs 3 times. Further, maps can also manage ObjectIdentifier-->Object
mappings like {(12, obj1), (7, obj2), (10000, obj3), (9, obj4)} used
in Object Databases.
A map cannot contain two or more equal keys; a key can map to at most
one value. However, more than one key can map to identical values. For primitive
data types "equality" of keys is defined as identity (operator ==).
For maps using Object keys, the meaning of "equality" can be specified
by the user upon instance construction. It can either be defined to be identity
(operator ==) or to be given by the method {@link java.lang.Object#equals(Object)}.
Associations of kind (AnyType,Object) can be of the form (AnyKey,null)
, i.e. values can be null.
The classes of this package make no guarantees as to the order of the elements
returned by iterators; in particular, they do not guarantee that the order will
remain constant over time.
Copying
Any map can be copied. A copy is equal to the original but entirely
independent of the original. So changes in the copy are not reflected in the
original, and vice-versa.
3. Package organization
For most primitive data types and for objects there exists a separate map version.
All versions are just the same, except that they operate on different data types.
Colt includes two kinds of implementations for maps: The two different implementations
are tagged Chained and Open.
Note: Chained is no more included. Wherever it is mentioned it is of historic interest only.
- Chained uses extendible separate chaining with chains holding unsorted
dynamically linked collision lists.
- Open uses extendible open addressing with double hashing.
Class naming follows the schema <Implementation><KeyType><ValueType>HashMap.
For example, a {@link org.apache.mahout.math.map.OpenIntDoubleHashMap} holds (int-->double)
associations and is implemented with open addressing. A {@link org.apache.mahout.math.map.OpenIntObjectHashMap}
holds (int-->Object) associations and is implemented with open addressing.
The classes for maps of a given (key,value) type are derived from a common
abstract base class tagged Abstract<KeyType><ValueType>Map.
For example, all maps operating on (int-->double) associations are
derived from {@link org.apache.mahout.math.map.AbstractIntDoubleMap}, which in turn is derived
from an abstract base class tying together all maps regardless of assocation
type, {@link org.apache.mahout.math.map.AbstractMap}. The abstract base classes provide skeleton
implementations for all but few methods. Experimental layouts (such as chaining,
open addressing, extensible hashing, red-black-trees, etc.) can easily be implemented
and inherit a rich set of functionality. Have a look at the javadoc tree
view to get the broad picture.
4. Example usage
int[] keys = {0 , 3 , 100000, 9 };
double[] values = {100.0, 1000.0, 70.0 , 71.0};
AbstractIntDoubleMap map = new OpenIntDoubleHashMap();
// add several associations
for (int i=0; i < keys.length; i++) map.put(keys[i], values[i]);
log.info("map="+map);
log.info("size="+map.size());
log.info(map.containsKey(3));
log.info("get(3)="+map.get(3));
log.info(map.containsKey(4));
log.info("get(4)="+map.get(4));
log.info(map.containsValue(71.0));
log.info("keyOf(71.0)="+map.keyOf(71.0));
// remove one association
map.removeKey(3);
log.info("\nmap="+map);
log.info(map.containsKey(3));
log.info("get(3)="+map.get(3));
log.info(map.containsValue(1000.0));
log.info("keyOf(1000.0)="+map.keyOf(1000.0));
// clear
map.clear();
log.info("\nmap="+map);
log.info("size="+map.size());
yields the following output
map=[0->100.0, 3->1000.0, 9->71.0, 100000->70.0]
size=4
true
get(3)=1000.0
false
get(4)=0.0
true
keyOf(71.0)=9
map=[0->100.0, 9->71.0, 100000->70.0]
false
get(3)=0.0
false
keyOf(1000.0)=-2147483648
map=[]
size=0
5. Notes
Note that implementations are not synchronized.
Choosing efficient parameters for hash maps is not always easy.
However, since parameters determine efficiency and memory requirements, here is a quick guide how to choose them.
If your use case does not heavily operate on hash maps but uses them just because they provide
convenient functionality, you can safely skip this section.
For those of you who care, read on.
There are three parameters that can be customized upon map construction: initialCapacity,
minLoadFactor and maxLoadFactor.
The more memory one can afford, the faster a hash map.
The hash map's capacity is the maximum number of associations that can be added without needing to allocate new
internal memory.
A larger capacity means faster adding, searching and removing.
The initialCapacity corresponds to the capacity used upon instance construction.
The loadFactor of a hash map measures the degree of "fullness".
It is given by the number of assocations (size())
divided by the hash map capacity (0.0 <= loadFactor <= 1.0).
The more associations are added, the larger the loadFactor and the more hash map performance degrades.
Therefore, when the loadFactor exceeds a customizable threshold (maxLoadFactor), the hash map is
automatically grown.
In such a way performance degradation can be avoided.
Similarly, when the loadFactor falls below a customizable threshold (minLoadFactor), the hash map is
automatically shrinked.
In such a way excessive memory consumption can be avoided.
Automatic resizing (both growing and shrinking) obeys the following invariant:
capacity * minLoadFactor <= size() <= capacity * maxLoadFactor
The term capacity * minLoadFactor is called the low water mark,
capacity * maxLoadFactor is called the high water mark. In other
words, the number of associations may vary within the water mark constraints.
When it goes out of range, the map is automatically resized and memory consumption
changes proportionally.
- To tune for memory at the expense of performance, both increase minLoadFactor and maxLoadFactor.
- To tune for performance at the expense of memory, both decrease minLoadFactor and maxLoadFactor.
As as special case set minLoadFactor=0 to avoid any automatic shrinking.
Resizing large hash maps can be time consuming, O(size()), and should be avoided if possible (maintaining
primes is not the reason).
Unnecessary growing operations can be avoided if the number of associations is known before they are added, or can be
estimated.
In such a case good parameters are as follows:
For chaining:
Set the initialCapacity = 1.4*expectedSize or greater.
Set the maxLoadFactor = 0.8 or greater.
For open addressing:
Set the initialCapacity = 2*expectedSize or greater. Alternatively call ensureCapacity(...).
Set the maxLoadFactor = 0.5.
Never set maxLoadFactor > 0.55; open addressing exponentially slows down beyond that point.
In this way the hash map will never need to grow and still stay fast.
It is never a good idea to set maxLoadFactor < 0.1,
because the hash map would grow too often.
If it is entirelly unknown how many associations the application will use,
the default constructor should be used. The map will grow and shrink as needed.
Comparision of chaining and open addressing
Chaining is faster than open addressing, when assuming unconstrained memory
consumption. Open addressing is more space efficient than chaining, because
it does not create entry objects but uses primitive arrays which are considerably
smaller. Entry objects consume significant amounts of memory compared to the
information they actually hold. Open addressing also poses no problems to the
garbage collector. In contrast, chaining can create millions of entry objects
which are linked; a nightmare for any garbage collector. In addition, entry
object creation is a bit slow.
Therefore, with the same amount of memory, or even less memory, hash maps with
larger capacity can be maintained under open addressing, which yields smaller
loadFactors, which in turn keeps performance competitive with chaining. In our
benchmarks, using significantly less memory, open addressing usually is not
more than 1.2-1.5 times slower than chaining.
Further readings:
Knuth D., The Art of Computer Programming: Searching and Sorting, 3rd ed.
Griswold W., Townsend G., The Design and Implementation of Dynamic Hashing for Sets and Tables in Icon, Software -
Practice and Experience, Vol. 23(4), 351-367 (April 1993).
Larson P., Dynamic hash tables, Comm. of the ACM, 31, (4), 1988.
Performance:
Time complexity:
The classes offer expected time complexity O(1) (i.e. constant time) for the basic operations
put, get, removeKey, containsKey and size,
assuming the hash function disperses the elements properly among the buckets.
Otherwise, pathological cases, although highly improbable, can occur, degrading performance to O(N) in the
worst case.
Operations containsValue and keyOf are O(N).
Memory requirements for open addressing:
worst case: memory [bytes] = (1/minLoadFactor) * size() * (1 + sizeOf(key) + sizeOf(value)).
best case: memory [bytes] = (1/maxLoadFactor) * size() * (1 + sizeOf(key) + sizeOf(value)).
Where sizeOf(int) = 4, sizeOf(double) = 8, sizeOf(Object) = 4, etc.
Thus, an OpenIntIntHashMap with minLoadFactor=0.25 and maxLoadFactor=0.5 and 1000000 associations uses
between 17 MB and 34 MB.
The same map with 1000 associations uses between 17 and 34 KB.