数据结构之BloomFilter

标签： 数据结构 bloomfilter | 发表时间：2014-12-04 10:12 | 作者：ljl_xyf

出处：http://www.iteye.com

BloomFilter是什么？

BloomFilter主要提供两种操作: add()和contains()，作用分别是将元素加入其中以及判断一个元素是否在其中，类似于Java中的Set接口，它内部采用的byte数组来节省空间。其独特之处在于contains()方法，当我们需要查询某个元素是否包含在BloomFilter中时，如果返回true，结果可能是不正确的，也就是元素也有可能不在其中；但是如果返回的是false，那么元素一定不在其中。这也就是所谓的no false negative和small probability of false positive。通俗地来就是，“说了没有就是没有”，“说有但实际上有可能没有”。

为什么要使用BloomFilter？

任何一种数据结构，特别是比较巧妙的，都有它的应用场景，BloomFilter也不例外。许多实际背景和应用这里不再啰嗦，举个简单例子：我们需要存储大量的URL，并且对于新得到的URL需要知道是否已经存储了。如果将这些URL全扔进Set，那就悲剧了，内存可吃不消。这个时候就要想到用byte来存储了，可以节省大量的空间。

BloomFilter实现

既然涉及到byte存储，那么必然需要一种映射关系了，也就是需要知道一个元素用哪些byte来表示，这也就是hash函数所干的事情。直观地想，一个元素一个byte基本上不可能，一般情况下这样的hash函数可以说不存在，所以这里假设用k位来表示，一般采用k次hash来确定这些byte。实际上，BloomFilter中一般包含m(byte数组的大小),k（hash次数）和n(需要存储的数据量)。在元素加入实现add()操作时，连续k次hash，将得到的对应k位全置为1。当查询元素是否在集合中时，也是连续k次hash，如果这k次得到的位不是全为1，那么返回false；否则返回 true。传统的算法优化都是以时间换空间或者以空间换时间，BloomFilter则是以正确率换空间。

Java代码

Class BloomFilter<E> {
public BloomFilter(int m, int k){……}
public void add(E obj){……}
public boolean contains(E obj){……}
}

BloomFilter相关结论

根据上面的标记，可以采用数学方法推导出false positive的概率大约是p = (1-exp(-kn/m))^k，那么由此可得出，最优的k=ln(2)*(m/n)。我们可以计算一下上面提到的例子，假设共有10000000个 URL(n=10000000)，如果采用m=80000000，k=8，得出p大约2%；但是所占用的内存只有10M，相同情况下使用Set则需要1G 的内存。

具体的实现如下(来自项目http://code.google.com/p/java-bloomfilter/，代码实现不错，可直接使用):

Java代码

/**
* This program is free software: you can redistribute it and/or modify
* it under the terms of the GNU Lesser General Public License as published by
* the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Lesser General Public License for more details.
*
* You should have received a copy of the GNU Lesser General Public License
* along with this program. If not, see <http://www.gnu.org/licenses/>.
*/
package com.skjegstad.utils;
import java.io.Serializable;
import java.nio.charset.Charset;
import java.security.MessageDigest;
import java.security.NoSuchAlgorithmException;
import java.util.BitSet;
import java.util.Collection;
/**
* Implementation of a Bloom-filter, as described here:
* http://en.wikipedia.org/wiki/Bloom_filter
*
* For updates and bugfixes, see http://github.com/magnuss/java-bloomfilter
*
* Inspired by the SimpleBloomFilter-class written by Ian Clarke. This
* implementation provides a more evenly distributed Hash-function by
* using a proper digest instead of the Java RNG. Many of the changes
* were proposed in comments in his blog:
* http://blog.locut.us/2008/01/12/a-decent-stand-alone-java-bloom-filter-implementation/
*
* @param <E> Object type that is to be inserted into the Bloom filter, e.g. String or Integer.
* @author Magnus Skjegstad <[email protected]>
*/
public class BloomFilter<E> implements Serializable {
private BitSet bitset;
private int bitSetSize;
private double bitsPerElement;
private int expectedNumberOfFilterElements;
private int numberOfAddedElements;
private int k; // number of hash functions
static final Charset charset = Charset.forName("UTF-8");
static final String hashName = "MD5";
static final MessageDigest digestFunction;
static { // The digest method is reused between instances
MessageDigest tmp;
try {
tmp = java.security.MessageDigest.getInstance(hashName);
} catch (NoSuchAlgorithmException e) {
tmp = null;
}
digestFunction = tmp;
}
/**
* Constructs an empty Bloom filter. The total length of the Bloom filter will be
* c*n.
*
* @param c is the number of bits used per element.
* @param n is the expected number of elements the filter will contain.
* @param k is the number of hash functions used.
*/
public BloomFilter(double c, int n, int k) {
this.expectedNumberOfFilterElements = n;
this.k = k;
this.bitsPerElement = c;
this.bitSetSize = (int)Math.ceil(c * n);
numberOfAddedElements = 0;
this.bitset = new BitSet(bitSetSize);
}
/**
* Constructs an empty Bloom filter. The optimal number of hash functions (k) is estimated from
* the total size of the Bloom
* and the number of expected elements.
*
* @param bitSetSize defines how many bits should be used in total for the filter.
* @param expectedNumberOElements defines the maximum number of elements the filter is
* expected to contain.
*/
public BloomFilter(int bitSetSize, int expectedNumberOElements) {
this(bitSetSize / (double)expectedNumberOElements,
expectedNumberOElements,
(int) Math.round((bitSetSize / (double)expectedNumberOElements) * Math.log(2.0)));
}
/**
* Constructs an empty Bloom filter with a given false positive probability. The number of bits per
* element and the number of hash functions is estimated
* to match the false positive probability.
*
* @param falsePositiveProbability is the desired false positive probability.
* @param expectedNumberOfElements is the expected number of elements in the Bloom filter.
*/
public BloomFilter(double falsePositiveProbability, int expectedNumberOfElements) {
this(Math.ceil(-(Math.log(falsePositiveProbability) / Math.log(2))) / Math.log(2),
expectedNumberOfElements,
(int)Math.ceil(-(Math.log(falsePositiveProbability) / Math.log(2))));
}
/**
* Construct a new Bloom filter based on existing Bloom filter data.
*
* @param bitSetSize defines how many bits should be used for the filter.
* @param expectedNumberOfFilterElements defines the maximum number of elements the filter
* is expected to contain.
* @param actualNumberOfFilterElements specifies how many elements have been inserted into
* the <code>filterData</code> BitSet.
* @param filterData a BitSet representing an existing Bloom filter.
*/
public BloomFilter(int bitSetSize, int expectedNumberOfFilterElements,
int actualNumberOfFilterElements, BitSet filterData) {
this(bitSetSize, expectedNumberOfFilterElements);
this.bitset = filterData;
this.numberOfAddedElements = actualNumberOfFilterElements;
}
/**
* Generates a digest based on the contents of a String.
*
* @param val specifies the input data.
* @param charset specifies the encoding of the input data.
* @return digest as long.
*/
public static int createHash(String val, Charset charset) {
return createHash(val.getBytes(charset));
}
/**
* Generates a digest based on the contents of a String.
*
* @param val specifies the input data. The encoding is expected to be UTF-8.
* @return digest as long.
*/
public static int createHash(String val) {
return createHash(val, charset);
}
/**
* Generates a digest based on the contents of an array of bytes.
*
* @param data specifies input data.
* @return digest as long.
*/
public static int createHash(byte[] data) {
return createHashes(data, 1)[0];
}
/**
* Generates digests based on the contents of an array of bytes and splits the result into
* 4-byte int's and store them in an array. The
* digest function is called until the required number of int's are produced. For each call to
* digest a salt
* is prepended to the data. The salt is increased by 1 for each call.
*
* @param data specifies input data.
* @param hashes number of hashes/int's to produce.
* @return array of int-sized hashes
*/
public static int[] createHashes(byte[] data, int hashes) {
int[] result = new int[hashes];
int k = 0;
byte salt = 0;
while (k < hashes) {
byte[] digest;
synchronized (digestFunction) {
digestFunction.update(salt);
salt++;
digest = digestFunction.digest(data);
}
for (int i = 0; i < digest.length/4 && k < hashes; i++) {
int h = 0;
for (int j = (i*4); j < (i*4)+4; j++) {
h <<= 8;
h |= ((int) digest[j]) & 0xFF;
}
result[k] = h;
k++;
}
}
return result;
}
/**
* Compares the contents of two instances to see if they are equal.
*
* @param obj is the object to compare to.
* @return True if the contents of the objects are equal.
*/
@Override
public boolean equals(Object obj) {
if (obj == null) {
return false;
}
if (getClass() != obj.getClass()) {
return false;
}
final BloomFilter<E> other = (BloomFilter<E>) obj;
if (this.expectedNumberOfFilterElements != other.expectedNumberOfFilterElements) {
return false;
}
if (this.k != other.k) {
return false;
}
if (this.bitSetSize != other.bitSetSize) {
return false;
}
if (this.bitset != other.bitset && (this.bitset == null || !this.bitset.equals(other.bitset))) {
return false;
}
return true;
}
/**
* Calculates a hash code for this class.
* @return hash code representing the contents of an instance of this class.
*/
@Override
public int hashCode() {
int hash = 7;
hash = 61 * hash + (this.bitset != null ? this.bitset.hashCode() : 0);
hash = 61 * hash + this.expectedNumberOfFilterElements;
hash = 61 * hash + this.bitSetSize;
hash = 61 * hash + this.k;
return hash;
}
/**
* Calculates the expected probability of false positives based on
* the number of expected filter elements and the size of the Bloom filter.
*
* The value returned by this method is the expected rate of false
* positives, assuming the number of inserted elements equals the number of
* expected elements. If the number of elements in the Bloom filter is less
* than the expected value, the true probability of false positives will be lower.
*
* @return expected probability of false positives.
*/
public double expectedFalsePositiveProbability() {
return getFalsePositiveProbability(expectedNumberOfFilterElements);
}
/**
* Calculate the probability of a false positive given the specified
* number of inserted elements.
*
* @param numberOfElements number of inserted elements.
* @return probability of a false positive.
*/
public double getFalsePositiveProbability(double numberOfElements) {
// (1 - e^(-k * n / m)) ^ k
return Math.pow((1 - Math.exp(-k * (double) numberOfElements
/ (double) bitSetSize)), k);
}
/**
* Get the current probability of a false positive. The probability is calculated from
* the size of the Bloom filter and the current number of elements added to it.
*
* @return probability of false positives.
*/
public double getFalsePositiveProbability() {
return getFalsePositiveProbability(numberOfAddedElements);
}
/**
* Returns the value chosen for K.
*
* K is the optimal number of hash functions based on the size
* of the Bloom filter and the expected number of inserted elements.
*
* @return optimal k.
*/
public int getK() {
return k;
}
/**
* Sets all bits to false in the Bloom filter.
*/
public void clear() {
bitset.clear();
numberOfAddedElements = 0;
}
/**
* Adds an object to the Bloom filter. The output from the object's
* toString() method is used as input to the hash functions.
*
* @param element is an element to register in the Bloom filter.
*/
public void add(E element) {
add(element.toString().getBytes(charset));
}
/**
* Adds an array of bytes to the Bloom filter.
*
* @param bytes array of bytes to add to the Bloom filter.
*/
public void add(byte[] bytes) {
int[] hashes = createHashes(bytes, k);
for (int hash : hashes)
bitset.set(Math.abs(hash % bitSetSize), true);
numberOfAddedElements ++;
}
/**
* Adds all elements from a Collection to the Bloom filter.
* @param c Collection of elements.
*/
public void addAll(Collection<? extends E> c) {
for (E element : c)
add(element);
}
/**
* Returns true if the element could have been inserted into the Bloom filter.
* Use getFalsePositiveProbability() to calculate the probability of this
* being correct.
*
* @param element element to check.
* @return true if the element could have been inserted into the Bloom filter.
*/
public boolean contains(E element) {
return contains(element.toString().getBytes(charset));
}
/**
* Returns true if the array of bytes could have been inserted into the Bloom filter.
* Use getFalsePositiveProbability() to calculate the probability of this
* being correct.
*
* @param bytes array of bytes to check.
* @return true if the array could have been inserted into the Bloom filter.
*/
public boolean contains(byte[] bytes) {
int[] hashes = createHashes(bytes, k);
for (int hash : hashes) {
if (!bitset.get(Math.abs(hash % bitSetSize))) {
return false;
}
}
return true;
}
/**
* Returns true if all the elements of a Collection could have been inserted
* into the Bloom filter. Use getFalsePositiveProbability() to calculate the
* probability of this being correct.
* @param c elements to check.
* @return true if all the elements in c could have been inserted into the Bloom filter.
*/
public boolean containsAll(Collection<? extends E> c) {
for (E element : c)
if (!contains(element))
return false;
return true;
}
/**
* Read a single bit from the Bloom filter.
* @param bit the bit to read.
* @return true if the bit is set, false if it is not.
*/
public boolean getBit(int bit) {
return bitset.get(bit);
}
/**
* Set a single bit in the Bloom filter.
* @param bit is the bit to set.
* @param value If true, the bit is set. If false, the bit is cleared.
*/
public void setBit(int bit, boolean value) {
bitset.set(bit, value);
}
/**
* Return the bit set used to store the Bloom filter.
* @return bit set representing the Bloom filter.
*/
public BitSet getBitSet() {
return bitset;
}
/**
* Returns the number of bits in the Bloom filter. Use count() to retrieve
* the number of inserted elements.
*
* @return the size of the bitset used by the Bloom filter.
*/
public int size() {
return this.bitSetSize;
}
/**
* Returns the number of elements added to the Bloom filter after it
* was constructed or after clear() was called.
*
* @return number of elements added to the Bloom filter.
*/
public int count() {
return this.numberOfAddedElements;
}
/**
* Returns the expected number of elements to be inserted into the filter.
* This value is the same value as the one passed to the constructor.
*
* @return expected number of elements.
*/
public int getExpectedNumberOfElements() {
return expectedNumberOfFilterElements;
}
/**
* Get expected number of bits per element when the Bloom filter is full. This value is set by the constructor
* when the Bloom filter is created. See also getBitsPerElement().
*
* @return expected number of bits per element.
*/
public double getExpectedBitsPerElement() {
return this.bitsPerElement;
}
/**
* Get actual number of bits per element based on the number of elements that have currently been inserted and the length
* of the Bloom filter. See also getExpectedBitsPerElement().
*
* @return number of bits per element.
*/
public double getBitsPerElement() {
return this.bitSetSize / (double)numberOfAddedElements;
}
}

实际应用中，重要的往往是m,n,k的选择以及hash函数的设定。在Oracle 11g中就采用了BloomFilter来实现分布式数据库中两表的join操作；在网络中应用则更加广泛，例如分布式web代理Squid；此外还有拼写检查、海量数据处理等领域中都能搞看到它的身影。

已有 0 人发表留言，猛击->> 这里<<-参与讨论

ITeye推荐

—软件人才免语言低担保赴美带薪读研！—

数据结构之BloomFilter

相关 [数据结构 bloomfilter] 推荐：

数据结构之BloomFilter

BloomFilter--大规模数据处理利器

Hbase 布隆过滤器BloomFilter介绍

如何使用bloomfilter构建大型Java缓存系统

使用graphviz画数据结构

数据结构之红黑树

数据结构-二分法查找

redis数据结构缓存运用

可视化的数据结构和算法

相关文章

订阅