Package edu.berkeley.nlp.lm.values
Class CompressibleProbBackoffValueContainer
java.lang.Object
edu.berkeley.nlp.lm.values.CompressibleProbBackoffValueContainer
- All Implemented Interfaces:
CompressibleValueContainer<ProbBackoffPair>
,ProbBackoffValueContainer
,ValueContainer<ProbBackoffPair>
,Serializable
public final class CompressibleProbBackoffValueContainer
extends Object
implements ProbBackoffValueContainer
- See Also:
-
Field Summary
FieldsModifier and TypeFieldDescriptionprotected final int
protected final long[]
protected final boolean
protected final int[]
protected boolean
protected final VariableLengthBitCompressor
protected final int
protected final CustomWidthArray[]
protected int
-
Constructor Summary
ConstructorsConstructorDescriptionCompressibleProbBackoffValueContainer
(int valueRadix, boolean storePrefixIndexes, long[] numNgramsForEachOrder, float[] probsForRank, float[] backoffsForRank, Indexer<Float> probIndexer, int wordWidth, Indexer<Float> backoffIndexer, int backoffWidth) CompressibleProbBackoffValueContainer
(LongToIntHashMap countCounter, int valueRadix, boolean storePrefixes, long[] numNgramsForEachOrder) -
Method Summary
Modifier and TypeMethodDescriptionboolean
add
(int[] ngram, int startPos, int endPos, int ngramOrder, long offset, long prefixOffset, int word, ProbBackoffPair val_, long suffixOffset, boolean ngramIsNew) Adds a new value at the specified offset.void
clearStorageAfterCompression
(int ngramOrder) void
clearStorageForOrder
(int ngramOrder) createFreshValues
(long[] numNgramsForEachOrder_) Creates a fresh value container for copying purposes.final void
decompress
(BitStream bits, int ngramOrder, boolean justConsume, ProbBackoffPair outputVal) Reads and decompresses from the bit stream bits.final float
getBackoff
(int ngramOrder, long index) final float
getBackoff
(CustomWidthArray valueRanksForNgramOrder, long index) getCompressed
(long offset, int ngramOrder) Compresses the value at the given offset into a list of bits.protected long
getCountRank
(long val) protected ProbBackoffPair
void
getFromOffset
(long index, int ngramOrder, ProbBackoffPair outputVal) Gets the value living at a particular offset.protected void
getFromRank
(long rank, ProbBackoffPair outputVal) final long
getInternalVal
(int ngramOrder, long index) final float
getProb
(int ngramOrder, long index) final float
getProb
(CustomWidthArray valueRanksForOrder, long index) protected long
getRank
(int ngramOrder, long offset) Creates a fresh value of object (useful for passing as an output parameter)long
getSuffixOffset
(long index, int ngramOrder) protected int
getSuffixOffsetFromInternalVal
(long internalVal) int
numValueBits
(int ngramOrder) void
Destructively sets internal storage from another object.void
setMap
(NgramMap<ProbBackoffPair> map) Initializes a value container with the map that contains itvoid
setSizeAtLeast
(long size, int ngramOrder) Sets internal storage for size for a particular n-gram orderboolean
void
swap
(long a, long b, int ngramOrder) Swaps values at offsets a and b.void
trim()
Final clean up of storage.void
trimAfterNgram
(int ngramOrder, long size) Clear storage after an n-gram order is completeprotected boolean
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
Methods inherited from interface edu.berkeley.nlp.lm.values.ProbBackoffValueContainer
getSuffixOffset
Methods inherited from interface edu.berkeley.nlp.lm.values.ValueContainer
add, clearStorageForOrder, numValueBits, setMap, setSizeAtLeast, storeSuffixoffsets, trimAfterNgram
-
Field Details
-
valueRanks
-
storeSuffixIndexes
protected final boolean storeSuffixIndexes -
valueCoder
-
valueRadix
protected final int valueRadix -
valueWidth
protected int valueWidth -
defaultValRank
protected final int defaultValRank- See Also:
-
numNgramsForEachOrder
protected final long[] numNgramsForEachOrder -
suffixBitsForOrder
protected final int[] suffixBitsForOrder -
useMapValueArray
protected boolean useMapValueArray
-
-
Constructor Details
-
CompressibleProbBackoffValueContainer
public CompressibleProbBackoffValueContainer(LongToIntHashMap countCounter, int valueRadix, boolean storePrefixes, long[] numNgramsForEachOrder) -
CompressibleProbBackoffValueContainer
public CompressibleProbBackoffValueContainer(int valueRadix, boolean storePrefixIndexes, long[] numNgramsForEachOrder, float[] probsForRank, float[] backoffsForRank, Indexer<Float> probIndexer, int wordWidth, Indexer<Float> backoffIndexer, int backoffWidth) - Parameters:
valueRadix
-storePrefixIndexes
-maxNgramOrder
-hasBackoffValIndexer
-noBackoffValIndexer
-probsAndBackoffsForRank
-probsForRank
-hasBackoffValIndexer
-
-
-
Method Details
-
createFreshValues
Description copied from interface:ValueContainer
Creates a fresh value container for copying purposes.- Specified by:
createFreshValues
in interfaceValueContainer<ProbBackoffPair>
- Returns:
-
getProb
public final float getProb(int ngramOrder, long index) - Specified by:
getProb
in interfaceProbBackoffValueContainer
-
getInternalVal
public final long getInternalVal(int ngramOrder, long index) -
getProb
-
getFromOffset
Description copied from interface:ValueContainer
Gets the value living at a particular offset.- Specified by:
getFromOffset
in interfaceValueContainer<ProbBackoffPair>
-
getBackoff
public final float getBackoff(int ngramOrder, long index) - Specified by:
getBackoff
in interfaceProbBackoffValueContainer
-
getBackoff
-
getDefaultVal
-
getFromRank
-
getScratchValue
Description copied from interface:ValueContainer
Creates a fresh value of object (useful for passing as an output parameter)- Specified by:
getScratchValue
in interfaceProbBackoffValueContainer
- Specified by:
getScratchValue
in interfaceValueContainer<ProbBackoffPair>
- Returns:
-
setFromOtherValues
Description copied from interface:ValueContainer
Destructively sets internal storage from another object.- Specified by:
setFromOtherValues
in interfaceValueContainer<ProbBackoffPair>
-
trim
public void trim()Description copied from interface:ValueContainer
Final clean up of storage.- Specified by:
trim
in interfaceValueContainer<ProbBackoffPair>
-
getCountRank
protected long getCountRank(long val) -
getCompressed
Description copied from interface:CompressibleValueContainer
Compresses the value at the given offset into a list of bits.- Specified by:
getCompressed
in interfaceCompressibleValueContainer<ProbBackoffPair>
- Returns:
-
decompress
public final void decompress(BitStream bits, int ngramOrder, boolean justConsume, ProbBackoffPair outputVal) Description copied from interface:CompressibleValueContainer
Reads and decompresses from the bit stream bits.- Specified by:
decompress
in interfaceCompressibleValueContainer<ProbBackoffPair>
justConsume
- If true, nothing is returned, and the function simply consumes the appropriate number of bits from the BitStream.
-
setMap
Description copied from interface:ValueContainer
Initializes a value container with the map that contains it- Specified by:
setMap
in interfaceValueContainer<V extends LongRepresentable<V>>
-
useValueStoringArray
protected boolean useValueStoringArray() -
swap
public void swap(long a, long b, int ngramOrder) Description copied from interface:CompressibleValueContainer
Swaps values at offsets a and b.- Specified by:
swap
in interfaceCompressibleValueContainer<V extends LongRepresentable<V>>
-
add
public boolean add(int[] ngram, int startPos, int endPos, int ngramOrder, long offset, long prefixOffset, int word, ProbBackoffPair val_, long suffixOffset, boolean ngramIsNew) Description copied from interface:ValueContainer
Adds a new value at the specified offset.- Specified by:
add
in interfaceValueContainer<V extends LongRepresentable<V>>
ngramOrder
- As always, ngramOrder is 0-based (0=unigram)- Returns:
- Whether or not the add was successful
-
setSizeAtLeast
public void setSizeAtLeast(long size, int ngramOrder) Description copied from interface:ValueContainer
Sets internal storage for size for a particular n-gram order- Specified by:
setSizeAtLeast
in interfaceValueContainer<V extends LongRepresentable<V>>
-
getSuffixOffset
public long getSuffixOffset(long index, int ngramOrder) -
getSuffixOffsetFromInternalVal
protected int getSuffixOffsetFromInternalVal(long internalVal) - Parameters:
internalVal
-- Returns:
-
getRank
protected long getRank(int ngramOrder, long offset) -
clearStorageAfterCompression
public void clearStorageAfterCompression(int ngramOrder) - Specified by:
clearStorageAfterCompression
in interfaceCompressibleValueContainer<V extends LongRepresentable<V>>
-
trimAfterNgram
public void trimAfterNgram(int ngramOrder, long size) Description copied from interface:ValueContainer
Clear storage after an n-gram order is complete- Specified by:
trimAfterNgram
in interfaceValueContainer<V extends LongRepresentable<V>>
-
clearStorageForOrder
public void clearStorageForOrder(int ngramOrder) - Specified by:
clearStorageForOrder
in interfaceValueContainer<V extends LongRepresentable<V>>
-
storeSuffixoffsets
public boolean storeSuffixoffsets()- Specified by:
storeSuffixoffsets
in interfaceValueContainer<V extends LongRepresentable<V>>
-
numValueBits
public int numValueBits(int ngramOrder) - Specified by:
numValueBits
in interfaceValueContainer<V extends LongRepresentable<V>>
-