nl.inl.blacklab.forwardindex

Class Terms

java.lang.Object

nl.inl.blacklab.forwardindex.Terms

Direct Known Subclasses:

MockTerms

public abstract class Terms
extends Object

Keeps a list of unique terms and their sort positions.

Field Summary

Fields

Modifier and Type	Field and Description
protected static int	BYTES_PER_INT Number of bytes per int
protected static Charset	DEFAULT_CHARSET
protected static int	DEFAULT_MAX_MAP_SIZE We set this to a lower value on Windows because we can't properly truncate the file due to the file still being mapped (there is no clean way to unmap a mapped file in Java, and Windows doesn't allow truncating a mapped file).
static int	NO_TERM

Constructor Summary

Constructors

Constructor and Description
Terms()

Method Summary

Modifier and Type	Method and Description
abstract void	clear() Clear the Terms object.
int	compareSortPosition(int termId1, int termId2, MatchSensitivity sensitivity) Compare two terms (from their term ids) based on their sort positions
int	deserializeToken(String term)
abstract String	get(int id) Get a term by id.
abstract int	idToSortPosition(int id, MatchSensitivity sensitivity) Get the sort position for a term based on its term id
abstract void	indexOf(org.eclipse.collections.api.set.primitive.MutableIntSet results, String term, MatchSensitivity sensitivity) Get the index number(s) of terms matching a string.
abstract int	indexOf(String term) Get the existing index number of a term, or add it to the term list and assign it a new index number.
void	initialize()
abstract int	numberOfTerms()
static Terms	openForReading(Collators collators, File termsFile, boolean useBlockBasedTermsFile, boolean buildTermIndexesOnInit)
static Terms	openForWriting(Collators collators, File termsFile, boolean useBlockBasedTermsFile)
protected IntBuffer	readFromFileChannel(FileChannel fc, long fileLength)
String	serializeTerm(int valueTokenId)
protected abstract void	setBlockBasedFile(boolean useBlockBasedTermsFile)
abstract boolean	termsEqual(int[] termId, MatchSensitivity sensitivity)
void	toSortOrder(int[] termId, int[] sortOrder, MatchSensitivity sensitivity) Convert an array of term ids to sort positions
abstract void	write(File termsFile) Write the terms file

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

NO_TERM

public static final int NO_TERM

See Also:

Constant Field Values

DEFAULT_CHARSET

protected static final Charset DEFAULT_CHARSET

DEFAULT_MAX_MAP_SIZE

protected static final int DEFAULT_MAX_MAP_SIZE

We set this to a lower value on Windows because we can't properly truncate the file due to the file still being mapped (there is no clean way to unmap a mapped file in Java, and Windows doesn't allow truncating a mapped file). The lower value on Windows prevents too much wasted space.

BYTES_PER_INT

protected static final int BYTES_PER_INT

Number of bytes per int

See Also:

Constant Field Values

Constructor Detail

Terms

public Terms()

Method Detail

initialize

public void initialize()

indexOf

public abstract int indexOf(String term)

Get the existing index number of a term, or add it to the term list and assign it a new index number. In index mode, this is fast. In search mode, this is slower, because we have to do a binary search through the memory-mapped terms file. However, this is only done rarely. If you care about this being fast, call buildTermIndex() at the start of your application.

Parameters:

term - the term to get the index number for

Returns:

the term's index number

indexOf

public abstract void indexOf(org.eclipse.collections.api.set.primitive.MutableIntSet results,
                             String term,
                             MatchSensitivity sensitivity)

Get the index number(s) of terms matching a string. This is used in search mode when translating queries into NFAs. Depending on case-sensitivity settings, a single term string may match multiple terms.

Parameters:

results - (out) index numbers for the matching term(s)

term - the term to get the index number for

sensitivity - compare sensitively? (case-sensitivity currently switches both case- and diacritics-sensitivity)

clear

public abstract void clear()

Clear the Terms object.

readFromFileChannel

protected IntBuffer readFromFileChannel(FileChannel fc,
                                        long fileLength)
                                 throws IOException

Throws:

IOException

write

public abstract void write(File termsFile)

Write the terms file

Parameters:

termsFile - where to write the terms file

get

public abstract String get(int id)

Get a term by id. Only works in search mode.

Parameters:

id - the term id

Returns:

the corresponding term

numberOfTerms

public abstract int numberOfTerms()

Returns:

the number of terms in this object

idToSortPosition

public abstract int idToSortPosition(int id,
                                     MatchSensitivity sensitivity)

Get the sort position for a term based on its term id

Parameters:

id - the term id

sensitivity - whether we want the sensitive or insensitive sort position

Returns:

the sort position

toSortOrder

public void toSortOrder(int[] termId,
                        int[] sortOrder,
                        MatchSensitivity sensitivity)

Convert an array of term ids to sort positions

Parameters:

termId - the term ids

sortOrder - the sort positions

sensitivity - whether we want the sensitive or insensitive sort positions

compareSortPosition

public int compareSortPosition(int termId1,
                               int termId2,
                               MatchSensitivity sensitivity)

Compare two terms (from their term ids) based on their sort positions

Parameters:

termId1 - id of the first term

termId2 - id of the second term

sensitivity - whether we want to compare sensitively or insensitively

Returns:

the comparison result (negative if term1 < term2, zero if equal, positive if term1 > term2)

setBlockBasedFile

protected abstract void setBlockBasedFile(boolean useBlockBasedTermsFile)

openForReading

public static Terms openForReading(Collators collators,
                                   File termsFile,
                                   boolean useBlockBasedTermsFile,
                                   boolean buildTermIndexesOnInit)

openForWriting

public static Terms openForWriting(Collators collators,
                                   File termsFile,
                                   boolean useBlockBasedTermsFile)

termsEqual

public abstract boolean termsEqual(int[] termId,
                                   MatchSensitivity sensitivity)

deserializeToken

public int deserializeToken(String term)

serializeTerm

public String serializeTerm(int valueTokenId)