nl.inl.blacklab.forwardindex

Class AnnotationForwardIndex

java.lang.Object

nl.inl.blacklab.forwardindex.AnnotationForwardIndex

Direct Known Subclasses:

MockForwardIndex

public abstract class AnnotationForwardIndex
extends Object

A component that can quickly tell you what word occurs at a specific position of a specific document.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	AnnotationForwardIndex.CollatorVersion Different versions of insensitive collator
static interface	AnnotationForwardIndex.ForwardIndexDocTask A task to perform on a document in the forward index.

Field Summary

Fields

Modifier and Type	Field and Description
protected boolean	initialized Has the tokens file been mapped?

Constructor Summary

Constructors

Constructor and Description
AnnotationForwardIndex(Annotation annotation, File dir, Collators collators, boolean largeTermsFileSupport)

Method Summary

Modifier and Type	Method and Description
int	addDocument(List<String> content) Store the given content and assign an id to it
abstract int	addDocument(List<String> content, List<Integer> posIncr) Store the given content and assign an id to it.
Annotation	annotation() The annotation for which this is the forward index
boolean	canDoNfaMatching()
abstract void	close() Close the forward index.
abstract void	deleteDocument(int fiid) Delete a document from the forward index
void	deleteDocumentByLuceneDoc(org.apache.lucene.document.Document d)
abstract int	docLength(int fiid) Gets the length (in tokens) of a document
void	forEachDocument(AnnotationForwardIndex.ForwardIndexDocTask task) Perform a task on each document in the forward index.
abstract int	freeBlocks()
abstract long	freeSpace()
int	getToken(int fiid, int pos)
abstract Set<Integer>	idSet()
void	initialize()
abstract int	numDocs()
static AnnotationForwardIndex	open(File dir, boolean indexMode, Collator collator, boolean create, Annotation annotation, boolean buildTermIndexesOnInit) Open a forward index.
abstract List<int[]>	retrievePartsInt(int fiid, int[] start, int[] end) Retrieve one or more parts from the specified content, in the form of token ids.
protected void	setLargeTermsFileSupport(boolean b)
Terms	terms() Get the Terms object in order to translate ids to token strings
String	toString()
long	totalSize()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

initialized

protected boolean initialized

Has the tokens file been mapped?

Constructor Detail

AnnotationForwardIndex

public AnnotationForwardIndex(Annotation annotation,
                              File dir,
                              Collators collators,
                              boolean largeTermsFileSupport)

Method Detail

open

public static AnnotationForwardIndex open(File dir,
                                          boolean indexMode,
                                          Collator collator,
                                          boolean create,
                                          Annotation annotation,
                                          boolean buildTermIndexesOnInit)

Open a forward index. Automatically figures out the forward index version and instantiates the right class.

Parameters:

dir - forward index directory

indexMode - true iff we're in index mode (writing to the forward index); otherwise it will be read-only.

collator - collator to use for sorting

create - if true, create a new forward index

annotation - annotation for which this is the forward index, or null if we don't know (yet)

fiidLookup - how to look up fiid given docId

buildTermIndexesOnInit - whether to build term indexes right away or lazily

Returns:

the forward index object

initialize

public void initialize()

close

public abstract void close()

Close the forward index. Writes the table of contents to disk if modified.

addDocument

public abstract int addDocument(List<String> content,
                                List<Integer> posIncr)

Store the given content and assign an id to it. Note that if more than one token occurs at any position, we only store the first in the forward index.

Parameters:

content - the content to store

posIncr - the associated position increments, or null if position increment is always 1.

Returns:

the id assigned to the content

addDocument

public int addDocument(List<String> content)

Store the given content and assign an id to it

Parameters:

content - the content to store

Returns:

the id assigned to the content

deleteDocument

public abstract void deleteDocument(int fiid)

Delete a document from the forward index

Parameters:

fiid - id of the document to delete

deleteDocumentByLuceneDoc

public void deleteDocumentByLuceneDoc(org.apache.lucene.document.Document d)

retrievePartsInt

public abstract List<int[]> retrievePartsInt(int fiid,
                                             int[] start,
                                             int[] end)

Retrieve one or more parts from the specified content, in the form of token ids. This is more efficient than retrieving the whole content, or retrieving parts in separate calls, because the file is only opened once and random access is used to read only the required parts. NOTE: if offset and length are both -1, retrieves the whole content. This is used by the retrieve(id) method. NOTE2: Mapped file IO on Windows has some issues that sometimes cause an OutOfMemoryError on the FileChannel.map() call (which makes no sense, because memory mapping only uses address space, it doesn't try to read the whole file). Possibly this could be solved by using 64-bit Java, but we haven't tried. For now we just disable memory mapping on Windows.

Parameters:

fiid - forward index document id

start - the starting points of the parts to retrieve (in words) (-1 for start of document)

end - the end points (i.e. first token beyond) of the parts to retrieve (in words) (-1 for end of document)

Returns:

the parts

terms

public Terms terms()

Get the Terms object in order to translate ids to token strings

Returns:

the Terms object

numDocs

public abstract int numDocs()

Returns:

the number of documents in the forward index

freeSpace

public abstract long freeSpace()

Returns:

the amount of space in free blocks in the forward index.

freeBlocks

public abstract int freeBlocks()

Returns:

the number of free blocks in the forward index.

totalSize

public long totalSize()

Returns:

total size in bytes of the tokens file.

docLength

public abstract int docLength(int fiid)

Gets the length (in tokens) of a document

Parameters:

fiid - forward index id of a document

Returns:

length of the document

setLargeTermsFileSupport

protected void setLargeTermsFileSupport(boolean b)

forEachDocument

public void forEachDocument(AnnotationForwardIndex.ForwardIndexDocTask task)

Perform a task on each document in the forward index.

Parameters:

task - the task to perform

getToken

public int getToken(int fiid,
                    int pos)

annotation

public Annotation annotation()

The annotation for which this is the forward index

Returns:

annotation

idSet

public abstract Set<Integer> idSet()

canDoNfaMatching

public boolean canDoNfaMatching()

toString

public String toString()

Overrides:

toString in class Object