nl.inl.blacklab.index

Class DocIndexer

java.lang.Object

nl.inl.blacklab.index.DocIndexer

All Implemented Interfaces:

AutoCloseable

Direct Known Subclasses:

DocIndexerAbstract, DocIndexerBase

public abstract class DocIndexer
extends Object
implements AutoCloseable

Indexes a file.

Field Summary

Fields

Modifier and Type	Field and Description
protected org.apache.lucene.document.Document	currentLuceneDoc The Lucene Document we're currently constructing (corresponds to the document we're indexing)
protected String	documentName File we're currently parsing.
protected DocWriter	docWriter
protected static org.apache.logging.log4j.Logger	logger
static int	MAX_DOCVALUES_LENGTH
protected Map<String,List<String>>	metadataFieldValues Document metadata.
protected boolean	omitNorms Do we want to omit norms? (Default: yes)
protected Map<String,String>	parameters Parameters passed to this indexer

Constructor Summary

Constructors

Constructor and Description
DocIndexer()

Method Summary

Modifier and Type	Method and Description
void	addMetadataField(String name, String value)
protected void	addMetadataFieldsFromParameters() If any metadata fields were supplied in the indexer parameters, add them now.
void	addMetadataToDocument() When all metadata values have been set, call this to add the to the Lucene document.
void	addNumericFields(Collection<String> fields)
protected void	addToForwardIndex(AnnotatedFieldWriter field) Add the field, with all its properties, to the forward index.
abstract void	close()
protected abstract int	getCharacterPosition()
org.apache.lucene.document.Document	getCurrentLuceneDoc()
DocWriter	getDocWriter() Returns our DocWriter object
List<String>	getMetadataField(String name)
FieldType	getMetadataFieldTypeFromIndexerProperties(String fieldName) Deprecated. use a DocIndexerConfig-based indexer
String	getParameter(String name) Deprecated. use a DocIndexerConfig-based indexer
boolean	getParameter(String name, boolean defaultValue) Deprecated. use a DocIndexerConfig-based indexer
int	getParameter(String name, int defaultValue) Deprecated. use a DocIndexerConfig-based indexer
String	getParameter(String name, String defaultValue) Deprecated. use ConfigInputFormat, IndexMetadata
AnnotationWriter.SensitivitySetting	getSensitivitySetting(String annotationName) Deprecated.
boolean	hasParameter(String name) Deprecated. use a DocIndexerConfig-based indexer
abstract void	index() Index documents contained in a file.
protected org.apache.lucene.document.FieldType	luceneTypeFromIndexMetadataType(FieldType type)
protected String	optTranslateFieldName(String from) Translate a field name before adding it to the Lucene document.
abstract void	reportCharsProcessed() Report the amount of new characters processed since the last call
abstract void	reportTokensProcessed() Report the amounf of new tokens processed since the last call
void	setDocument(byte[] contents, Charset cs) Set the document to index.
void	setDocument(File file, Charset charset) Set the document to index.
void	setDocument(InputStream is, Charset cs) Set the document to index.
abstract void	setDocument(Reader reader) Set the document to index.
void	setDocumentName(String documentName) Set the file name of the document to index.
void	setDocWriter(DocWriter docWriter) Set the DocWriter object.
void	setOmitNorms(boolean b) Enables or disables norms.
void	setParameter(String name, String value) Deprecated. use a DocIndexerConfig-based indexer
void	setParameters(Map<String,String> param) Deprecated. use a DocIndexerConfig-based indexer
protected boolean	tokenizeField(String name)
protected void	warn(String msg)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

logger

protected static final org.apache.logging.log4j.Logger logger

MAX_DOCVALUES_LENGTH

public static final int MAX_DOCVALUES_LENGTH

See Also:

Constant Field Values

docWriter

protected DocWriter docWriter

omitNorms

protected boolean omitNorms

Do we want to omit norms? (Default: yes)

documentName

protected String documentName

File we're currently parsing. This can be useful for storing the original filename in the index.

currentLuceneDoc

protected org.apache.lucene.document.Document currentLuceneDoc

The Lucene Document we're currently constructing (corresponds to the document we're indexing)

metadataFieldValues

protected Map<String,List<String>> metadataFieldValues

Document metadata. Added at the end to deal with unknown values, multiple occurrences (only the first is actually indexed, because of DocValues, among others), etc.

parameters

protected Map<String,String> parameters

Parameters passed to this indexer

Constructor Detail

DocIndexer

public DocIndexer()

Method Detail

close

public abstract void close()

Specified by:

close in interface AutoCloseable

getCurrentLuceneDoc

public org.apache.lucene.document.Document getCurrentLuceneDoc()

getDocWriter

public DocWriter getDocWriter()

Returns our DocWriter object

Returns:

the DocWriter object

setDocWriter

public void setDocWriter(DocWriter docWriter)

Set the DocWriter object. We use this to add documents to the index. Called by Indexer when the DocIndexer is instantiated.

Parameters:

docWriter - our DocWriter object

setDocumentName

public void setDocumentName(String documentName)

Set the file name of the document to index.

Parameters:

documentName - name of the document

setDocument

public abstract void setDocument(Reader reader)

Set the document to index. NOTE: you should generally prefer calling the File or byte[] versions of this method, as those can be more efficient (e.g. when using DocIndexer that parses using VTD-XML).

Parameters:

reader - document

setDocument

public void setDocument(InputStream is,
                        Charset cs)

Set the document to index.

Parameters:

is - document contents

cs - charset to use if no BOM found, or null for the default (utf-8)

setDocument

public void setDocument(byte[] contents,
                        Charset cs)

Set the document to index.

Parameters:

contents - document contents

cs - charset to use if no BOM found, or null for the default (utf-8)

setDocument

public void setDocument(File file,
                        Charset charset)
                 throws FileNotFoundException

Set the document to index.

Parameters:

file - file to index

charset - charset to use if no BOM found, or null for the default (utf-8)

Throws:

FileNotFoundException - if not found

index

public abstract void index()
                    throws IOException,
                           MalformedInputFile,
                           PluginException

Index documents contained in a file.

Throws:

MalformedInputFile - if the input file wasn't valid

IOException - if an I/O error occurred

PluginException - if an error occurred in a plugin

hasParameter

@Deprecated
public boolean hasParameter(String name)

Deprecated. use a DocIndexerConfig-based indexer

Check if the specified parameter has a value

Parameters:

name - parameter name

Returns:

true iff the parameter has a value

setParameter

@Deprecated
public void setParameter(String name,
                                     String value)

Deprecated. use a DocIndexerConfig-based indexer

Set a parameter for this indexer (such as which type of metadata block to process)

Parameters:

name - parameter name

value - parameter value

setParameters

@Deprecated
public void setParameters(Map<String,String> param)

Deprecated. use a DocIndexerConfig-based indexer

Set a number of parameters for this indexer

Parameters:

param - the parameter names and values

getParameter

@Deprecated
public String getParameter(String name,
                                       String defaultValue)

Deprecated. use ConfigInputFormat, IndexMetadata

Get a parameter that was set for this indexer

Parameters:

name - parameter name

defaultValue - parameter default value

Returns:

the parameter value (or the default value if it was not specified)

getParameter

@Deprecated
public String getParameter(String name)

Deprecated. use a DocIndexerConfig-based indexer

Get a parameter that was set for this indexer

Parameters:

name - parameter name

Returns:

the parameter value (or null if it was not specified)

getParameter

@Deprecated
public boolean getParameter(String name,
                                        boolean defaultValue)

Deprecated. use a DocIndexerConfig-based indexer

Get a parameter that was set for this indexer

Parameters:

name - parameter name

defaultValue - parameter default value

Returns:

the parameter value (or the default value if it was not specified)

getParameter

@Deprecated
public int getParameter(String name,
                                    int defaultValue)

Deprecated. use a DocIndexerConfig-based indexer

Get a parameter that was set for this indexer

Parameters:

name - parameter name

defaultValue - parameter default value

Returns:

the parameter value (or the default value if it was not specified)

tokenizeField

protected boolean tokenizeField(String name)

getMetadataFieldTypeFromIndexerProperties

@Deprecated
public FieldType getMetadataFieldTypeFromIndexerProperties(String fieldName)

Deprecated. use a DocIndexerConfig-based indexer

Return the fieldtype to use for the specified field.

Parameters:

fieldName - the field name

Returns:

the fieldtype

luceneTypeFromIndexMetadataType

protected org.apache.lucene.document.FieldType luceneTypeFromIndexMetadataType(FieldType type)

setOmitNorms

public void setOmitNorms(boolean b)

Enables or disables norms. Norms are disabled by default. The method name was chosen to match Lucene's Field.setOmitNorms(). Norms are only required if you want to use document-length-normalized scoring.

Parameters:

b - if true, doesn't store norms; if false, does store norms

addNumericFields

public void addNumericFields(Collection<String> fields)

warn

protected void warn(String msg)

getMetadataField

public List<String> getMetadataField(String name)

addMetadataField

public void addMetadataField(String name,
                             String value)

optTranslateFieldName

protected String optTranslateFieldName(String from)

Translate a field name before adding it to the Lucene document. By default, simply returns the input. May be overridden to change the name of a metadata field as it is indexed.

Parameters:

from - original metadata field name

Returns:

new name

addMetadataToDocument

public void addMetadataToDocument()

When all metadata values have been set, call this to add the to the Lucene document. We do it this way because we don't want to add multiple values for a field (DocValues and Document.get() only deal with the first value added), and we want to set an "unknown value" in certain conditions, depending on the configuration.

addMetadataFieldsFromParameters

protected void addMetadataFieldsFromParameters()

If any metadata fields were supplied in the indexer parameters, add them now. NOTE: we always add these untokenized (because they're usually just indications of which data set a set of files belongs to), but that means they don't get lowercased or de-accented. Because metadata queries are always desensitized, you can't use uppercase or accented letters in these values or they will never be found. This should be addressed. NOTE2: This way of adding metadata values is deprecated. Use an input format config file instead, and configure a field with a fixed value.

getSensitivitySetting

@Deprecated
public AnnotationWriter.SensitivitySetting getSensitivitySetting(String annotationName)

Deprecated.

addToForwardIndex

protected void addToForwardIndex(AnnotatedFieldWriter field)

Add the field, with all its properties, to the forward index.

Parameters:

field - field to add to the forward index

getCharacterPosition

protected abstract int getCharacterPosition()

reportCharsProcessed

public abstract void reportCharsProcessed()

Report the amount of new characters processed since the last call

reportTokensProcessed

public abstract void reportTokensProcessed()

Report the amounf of new tokens processed since the last call