nl.inl.blacklab.indexers.preprocess

Class DocIndexerConvertAndTag

java.lang.Object

nl.inl.blacklab.index.DocIndexer

nl.inl.blacklab.indexers.config.DocIndexerBase

nl.inl.blacklab.indexers.config.DocIndexerConfig

nl.inl.blacklab.indexers.preprocess.DocIndexerConvertAndTag

All Implemented Interfaces:

AutoCloseable

public class DocIndexerConvertAndTag
extends DocIndexerConfig

Wrapper class for a regular DocIndexer. It's activated when a format has the "convertPlugin" or "tagPlugin" properties. This DocIndexer will first run the to-be-indexed file through the convert and tagging plugins before handing result off to the actual DocIndexer.

It shares the ConfigInputFormat object with the actual DocIndexer, and should be considered an internal implementation detail of the DocIndexer system.

Field Summary

Fields inherited from class nl.inl.blacklab.indexers.config.DocIndexerConfig

config

Fields inherited from class nl.inl.blacklab.indexers.config.DocIndexerBase

wordsDone

Fields inherited from class nl.inl.blacklab.index.DocIndexer

currentLuceneDoc, documentName, docWriter, logger, MAX_DOCVALUES_LENGTH, metadataFieldValues, omitNorms, parameters

Constructor Summary

Constructors

Constructor and Description
DocIndexerConvertAndTag(DocIndexerConfig actualIndexer, ConfigInputFormat config)

Method Summary

Modifier and Type	Method and Description
void	addMetadataField(String fieldName, String value)
void	addNumericFields(Collection<String> fields)
void	close()
protected int	getCharacterPosition()
org.apache.lucene.document.Document	getCurrentLuceneDoc()
DocWriter	getDocWriter() Returns our DocWriter object
void	index() Index documents contained in a file.
void	indexSpecificDocument(String documentExpr) Index a specific document.
void	setConfigInputFormat(ConfigInputFormat config)
void	setDocument(InputStream is, Charset cs) Set the document to index.
void	setDocument(Reader reader) Use setDocument(InputStream, Charset) if at all possible.
void	setDocWriter(DocWriter indexer) Set the DocWriter object.
void	setOmitNorms(boolean b) Enables or disables norms.
boolean	shouldAddDefaultPunctuation()
protected void	storeDocument() Store (or finish storing) the document in the content store.

Methods inherited from class nl.inl.blacklab.indexers.config.DocIndexerConfig

fromConfig, getSensitivitySetting, init, opChatFormatAgeToMonths, optTranslateFieldName, processLinkedDocument, processMetadataValue, processString, processStringMultipleValues, replaceDollarRefs

Methods inherited from class nl.inl.blacklab.indexers.config.DocIndexerBase

addAnnotatedField, addEndChar, addStartChar, annotation, beginWord, dedupe, endDocument, endWord, getAnnotatedField, getAnnotatedFields, getAnnotation, getContentStoreName, getCurrentTokenPosition, getMainAnnotatedField, getMetadataFetcher, indexLinkedDocument, inlineTag, isStoreDocuments, propMain, propPunct, propTags, punctuation, reportCharsProcessed, reportTokensProcessed, resolveFileReference, setAddDefaultPunctuation, setCurrentAnnotatedFieldName, setPreventNextDefaultPunctuation, setStoreDocuments, startDocument, storeWholeDocument, trace, traceln

Methods inherited from class nl.inl.blacklab.index.DocIndexer

addMetadataFieldsFromParameters, addMetadataToDocument, addToForwardIndex, getMetadataField, getMetadataFieldTypeFromIndexerProperties, getParameter, getParameter, getParameter, getParameter, getSensitivitySetting, hasParameter, luceneTypeFromIndexMetadataType, setDocument, setDocument, setDocumentName, setParameter, setParameters, tokenizeField, warn

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

DocIndexerConvertAndTag

public DocIndexerConvertAndTag(DocIndexerConfig actualIndexer,
                               ConfigInputFormat config)

Method Detail

close

public void close()
           throws BlackLabRuntimeException

Specified by:

close in interface AutoCloseable

Specified by:

close in class DocIndexer

Throws:

BlackLabRuntimeException

setDocument

public void setDocument(Reader reader)

Use setDocument(InputStream, Charset) if at all possible.

Specified by:

setDocument in class DocIndexer

Parameters:

reader - document

setDocument

public void setDocument(InputStream is,
                        Charset cs)

Description copied from class: DocIndexer

Set the document to index.

Overrides:

setDocument in class DocIndexer

Parameters:

is - document contents

cs - charset to use if no BOM found, or null for the default (utf-8)

index

public void index()
           throws PluginException,
                  MalformedInputFile,
                  IOException

Description copied from class: DocIndexer

Index documents contained in a file.

Overrides:

index in class DocIndexerConfig

Throws:

PluginException - if an error occurred in a plugin

MalformedInputFile - if the input file wasn't valid

IOException - if an I/O error occurred

getCharacterPosition

protected int getCharacterPosition()

Specified by:

getCharacterPosition in class DocIndexer

storeDocument

protected void storeDocument()

Description copied from class: DocIndexerBase

Store (or finish storing) the document in the content store. Also set the content id field so we know how to retrieve it later.

Specified by:

storeDocument in class DocIndexerBase

setDocWriter

public void setDocWriter(DocWriter indexer)

Description copied from class: DocIndexer

Set the DocWriter object. We use this to add documents to the index. Called by Indexer when the DocIndexer is instantiated.

Overrides:

setDocWriter in class DocIndexer

Parameters:

indexer - our DocWriter object

addMetadataField

public void addMetadataField(String fieldName,
                             String value)

Overrides:

addMetadataField in class DocIndexerBase

addNumericFields

public void addNumericFields(Collection<String> fields)

Overrides:

addNumericFields in class DocIndexer

getCurrentLuceneDoc

public org.apache.lucene.document.Document getCurrentLuceneDoc()

Overrides:

getCurrentLuceneDoc in class DocIndexer

getDocWriter

public DocWriter getDocWriter()

Description copied from class: DocIndexer

Returns our DocWriter object

Overrides:

getDocWriter in class DocIndexer

Returns:

the DocWriter object

indexSpecificDocument

public void indexSpecificDocument(String documentExpr)

Description copied from class: DocIndexerBase

Index a specific document. Only supported by DocIndexerBase.

Overrides:

indexSpecificDocument in class DocIndexerConfig

Parameters:

documentExpr - Expression (e.g. XPath) used to find the document to index in the file

setConfigInputFormat

public void setConfigInputFormat(ConfigInputFormat config)

Overrides:

setConfigInputFormat in class DocIndexerConfig

setOmitNorms

public void setOmitNorms(boolean b)

Description copied from class: DocIndexer

Enables or disables norms. Norms are disabled by default. The method name was chosen to match Lucene's Field.setOmitNorms(). Norms are only required if you want to use document-length-normalized scoring.

Overrides:

setOmitNorms in class DocIndexer

Parameters:

b - if true, doesn't store norms; if false, does store norms

shouldAddDefaultPunctuation

public boolean shouldAddDefaultPunctuation()

Overrides:

shouldAddDefaultPunctuation in class DocIndexerBase