nl.inl.util

Class LuceneUtil

java.lang.Object

nl.inl.util.LuceneUtil

public final class LuceneUtil
extends Object

Method Summary

Modifier and Type	Method and Description
static List<String>	findTermsByPrefix(org.apache.lucene.index.IndexReader index, String fieldName, String prefix, boolean sensitive) Find terms in the index based on a prefix.
static List<String>	findTermsByPrefix(org.apache.lucene.index.IndexReader index, String fieldName, String prefix, boolean sensitive, int maxResults) Find terms in the index based on a prefix.
static List<String>	getFieldTerms(org.apache.lucene.index.IndexReader index, String fieldName) Return the list of terms that occur in a field.
static List<String>	getFieldTerms(org.apache.lucene.index.IndexReader index, String fieldName, int maxResults) Return the list of terms that occur in a field.
static void	getFrequenciesFromTermVector(org.apache.lucene.index.IndexReader reader, int doc, String luceneName, Map<String,Integer> freq) Add term frequencies for a single document to a frequency map.
static org.apache.lucene.index.IndexWriterConfig	getIndexWriterConfig(org.apache.lucene.analysis.Analyzer analyzer, boolean create)
static Set<String>	getMatchingTermsFromIndex(org.apache.lucene.index.IndexReader reader, String luceneName, Collection<String> searchTerms, int maxEdits) Get all the terms in the index with low edit distance from the supplied term
static Map<String,Set<String>>	getOldSingleFieldSubprops(org.apache.lucene.index.IndexReader index, String fieldName) Enumerate all the terms in the given Lucene field, collecting all the subproperty names and values.
static long	getSumTotalTermFreq(org.apache.lucene.index.IndexReader reader, String luceneField)
static String[]	getWordsFromTermVector(org.apache.lucene.index.IndexReader reader, int doc, String luceneName, int start, int end) Get all words between the specified start and end positions from the term vector.
static String[]	getWordsFromTermVector(org.apache.lucene.index.IndexReader reader, int doc, String luceneName, int start, int end, boolean partialOk) Get all words between the specified start and end positions from the term vector.
static org.apache.lucene.search.Query	parseLuceneQuery(String luceneQuery, org.apache.lucene.analysis.Analyzer analyzer, String defaultField) Parse a query in the Lucene query language format (QueryParser supplied with Lucene).
static Map<String,Integer>	termFrequencies(org.apache.lucene.search.IndexSearcher indexSearcher, org.apache.lucene.search.Query documentFilterQuery, AnnotationSensitivity annotSensitivity, Set<String> searchTerms)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

getMatchingTermsFromIndex

public static Set<String> getMatchingTermsFromIndex(org.apache.lucene.index.IndexReader reader,
                                                    String luceneName,
                                                    Collection<String> searchTerms,
                                                    int maxEdits)

Get all the terms in the index with low edit distance from the supplied term

Parameters:

reader - the index

luceneName - the field to search in

searchTerms - search terms

maxEdits - maximum edit distance (Levenshtein algorithm) for matches (i.e. lower is more similar)

Returns:

the set of terms in the index that are close to our search term

Throws:

BooleanQuery.TooManyClauses - if the expansion resulted in too many terms

parseLuceneQuery

public static org.apache.lucene.search.Query parseLuceneQuery(String luceneQuery,
                                                              org.apache.lucene.analysis.Analyzer analyzer,
                                                              String defaultField)
                                                       throws org.apache.lucene.queryparser.classic.ParseException

Parse a query in the Lucene query language format (QueryParser supplied with Lucene).

Parameters:

luceneQuery - the query string

analyzer - analyzer to use

defaultField - default search field

Returns:

the query

Throws:

org.apache.lucene.queryparser.classic.ParseException - on syntax error

getWordsFromTermVector

public static String[] getWordsFromTermVector(org.apache.lucene.index.IndexReader reader,
                                              int doc,
                                              String luceneName,
                                              int start,
                                              int end)

Get all words between the specified start and end positions from the term vector. NOTE: this may return an array of less than the size requested, if the document ends before the requested end position.

Parameters:

reader - the index

doc - doc id

luceneName - the index field from which to use the term vector

start - start position (first word we want to request)

end - end position (last word we want to request)

Returns:

the words found, in order

getWordsFromTermVector

public static String[] getWordsFromTermVector(org.apache.lucene.index.IndexReader reader,
                                              int doc,
                                              String luceneName,
                                              int start,
                                              int end,
                                              boolean partialOk)

Get all words between the specified start and end positions from the term vector. NOTE: this may return an array of less than the size requested, if the document ends before the requested end position.

Parameters:

reader - the index

doc - doc id

luceneName - the index field from which to use the term vector

start - start position (first word we want to request)

end - end position (last word we want to request)

partialOk - is it okay if we're missing words in the middle, or do we need them all? (debug)

Returns:

the words found, in order

getFrequenciesFromTermVector

public static void getFrequenciesFromTermVector(org.apache.lucene.index.IndexReader reader,
                                                int doc,
                                                String luceneName,
                                                Map<String,Integer> freq)

Add term frequencies for a single document to a frequency map.

Parameters:

reader - the index

doc - doc id

luceneName - the index field from which to use the term vector

freq - where to add to the token frequencies

getFieldTerms

public static List<String> getFieldTerms(org.apache.lucene.index.IndexReader index,
                                         String fieldName)

Return the list of terms that occur in a field.

Parameters:

index - the index

fieldName - the field

Returns:

the matching terms

getFieldTerms

public static List<String> getFieldTerms(org.apache.lucene.index.IndexReader index,
                                         String fieldName,
                                         int maxResults)

Return the list of terms that occur in a field.

Parameters:

index - the index

fieldName - the field

maxResults - maximum number to return (or -1 for no limit)

Returns:

the matching terms

findTermsByPrefix

public static List<String> findTermsByPrefix(org.apache.lucene.index.IndexReader index,
                                             String fieldName,
                                             String prefix,
                                             boolean sensitive)

Find terms in the index based on a prefix. Useful for autocomplete. NOTE: no limit on the number of results!

Parameters:

index - the index

fieldName - the field

prefix - the prefix we're looking for

sensitive - match case-sensitively or not?

Returns:

the matching terms

findTermsByPrefix

public static List<String> findTermsByPrefix(org.apache.lucene.index.IndexReader index,
                                             String fieldName,
                                             String prefix,
                                             boolean sensitive,
                                             int maxResults)

Find terms in the index based on a prefix. Useful for autocomplete.

Parameters:

index - the index

fieldName - the field

prefix - the prefix we're looking for (null or empty string for all terms)

sensitive - match case-sensitively or not?

maxResults - max. number of results to return (or -1 for all)

Returns:

the matching terms

termFrequencies

public static Map<String,Integer> termFrequencies(org.apache.lucene.search.IndexSearcher indexSearcher,
                                                  org.apache.lucene.search.Query documentFilterQuery,
                                                  AnnotationSensitivity annotSensitivity,
                                                  Set<String> searchTerms)

Parameters:

indexSearcher -

documentFilterQuery - document filter, or null for all documents

annotSensitivity - field to get frequencies for

searchTerms - list of terms to get frequencies for, or null for all terms

Returns:

getIndexWriterConfig

public static org.apache.lucene.index.IndexWriterConfig getIndexWriterConfig(org.apache.lucene.analysis.Analyzer analyzer,
                                                                             boolean create)

getSumTotalTermFreq

public static long getSumTotalTermFreq(org.apache.lucene.index.IndexReader reader,
                                       String luceneField)

getOldSingleFieldSubprops

public static Map<String,Set<String>> getOldSingleFieldSubprops(org.apache.lucene.index.IndexReader index,
                                                                String fieldName)

Enumerate all the terms in the given Lucene field, collecting all the subproperty names and values. Usually used for part of speech, where all the features are stored as separate subproperties.

Parameters:

index - our index

fieldName - field in the Lucene index to enumerate terms from

Returns:

subproperties and their values