nl.inl.blacklab.contentstore

Class ContentStoreDirUtf8

java.lang.Object

nl.inl.blacklab.contentstore.ContentStore

nl.inl.blacklab.contentstore.ContentStoreDirAbstract

nl.inl.blacklab.contentstore.ContentStoreDirUtf8

Direct Known Subclasses:

ContentStoreDirZip

@NotThreadSafe
public class ContentStoreDirUtf8
extends ContentStoreDirAbstract

Store string content by id in a directory of compound files with a TOC file. Quickly retrieve (parts of) the string content. This implementation stores the strings in UTF-8 encoding to save disk space. To guarantee reasonably fast random access times, we keep track of "block offsets", which are byte offsets to the start of (fixed char size) blocks. Block size in bytes can be slightly larger than char size because some UTF-8 characters take up more than 1 byte. If the block size is 1000 chars, block offsets might be [0, 1011, 2015, 3020]. Thread-safety: not thread-safe in index mode, but thread-safe while searching

Nested Class Summary

Nested classes/interfaces inherited from class nl.inl.blacklab.contentstore.ContentStore

ContentStore.DocTask

Field Summary

Fields

Modifier and Type	Field and Description
protected int	newEntryBlockSizeCharacters What block size to use when adding a new document to the content store.

Fields inherited from class nl.inl.blacklab.contentstore.ContentStoreDirAbstract

dir

Constructor Summary

Constructors

Constructor and Description
ContentStoreDirUtf8(File dir)
ContentStoreDirUtf8(File dir, boolean create)

Method Summary

Modifier and Type	Method and Description
void	addToBlock(String contentPart) Add a piece of content to the current block
void	clear() Delete all content in the document store
void	close() Close the content store.
protected String	decodeBlock(byte[] buf, int offset, int length) Convert the byte buffer representation of a block back to the original String
void	delete(int id) Delete a document from the content store.
int	docLength(int id) Returns the document length in characters
protected byte[]	encodeBlock(String block) Convert the String representation of a block to a byte buffer
Set<Integer>	idSet() Returns the set of doc ids in the store.
void	initialize()
boolean	isDeleted(int id) Return true iff the entry with this id was deleted.
String	retrieve(int id) Retrieve content with given id
String[]	retrieveParts(int contentId, int[] start, int[] end) Retrieve one or more substrings from the specified content.
void	setBlockSizeCharacters(int size) Set the desired block size for new entries
void	setDataFileSizeHint(long dataFileSizeHint) Indicate preferred maximum size of data files (defaults to 10M)
protected void	setStoreType() Writes an empty file that indicates the type of store.
void	setWriteMapReserve(int writeMapReserve) Set the size of the write reserve (the amount of space allocated at the end of the file).
int	store(String content) Store the given content and assign an id to it
void	storePart(String content) Store part of a piece of large content.
void	writeCurrentBlock(OutputStream os) Encode and write the block we've compiled so far and reset for next block

Methods inherited from class nl.inl.blacklab.contentstore.ContentStoreDirAbstract

getStoreTypeVersion, setStoreType, toString

Methods inherited from class nl.inl.blacklab.contentstore.ContentStore

forEachDocument, open, retrievePart

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

newEntryBlockSizeCharacters

protected int newEntryBlockSizeCharacters

What block size to use when adding a new document to the content store. Contributing factors for choosing block size: - larger blocks improve compression ratio - larger blocks decrease number of blocks you have to read - smaller blocks decrease the decompression time - smaller blocks increase the chance that we only have to read one disk block for a single concordance (disk blocks are generally 2 or 4K) - consider OS I/O caching and memory mapping. Then it becomes the difference between reading a few bytes from memory and reading a few kilobytes and decompressing them. Right now, making concordances is often CPU-bound (because of decompression?)

Constructor Detail

ContentStoreDirUtf8

public ContentStoreDirUtf8(File dir)

Parameters:

dir - content store dir

ContentStoreDirUtf8

public ContentStoreDirUtf8(File dir,
                           boolean create)

Parameters:

dir - content store dir

create - if true, create a new content store

Method Detail

setWriteMapReserve

public void setWriteMapReserve(int writeMapReserve)

Set the size of the write reserve (the amount of space allocated at the end of the file). Larger reserves means less re-mapping. The default is 1M bytes.

Parameters:

writeMapReserve - size of the reserve in bytes.

setBlockSizeCharacters

public void setBlockSizeCharacters(int size)

Set the desired block size for new entries

Parameters:

size - the fixed block size in characters

setStoreType

protected void setStoreType()

Writes an empty file that indicates the type of store. Subclasses should override this to customize the type string.

clear

public void clear()

Delete all content in the document store

Specified by:

clear in class ContentStore

close

public void close()

Close the content store. Writes the table of contents (if modified)

Specified by:

close in class ContentStore

setDataFileSizeHint

public void setDataFileSizeHint(long dataFileSizeHint)

Indicate preferred maximum size of data files (defaults to 10M)

Parameters:

dataFileSizeHint -

addToBlock

public void addToBlock(String contentPart)

Add a piece of content to the current block

Parameters:

contentPart - content to add

writeCurrentBlock

public void writeCurrentBlock(OutputStream os)

Encode and write the block we've compiled so far and reset for next block

Parameters:

os - where to write to block to

storePart

public void storePart(String content)

Store part of a piece of large content. This may be called several times to store chunks of content, but MUST be *finished* by calling the "normal" store() method. You may call store() with the empty string if you wish.

Specified by:

storePart in class ContentStore

Parameters:

content - the content to store

encodeBlock

protected byte[] encodeBlock(String block)

Convert the String representation of a block to a byte buffer

Parameters:

block - the block content

Returns:

the byte buffer representation

decodeBlock

protected String decodeBlock(byte[] buf,
                             int offset,
                             int length)

Convert the byte buffer representation of a block back to the original String

Parameters:

buf - the byte buffer

offset - offset in the buffer

length - length of the block in bytes

Returns:

the original String content

store

public int store(String content)

Store the given content and assign an id to it

Specified by:

store in class ContentStore

Parameters:

content - the content to store

Returns:

the id assigned to the content

retrieve

public String retrieve(int id)

Retrieve content with given id

Specified by:

retrieve in class ContentStore

Parameters:

id - the id

Returns:

the string

retrieveParts

public String[] retrieveParts(int contentId,
                              int[] start,
                              int[] end)

Retrieve one or more substrings from the specified content. This is more efficient than retrieving the whole content, or retrieving parts in separate calls, because the file is only opened once and random access is used to read only the required parts. NOTE: if offset and length are both -1, retrieves the whole content. This is used by the retrieve(id) method.

Specified by:

retrieveParts in class ContentStore

Parameters:

contentId - id of the entry to get substrings from

start - the starting points of the substrings (in characters). -1 means "start of document"

end - the end points of the substrings (in characters). -1 means "end of document"

Returns:

the parts

delete

public void delete(int id)

Description copied from class: ContentStore

Delete a document from the content store.

Specified by:

delete in class ContentStore

Parameters:

id - content store id of the document to delete

isDeleted

public boolean isDeleted(int id)

Description copied from class: ContentStore

Return true iff the entry with this id was deleted.

Specified by:

isDeleted in class ContentStore

Parameters:

id - the entry to check

Returns:

true iff deleted

docLength

public int docLength(int id)

Description copied from class: ContentStore

Returns the document length in characters

Specified by:

docLength in class ContentStore

Parameters:

id - the document

Returns:

the length in characters

idSet

public Set<Integer> idSet()

Description copied from class: ContentStore

Returns the set of doc ids in the store. Note that the IDs of deleted document are still returned by this method. Use isDeleted() to check.

Specified by:

idSet in class ContentStore

Returns:

the set of doc ids

initialize

public void initialize()

Specified by:

initialize in class ContentStore