Meziantou.Framework

LineBreakDictionary Class

Namespace
Meziantou.Framework
Assembly
Meziantou.Framework.Unicode.dll
Package
Meziantou.Framework.Unicode 2.0.15
Source
LineBreakDictionary.cs
Available in package assets
net10.0net11.0

Finds the positions where a line can break in text that is written without spaces between words, from a list of the words of a language.

public sealed class LineBreakDictionary

Remarks

Thai, Lao, Khmer, Myanmar and the other scripts whose characters have the line breaking class Complex_Context (SA) do not separate words with spaces. The Unicode Line Breaking Algorithm (Unicode Standard Annex #14) leaves such text to a dictionary-based or another language-specific analysis, and without one it only breaks at spaces. This class segments a text into the words of a list that the caller provides: no word list is included in this package. The International Components for Unicode (ICU) publish word lists for Thai, Lao, Khmer and Burmese under the Unicode License, with additional notices for some of them in the license file of ICU.

The accepted input is plain text:

  • One word per line. The white space around a word is ignored, as are the blank lines.
  • What follows the first tab of a line is ignored: a list can have other columns, such as a frequency.
  • # starts a comment that ends with the line.
  • A word is kept as written: it is not normalized and its case is not changed. Pass the text in the normalization form of the word list.

A text is segmented as a whole, as specified here. Unicode.EnumerateLineBreaks applies the same rule to each run of Complex_Context characters of a text.

  • A boundary is the start of the text, the end of the text, or a grapheme cluster boundary that is not before a combining mark, that is a character that rule LB9 of the annex attaches to the character before it. A line never breaks inside a grapheme cluster, nor before a mark.
  • A segmentation is a set of occurrences of words of the dictionary in the text that do not overlap, each of them between two boundaries. The characters that are in none of them are uncovered.
  • The chosen segmentation has the fewest uncovered UTF-16 code units, then the fewest words. When several remain, at the first position where two of them differ the one that has the longer word at that position is preferred, a word being preferred to an uncovered character.
  • A line can break before each word of the chosen segmentation, except at the start of the text. Uncovered characters give no break: they stay with what precedes them, and a text that has no word has no break.
  • A text longer than 1,024 UTF-16 code units is segmented piece by piece, in a time that is proportional to its length: the rule is applied to the longest part of its first 1,024 code units that ends at a boundary; the words and the uncovered characters that start in the first 512 code units are kept, and the rule is applied in the same way to the text that follows them. When there is no such part, because a grapheme cluster and its marks are longer than 1,024 code units, they are uncovered.

An instance is immutable and can be used by several threads at the same time.

See also

Properties

NameDescription
WordCountGets the number of distinct words.

WordCount

LineBreakDictionary.cs

Gets the number of distinct words.

public int WordCount { get; }

Property value

int

Methods

NameDescription
Contains(ReadOnlySpan<char>)Determines whether a word is in the dictionary.
FindBreaks(ReadOnlySpan<char>, Span<int>)Finds the positions where a line can break in a text that is written without spaces between words.
Load(Stream, LineBreakDictionaryOptions?)Loads a word list from a UTF-8 stream.
Load(string, LineBreakDictionaryOptions?)Loads a word list from a UTF-8 file.
LoadAsync(Stream, LineBreakDictionaryOptions?, CancellationToken)Loads a word list from a UTF-8 stream.
LoadAsync(string, LineBreakDictionaryOptions?, CancellationToken)Loads a word list from a UTF-8 file.
Parse(ReadOnlySpan<byte>, LineBreakDictionaryOptions?)Creates a dictionary from the UTF-8 text of a word list.
Parse(ReadOnlySpan<char>, LineBreakDictionaryOptions?)Creates a dictionary from the text of a word list.

Contains(ReadOnlySpan<char>)

LineBreakDictionary.cs

Determines whether a word is in the dictionary.

public bool Contains(ReadOnlySpan<char> word)

Parameters

word ReadOnlySpan<char>
The word. The comparison is ordinal.

Returns

bool true when the word is in the dictionary.

FindBreaks(ReadOnlySpan<char>, Span<int>)

LineBreakDictionary.cs

Finds the positions where a line can break in a text that is written without spaces between words.

public int FindBreaks(ReadOnlySpan<char> text, Span<int> breakPositions)

Parameters

text ReadOnlySpan<char>
The text: a run of characters of the script of the dictionary, without the spaces or the punctuation around it. It is segmented as a whole, whatever the line breaking classes of its characters.
breakPositions Span<int>
The buffer that receives the positions, in increasing order. A position is the index of the UTF-16 code unit of text before which a line can break: the first one of a word of the dictionary. As many elements as the length of the text are always enough.

Returns

int The number of positions written; 0 when the text has no word of the dictionary after its start.

Exceptions

ArgumentException
breakPositions is too small.

Remarks

The remarks of LineBreakDictionary specify the segmentation. A break is never at the start or at the end of the text, inside a grapheme cluster, or before a combining mark. The method does not allocate, whatever the length of the text.

Load overloads

Load(Stream, LineBreakDictionaryOptions?)

LineBreakDictionary.cs

Loads a word list from a UTF-8 stream.

public static LineBreakDictionary Load(Stream stream, LineBreakDictionaryOptions? options = null)
Parameters
stream Stream
The stream. It is read to its end and left open.
options LineBreakDictionaryOptions?
The options, or null for LineBreakDictionaryOptions.Default.
Returns

LineBreakDictionary The dictionary.

Exceptions
ArgumentNullException
stream is null.
FormatException
The content is not valid UTF-8, or exceeds a limit of options.

Load(string, LineBreakDictionaryOptions?)

LineBreakDictionary.cs

Loads a word list from a UTF-8 file.

public static LineBreakDictionary Load(string path, LineBreakDictionaryOptions? options = null)
Parameters
path string
The path of the file.
options LineBreakDictionaryOptions?
The options, or null for LineBreakDictionaryOptions.Default.
Returns

LineBreakDictionary The dictionary.

Exceptions
ArgumentNullException
path is null.
FormatException
The content is not valid UTF-8, or exceeds a limit of options.
IOException
The file cannot be read.

LoadAsync overloads

LoadAsync(Stream, LineBreakDictionaryOptions?, CancellationToken)

LineBreakDictionary.cs

Loads a word list from a UTF-8 stream.

net10.0

public static Task<LineBreakDictionary> LoadAsync(Stream stream, LineBreakDictionaryOptions? options = null, CancellationToken cancellationToken = default)

net11.0

public static Task<LineBreakDictionary> LoadAsync(Stream stream, LineBreakDictionaryOptions? options = null, CancellationToken cancellationToken = default)
Parameters
stream Stream
The stream. It is read to its end and left open.
options LineBreakDictionaryOptions?
The options, or null for LineBreakDictionaryOptions.Default.
cancellationToken CancellationToken
The token to monitor for cancellation requests.
Returns

Task<LineBreakDictionary> The dictionary.

Exceptions
ArgumentNullException
stream is null.
FormatException
The content is not valid UTF-8, or exceeds a limit of options.
OperationCanceledException
The operation was canceled.

LoadAsync(string, LineBreakDictionaryOptions?, CancellationToken)

LineBreakDictionary.cs

Loads a word list from a UTF-8 file.

net10.0

public static Task<LineBreakDictionary> LoadAsync(string path, LineBreakDictionaryOptions? options = null, CancellationToken cancellationToken = default)

net11.0

public static Task<LineBreakDictionary> LoadAsync(string path, LineBreakDictionaryOptions? options = null, CancellationToken cancellationToken = default)
Parameters
path string
The path of the file.
options LineBreakDictionaryOptions?
The options, or null for LineBreakDictionaryOptions.Default.
cancellationToken CancellationToken
The token to monitor for cancellation requests.
Returns

Task<LineBreakDictionary> The dictionary.

Exceptions
ArgumentNullException
path is null.
FormatException
The content is not valid UTF-8, or exceeds a limit of options.
IOException
The file cannot be read.
OperationCanceledException
The operation was canceled.

Parse overloads

Parse(ReadOnlySpan<byte>, LineBreakDictionaryOptions?)

LineBreakDictionary.cs

Creates a dictionary from the UTF-8 text of a word list.

public static LineBreakDictionary Parse(ReadOnlySpan<byte> utf8Words, LineBreakDictionaryOptions? options = null)
Parameters
utf8Words ReadOnlySpan<byte>
The word list, in UTF-8: one word per line.
options LineBreakDictionaryOptions?
The options, or null for LineBreakDictionaryOptions.Default.
Returns

LineBreakDictionary The dictionary.

Exceptions
FormatException
The content is not valid UTF-8, or exceeds a limit of options.

Parse(ReadOnlySpan<char>, LineBreakDictionaryOptions?)

LineBreakDictionary.cs

Creates a dictionary from the text of a word list.

public static LineBreakDictionary Parse(ReadOnlySpan<char> words, LineBreakDictionaryOptions? options = null)
Parameters
words ReadOnlySpan<char>
The word list: one word per line.
options LineBreakDictionaryOptions?
The options, or null for LineBreakDictionaryOptions.Default.
Returns

LineBreakDictionary The dictionary.

Exceptions
FormatException
The content has an unpaired surrogate, or exceeds a limit of options.