Meziantou.Framework

Unicode Class

Namespace
Meziantou.Framework
Assembly
Meziantou.Framework.Unicode.dll
Package
Meziantou.Framework.Unicode 2.0.15
Source
Unicode.cs
Available in package assets
net10.0net11.0

Provides Unicode helper methods.

public static class Unicode

Properties

NameDescription
AllCharacters

AllCharacters

Unicode.cs
public static IReadOnlyCollection<UnicodeCharacterInfo> AllCharacters { get; }

Property value

IReadOnlyCollection<UnicodeCharacterInfo>

No documentation is available.

Methods

NameDescription
AreConfusable(string, string)Determines whether two strings are visually confusable, as defined by Unicode Technical Standard #39.
AreConfusable(string, string, UnicodeBidiDirection)Determines whether two strings are visually confusable when displayed in a paragraph of a given direction, as defined by Unicode Technical Standard #39.
EnumerateGraphemeClusters(ReadOnlySpan<char>)Enumerates the extended grapheme clusters of a text.
EnumerateLineBreaks(ReadOnlySpan<char>, LineBreakDictionary?, UnicodeLineBreakStrictness, UnicodeWordBreakBehavior)Enumerates the positions where a line can or must break, with a dictionary for the text that is written without spaces between words.
EnumerateLineBreaks(ReadOnlySpan<char>, UnicodeLineBreakStrictness, UnicodeWordBreakBehavior)Enumerates the positions where a line can or must break.
EnumerateScriptRuns(ReadOnlySpan<char>)Enumerates the runs of a text that are written in one script.
EnumerateSentenceBoundaries(ReadOnlySpan<char>)Enumerates the sentences of a text.
EnumerateWordBoundaries(ReadOnlySpan<char>)Enumerates the segments that the word boundaries of a text delimit.
GetCharacterInfo(char)Gets information about a Unicode character.
GetCharacterInfo(Rune)Gets information about a Unicode character.
GetConfusableSkeleton(string)Computes the skeleton of a string, as defined by Unicode Technical Standard #39.
GetConfusableSkeleton(string, UnicodeBidiDirection)Computes the bidirectional skeleton of a string for a paragraph direction, as defined by Unicode Technical Standard #39.
GetGraphemeClusterLength(ReadOnlySpan<char>)Gets the length of the first extended grapheme cluster of a text.
IsConfusableCharacter(Rune)Determines whether a Unicode character has a confusable replacement.
IsMixedScript(string)Determines whether a string mixes scripts in a way that Unicode Technical Standard #39 considers suspicious.
IsSingleScript(string)Determines whether a string is written in a single script, as defined by Unicode Technical Standard #39.
ReplaceConfusablesCharacters(char)Replaces a confusable Unicode character using the Unicode confusables table.
ReplaceConfusablesCharacters(string)Replaces confusable Unicode characters using the Unicode confusables table.
ReplaceConfusablesCharacters(Rune)Replaces a confusable Unicode character using the Unicode confusables table.
TryGetCharacterInfo(char, UnicodeCharacterInfo)Tries to get information about a Unicode character.
TryGetCharacterInfo(Rune, UnicodeCharacterInfo)Tries to get information about a Unicode character.

AreConfusable overloads

AreConfusable(string, string)

Unicode.cs

Determines whether two strings are visually confusable, as defined by Unicode Technical Standard #39.

public static bool AreConfusable(string a, string b)
Parameters
a string
The first string. Must be well-formed UTF-16.
b string
The second string. Must be well-formed UTF-16.
Returns

bool true when both strings have the same skeleton; otherwise false.

Exceptions
ArgumentException
The strings are different and one of them contains an unpaired surrogate.
Remarks

The strings are compared as displayed in a left-to-right paragraph (Unicode.GetConfusableSkeleton). This compares whole strings character by character after reduction. It does not consider script mixing, so two strings drawn from different scripts that are not individually confusable are still reported as not confusable.

See also

AreConfusable(string, string, UnicodeBidiDirection)

Unicode.cs

Determines whether two strings are visually confusable when displayed in a paragraph of a given direction, as defined by Unicode Technical Standard #39.

public static bool AreConfusable(string a, string b, UnicodeBidiDirection direction)
Parameters
a string
The first string. Must be well-formed UTF-16.
b string
The second string. Must be well-formed UTF-16.
direction UnicodeBidiDirection
The direction of the paragraph: see Unicode.GetConfusableSkeleton.
Returns

bool true when both strings have the same skeleton for direction; otherwise false.

Exceptions
ArgumentException
The strings are different and one of them contains an unpaired surrogate.
ArgumentOutOfRangeException
direction is not a defined value.
See also

EnumerateGraphemeClusters(ReadOnlySpan<char>)

Unicode.Segmentation.cs

Enumerates the extended grapheme clusters of a text.

public static UnicodeGraphemeClusterEnumerator EnumerateGraphemeClusters(ReadOnlySpan<char> text)

Parameters

text ReadOnlySpan<char>
The text.

Returns

UnicodeGraphemeClusterEnumerator An enumerator of the range of each cluster, in order; the ranges cover the whole text.

Remarks

See Unicode.GetGraphemeClusterLength for the definition of a cluster. The enumeration does not allocate.

See also

EnumerateLineBreaks overloads

EnumerateLineBreaks(ReadOnlySpan<char>, LineBreakDictionary?, UnicodeLineBreakStrictness, UnicodeWordBreakBehavior)

Unicode.Segmentation.cs

Enumerates the positions where a line can or must break, with a dictionary for the text that is written without spaces between words.

public static UnicodeLineBreakEnumerator EnumerateLineBreaks(ReadOnlySpan<char> text, LineBreakDictionary? dictionary, UnicodeLineBreakStrictness strictness = UnicodeLineBreakStrictness.Strict, UnicodeWordBreakBehavior wordBreak = UnicodeWordBreakBehavior.Normal)
Parameters
text ReadOnlySpan<char>
The text.
dictionary LineBreakDictionary?
The words of the scripts that are written without spaces between words, or null to only apply the rules of the Unicode Line Breaking Algorithm.
strictness UnicodeLineBreakStrictness
How strict the rules are.
wordBreak UnicodeWordBreakBehavior
Whether a line can break inside a word.
Returns

UnicodeLineBreakEnumerator An enumerator of the breaks, in order. The last one is at the end of the text and is mandatory; an empty text has no break.

Exceptions
ArgumentOutOfRangeException
strictness or wordBreak is not a defined value.
Remarks

The breaks are those of Unicode.EnumerateLineBreaks, and the breaks that dictionary finds in the text that the Unicode Line Breaking Algorithm leaves to a dictionary: the characters whose line breaking class is Complex_Context (SA), those of Thai, Lao, Khmer, Myanmar and a few other scripts. A break of the dictionary is not mandatory.

  • A run is a maximal sequence of consecutive Complex_Context characters that starts with a letter: the combining marks before the first letter belong to the character before them.
  • Each run is segmented into words as the remarks of LineBreakDictionary specify, and a line can break before each word of the dictionary, except at the start of the run. The breaks of a run are those of LineBreakDictionary.FindBreaks for the run.
  • The characters of a run that are in no word stay with what precedes them. A break is never inside a grapheme cluster, nor before a combining mark.
  • The breaks at the start and at the end of a run are those of the Unicode Line Breaking Algorithm, where a Complex_Context character is a letter or a combining mark (rule LB1). The other characters, such as spaces, digits, punctuation and zero width spaces, are not looked up in the dictionary.
  • UnicodeWordBreakBehavior.KeepAll keeps the breaks of the dictionary, as the word-break: keep-all of CSS does: they are between words, not inside a word.
  • With UnicodeWordBreakBehavior.BreakAll or UnicodeLineBreakStrictness.Anywhere a line already breaks between any two letters or grapheme clusters: the dictionary is not used and the breaks are those of the other overload.

The enumeration does not allocate, and its cost is proportional to the length of the text.

See also

EnumerateLineBreaks(ReadOnlySpan<char>, UnicodeLineBreakStrictness, UnicodeWordBreakBehavior)

Unicode.Segmentation.cs

Enumerates the positions where a line can or must break.

public static UnicodeLineBreakEnumerator EnumerateLineBreaks(ReadOnlySpan<char> text, UnicodeLineBreakStrictness strictness = UnicodeLineBreakStrictness.Strict, UnicodeWordBreakBehavior wordBreak = UnicodeWordBreakBehavior.Normal)
Parameters
text ReadOnlySpan<char>
The text.
strictness UnicodeLineBreakStrictness
How strict the rules are.
wordBreak UnicodeWordBreakBehavior
Whether a line can break inside a word.
Returns

UnicodeLineBreakEnumerator An enumerator of the breaks, in order. The last one is at the end of the text and is mandatory; an empty text has no break.

Exceptions
ArgumentOutOfRangeException
strictness or wordBreak is not a defined value.
Remarks

The breaks are those of the Unicode Line Breaking Algorithm (Unicode Standard Annex #14) for UnicodeProperties.UnicodeVersion, whatever the version of the runtime. With the default arguments the rules are not tailored: the classes that need information from outside the algorithm are resolved as the annex specifies (rule LB1), so Thai, Lao, Khmer and the other scripts that need a dictionary only break at spaces, and an object replacement character can break on both sides. The overload that takes a LineBreakDictionary also breaks such text between the words of a word list.

A break position is the index of the first character of the next line. A soft hyphen (U+00AD) gives an opportunity after it like any other: the caller knows it by looking at the character before the break.

The enumeration does not allocate.

See also

EnumerateScriptRuns(ReadOnlySpan<char>)

Unicode.Segmentation.cs

Enumerates the runs of a text that are written in one script.

public static UnicodeScriptRunEnumerator EnumerateScriptRuns(ReadOnlySpan<char> text)

Parameters

text ReadOnlySpan<char>
The text.

Returns

UnicodeScriptRunEnumerator An enumerator of the runs, in order; the runs cover the whole text.

Remarks

A run is what a text layout engine shapes with the rules and the font of one script. The Unicode Standard defines the scripts of a character (Unicode Standard Annex #24) and recommends how to resolve the characters that are shared between scripts, without specifying an algorithm. This method applies those recommendations as follows:

  • A run never ends inside an extended grapheme cluster: a combining mark has the script of its base.
  • A character whose script is UnicodeScript.Common or UnicodeScript.Inherited, such as a space, a digit or a punctuation mark, belongs to the run before it, or to the run after it at the start of the text.
  • The Script_Extensions of a character are honored: a character that is only used with some scripts does not continue a run of another script.
  • A closing bracket has the script of the run of its opening bracket, so that both brackets of a pair are in the same script.

The enumeration does not allocate.

See also

EnumerateSentenceBoundaries(ReadOnlySpan<char>)

Unicode.Segmentation.cs

Enumerates the sentences of a text.

public static UnicodeSentenceEnumerator EnumerateSentenceBoundaries(ReadOnlySpan<char> text)

Parameters

text ReadOnlySpan<char>
The text.

Returns

UnicodeSentenceEnumerator An enumerator of the range of each sentence, in order; the ranges cover the whole text.

Remarks

The boundaries are the default sentence boundaries of Unicode Standard Annex #29 for UnicodeProperties.UnicodeVersion. A sentence ends after a paragraph separator, or after its terminal punctuation (a full stop, a question mark, an exclamation mark...) with the closing punctuation, the spaces and the paragraph separator that follow it. A full stop does not end a sentence when it is followed by a digit, by an uppercase letter without a space between them (U.S.A.), or by a lowercase word (etc. and). No list of abbreviations is used: Mr. Smith is two sentences.

The enumeration does not allocate.

See also

EnumerateWordBoundaries(ReadOnlySpan<char>)

Unicode.Segmentation.cs

Enumerates the segments that the word boundaries of a text delimit.

public static UnicodeWordEnumerator EnumerateWordBoundaries(ReadOnlySpan<char> text)

Parameters

text ReadOnlySpan<char>
The text.

Returns

UnicodeWordEnumerator An enumerator of the range of each segment, in order; the ranges cover the whole text.

Remarks

The boundaries are the default word boundaries of Unicode Standard Annex #29 for UnicodeProperties.UnicodeVersion. A segment is a word, a number, a run of spaces, or a single punctuation mark or ideograph: there is a boundary on each side of a word, not only between words. No dictionary is used, so the scripts that are written without spaces are not split into words.

The enumeration does not allocate.

See also

GetCharacterInfo overloads

GetCharacterInfo(char)

Unicode.cs

Gets information about a Unicode character.

public static UnicodeCharacterInfo? GetCharacterInfo(char value)
Parameters
value char
The Unicode scalar value to inspect.
Returns

UnicodeCharacterInfo? The character information, or null when not found.

GetCharacterInfo(Rune)

Unicode.cs

Gets information about a Unicode character.

public static UnicodeCharacterInfo? GetCharacterInfo(Rune rune)
Parameters
rune Rune
The Unicode scalar value to inspect.
Returns

UnicodeCharacterInfo? The character information, or null when not found.

GetConfusableSkeleton overloads

GetConfusableSkeleton(string)

Unicode.cs

Computes the skeleton of a string, as defined by Unicode Technical Standard #39.

public static string GetConfusableSkeleton(string value)
Parameters
value string
The text to reduce. Must be well-formed UTF-16.
Returns

string The skeleton of value.

Exceptions
ArgumentException
value contains an unpaired surrogate.
Remarks

The skeleton is bidiSkeleton(LTR, X): the text as displayed in a left-to-right paragraph, reduced by internalSkeleton. See Unicode.GetConfusableSkeleton. It is a comparison key and nothing more: it is not displayable text, and it must not be shown to users or stored in place of the original. Two strings are confusable when their skeletons are equal, which is what Unicode.AreConfusable tests.

See also

GetConfusableSkeleton(string, UnicodeBidiDirection)

Unicode.cs

Computes the bidirectional skeleton of a string for a paragraph direction, as defined by Unicode Technical Standard #39.

public static string GetConfusableSkeleton(string value, UnicodeBidiDirection direction)
Parameters
value string
The text to reduce. Must be well-formed UTF-16.
direction UnicodeBidiDirection
The direction of the paragraph the text is displayed in: UnicodeBidiDirection.LeftToRight (LTR), UnicodeBidiDirection.RightToLeft (RTL), or UnicodeBidiDirection.Auto for the direction of the first strong character (FS).
Returns

string The skeleton of value, bidiSkeleton(direction, X).

Exceptions
ArgumentException
value contains an unpaired surrogate.
ArgumentOutOfRangeException
direction is not a defined value.
Remarks

The text is first put in display order with UnicodeBidi: rules up to L2 of the bidirectional algorithm, then L3 (the combining marks of a right-to-left character follow it) and L4 (the characters displayed with a mirrored glyph are replaced by their Bidi_Mirroring_Glyph). Each paragraph of the text is a single line. The result is then reduced by internalSkeleton: toNFD(toConfusable(removeDefaultIgnorable(toNFD(X)))).

The skeleton is a comparison key and nothing more: it is not displayable text, and it must not be shown to users or stored in place of the original.

See also

GetGraphemeClusterLength(ReadOnlySpan<char>)

Unicode.Segmentation.cs

Gets the length of the first extended grapheme cluster of a text.

public static int GetGraphemeClusterLength(ReadOnlySpan<char> text)

Parameters

text ReadOnlySpan<char>
The text.

Returns

int The number of UTF-16 code units of the cluster; 0 when text is empty.

Remarks

A grapheme cluster is what a reader perceives as one character: a base with its combining marks, a Hangul syllable, an emoji sequence, a flag. The boundaries are those of the extended grapheme clusters of Unicode Standard Annex #29 for UnicodeProperties.UnicodeVersion, whatever the version of the runtime.

An unpaired surrogate is a cluster on its own.

See also

IsConfusableCharacter(Rune)

Unicode.cs

Determines whether a Unicode character has a confusable replacement.

public static bool IsConfusableCharacter(Rune rune)

Parameters

rune Rune
The Unicode scalar value to inspect.

Returns

bool true when the character is confusable; otherwise false.

See also

IsMixedScript(string)

Unicode.Scripts.cs

Determines whether a string mixes scripts in a way that Unicode Technical Standard #39 considers suspicious.

public static bool IsMixedScript(string value)

Parameters

value string
The text to inspect.

Returns

bool true when the string is not single-script; otherwise false.

Remarks

Mixed-script text is the signal that distinguishes a homograph attack from two unrelated words: paypal spelled with Cyrillic look-alikes is mixed-script, while ordinary text in any one writing system is not.

See also

IsSingleScript(string)

Unicode.Scripts.cs

Determines whether a string is written in a single script, as defined by Unicode Technical Standard #39.

public static bool IsSingleScript(string value)

Parameters

value string
The text to inspect.

Returns

bool true when the resolved script set is not empty; otherwise false.

Remarks

A string is single-script when the augmented Script_Extensions sets of all its characters have a script in common. Characters whose script is Common or Inherited — digits, punctuation, combining marks — match any script and never make a string mixed on their own. Unassigned and private-use characters, whose script is Unknown, do not: they only match each other.

The empty string is single-script. Han, Hiragana, Katakana, Hangul, Bopomofo and Latin are augmented so that Japanese, Korean and Han text mixed with Latin, which legitimately combine scripts, is not reported as mixed.

See also

ReplaceConfusablesCharacters overloads

ReplaceConfusablesCharacters(char)

Unicode.cs

Replaces a confusable Unicode character using the Unicode confusables table.

public static string ReplaceConfusablesCharacters(char value)
Parameters
value char
The character to normalize.
Returns

string The replacement text for the character.

See also

ReplaceConfusablesCharacters(string)

Unicode.cs

Replaces confusable Unicode characters using the Unicode confusables table.

public static string ReplaceConfusablesCharacters(string str)
Parameters
str string
The text to normalize.
Returns

string The text with confusable characters replaced.

See also

ReplaceConfusablesCharacters(Rune)

Unicode.cs

Replaces a confusable Unicode character using the Unicode confusables table.

public static string ReplaceConfusablesCharacters(Rune rune)
Parameters
rune Rune
The character to normalize.
Returns

string The replacement text for the character.

See also

TryGetCharacterInfo overloads

TryGetCharacterInfo(char, UnicodeCharacterInfo)

Unicode.cs

Tries to get information about a Unicode character.

public static bool TryGetCharacterInfo(char value, out UnicodeCharacterInfo info)
Parameters
value char
The Unicode scalar value to inspect.
info UnicodeCharacterInfo
The character information when found.
Returns

bool true when the character exists in the Unicode data; otherwise false.

TryGetCharacterInfo(Rune, UnicodeCharacterInfo)

Unicode.cs

Tries to get information about a Unicode character.

public static bool TryGetCharacterInfo(Rune rune, out UnicodeCharacterInfo info)
Parameters
rune Rune
The Unicode scalar value to inspect.
info UnicodeCharacterInfo
The character information when found.
Returns

bool true when the character exists in the Unicode data; otherwise false.