Meziantou.Framework

Extracting text

edit

PdfPage.ExtractText returns the text of a page, with a line break between the lines:

var document = PdfDocument.Open(file);
var pageCount = document.Pages.Count;
var text = document.Pages[0].ExtractText();

Search a document

var document = PdfDocument.Open(path);

// The numbers of the pages whose text contains a word
var pages = new List<int>();
for (var i = 0; i < document.Pages.Count; i++)
{
    var text = document.Pages[i].ExtractText();
    if (text.Contains("revenue", StringComparison.OrdinalIgnoreCase))
    {
        pages.Add(i + 1);
    }
}

Find where the text is

PdfPage.ExtractTextLayout returns the same text with its position: the lines, their words and their glyphs, each with its bounding box. Use it to highlight the results of a search, to redact a word or to group the cells of a table:

var document = PdfDocument.Open(file);
var layout = document.Pages[0].ExtractTextLayout();

// The boxes of a word, in the coordinates of the page (points, from its bottom left corner)
var boxes = new List<PdfRectangle>();
foreach (var line in layout.Lines)
{
    foreach (var word in line.Words)
    {
        if (word.Text.Equals("grew", StringComparison.OrdinalIgnoreCase))
        {
            boxes.Add(word.Bounds);
        }
    }
}
  • The coordinates are in the default user space of the page: points, from the origin of its media box, with the y axis upward, before the rotation of the page (PdfPage.Rotation).
  • A glyph has its origin on the baseline, the point where the next glyph goes, its font size in user space, the name of its font and the angle of the writing direction (0 for horizontal text, -90 for text written vertically).
  • The box of a glyph goes from its origin to the next one, and from the descent to the ascent of its font. Text that is rotated or written vertically gets the smallest rectangle that contains its box.
  • The box of a line or a word contains the boxes of its glyphs. The words are the glyphs between two spaces.

Read the columns in order

By default, the lines are in the order of the content. A page with several columns whose lines are painted from left to right across the gutter gives the lines of the columns mixed together. With PdfTextReadingOrder.Layout, the extraction finds the blocks of the page from its blank spaces and reads the columns one after the other. Each block is a paragraph or a part of a column, a convenient unit to index or to give to a language model:

var document = PdfDocument.Open(file);
var options = new PdfTextExtractionOptions { ReadingOrder = PdfTextReadingOrder.Layout };
var layout = document.Pages[0].ExtractTextLayout(options);

// The blocks of the page (columns, paragraphs) in reading order, for example to index them separately
var chunks = new List<string>();
foreach (var block in layout.Blocks)
{
    chunks.Add(block.Text);
}

ExtractText takes the same options and returns the lines in the same order.

  • The page is cut recursively (an XY-cut): at the vertical blank spaces that cross it from top to bottom, at least as wide as the font size of the body of the page (the gutters, read from left to right), or else at the horizontal blank spaces that cross it from left to right, at least half the font size high (read from top to bottom). A title or a footer that crosses the gutter is read before or after the columns.
  • A line of the content is split at its gaps wider than its font size, then joined again when no gutter separates the pieces: the spaces of a justified line do not split it.
  • The positions are those of the page as it is displayed, after its rotation.
  • The lines of a block are sorted by their baselines, from top to bottom; a block that has text written vertically or rotated keeps the order of the content.
  • The columns are read from left to right, also for the right-to-left scripts. The cells of a table are blocks, read column by column when blank spaces separate the columns.

Search text written with ligatures

A font can show "fi" with one glyph whose character is the ligature fi (U+FB01): the extracted word does not contain the letters "fi". PdfTextExtractionOptions.ExpandLigatures replaces the Latin ligatures (U+FB00 to U+FB06) by their letters:

var document = PdfDocument.Open(file);
var options = new PdfTextExtractionOptions { ExpandLigatures = true };
var text = document.Pages[0].ExtractText(options);

By default, the text is exactly what the fonts give.

How the text is extracted

  • The glyphs are read in the order of the content, and turned back into characters with the encoding of their font, its ToUnicode map or the names of its glyphs. The documents the library writes have what the extraction needs, embedded fonts included.
  • The Chinese, Japanese and Korean fonts that use a predefined encoding (such as 90ms-RKSJ-H, GBK-EUC-H, UniJIS-UCS2-H or KSCms-UHC-H) give their text without a ToUnicode map: the codes are split with the code space of the encoding and decoded with its character set (Shift-JIS, EUC, GBK, GB 18030, Big Five, UHC or Unicode).
  • The text written vertically follows the vertical metrics of its font (/W2 and /DW2).
  • The glyphs are joined into lines by their positions. A gap between two glyphs of a line becomes a space, and a new line becomes a line break, even when the document draws each word separately.
  • A replacement text (/ActualText) replaces the glyphs it covers, such as a ligature or a right-to-left run.
  • The text of the forms the page paints is included.

Limits

  • By default, the order is the order of the content: a document that draws the text in another order than it is read gives it in that order. The reading order of the layout (Read the columns in order) follows the blank spaces of the page: columns that no blank space separates, such as text that flows around an image, are not detected, and the columns of vertical writing are not ordered.
  • A text drawn as paths or as an image (a scanned page) has no characters to extract.
  • A font without encoding information, such as some embedded fonts of old documents, can give wrong characters.
  • The predefined encodings of the CNS 11643 and GB/T 12345 character sets (CNS-EUC-H, GBT-EUC-H, ...) and the characters of the Hong Kong extensions of Big Five have no text without a ToUnicode map. The widths of the glyphs of a font with a predefined encoding other than Identity-H and Identity-V are its default width: the positions of its glyphs, and the spaces found from them, can be off.
  • The boxes of the glyphs use the ascent and descent of the font, not the outline of each glyph.
  • A page whose content runs too many operators, such as forms that paint forms many times, throws PdfResourceLimitException instead of running for a long time.