Meziantou.Framework

Introduction

edit

Meziantou.Framework.RobotsTxt parses robots.txt files as described by RFC 9309, including * wildcards and $ end anchors in paths. It answers the questions a crawler asks before fetching a page: may this user agent fetch this path, how long should it wait between requests, and where are the sitemaps.

What you can do with it

  • Check a path for a user agent with IsAllowed, from a path string or a Uri. See Getting started.
  • Find the group of directives that applies to a crawler: an exact user-agent match wins over the * group, and groups that name the same agent are merged. See User-agent groups.
  • Match paths with Allow and Disallow rules, * wildcards and $ anchors: the longest matching rule wins. See Path rules.
  • Read the crawl delay and the sitemaps declared by the file.
  • Audit a file: invalid lines never throw; they are reported with their line number. See Parse errors.

Supported directives

DirectiveScopeMeaning
User-agentGroupStarts a group, or adds an agent to the group being read.
AllowGroupThe paths that match may be fetched.
DisallowGroupThe paths that match may not be fetched. An empty value allows everything.
Crawl-delayGroupThe number of seconds to wait between requests, decimals allowed.
SitemapFileThe URL of a sitemap, wherever it appears in the file.

Directive names are case-insensitive, and # starts a comment that runs to the end of the line. Any other directive is reported as a parse error and skipped.

Namespace

Every type is in the Meziantou.Framework.RobotsTxt namespace: RobotsFile, RobotsGroup, RobotsRule, RobotsRuleKind, RobotsParseError and RobotsParseErrorKind.

Next steps