Meziantou.FrameworkRobotsFile Class
Represents a parsed robots.txt file.
public sealed class RobotsFileRemarks
The parser is lenient: unknown directives and malformed lines are skipped rather than throwing, matching the behaviour of most web crawlers. Each skipped line is recorded in RobotsFile.ParseErrors so callers can audit file quality.
Supported directives: User-agent, Allow, Disallow, Crawl-delay, and Sitemap.
Path patterns in Allow and Disallow directives support Google's wildcard extensions: * matches any sequence of characters, and a trailing $ anchors the pattern to the end of the path.
Example
var robots = RobotsFile.Parse("""
User-agent: *
Disallow: /private/
Allow: /public/
Sitemap: https://example.com/sitemap.xml
""");
Assert.True(robots.IsAllowed("Googlebot", "/public/page"));
Assert.False(robots.IsAllowed("Googlebot", "/private/page"));
Assert.Equal(["https://example.com/sitemap.xml"], robots.Sitemaps);
// Invalid lines are reported instead of throwing
Assert.Empty(robots.ParseErrors);Properties
| Name | Description |
|---|---|
| Groups | Gets the groups of directives defined in the file, in the order they appear. |
| ParseErrors | Gets the non-fatal parse errors encountered while reading the file. |
| Sitemaps | Gets the sitemap URLs declared by Sitemap: directives, in the order they appear. |
Groups
RobotsFile.csGets the groups of directives defined in the file, in the order they appear.
public IReadOnlyList<RobotsGroup> Groups { get; }Property value
ParseErrors
RobotsFile.csGets the non-fatal parse errors encountered while reading the file.
public IReadOnlyList<RobotsParseError> ParseErrors { get; }Property value
IReadOnlyList<RobotsParseError>
Remarks
The list is empty when the file is fully valid. Each entry records the 1-based RobotsParseError.LineNumber, the raw RobotsParseError.Line text, and the RobotsParseError.Kind of problem.
Sitemaps
RobotsFile.csGets the sitemap URLs declared by Sitemap: directives, in the order they appear.
public IReadOnlyList<string> Sitemaps { get; }Property value
Methods
| Name | Description |
|---|---|
| GetCrawlDelay(string) | Returns the crawl delay for the given userAgent, or null when none is specified. |
| GetGroup(string) | Returns the RobotsGroup that best matches the given userAgent. |
| IsAllowed(string, string) | Determines whether the given userAgent is allowed to access path. |
| IsAllowed(string, Uri) | Determines whether the given userAgent is allowed to access the URL represented by uri. |
| Parse(ReadOnlySpan<char>) | Parses a robots.txt file from a ReadOnlySpan<T> of characters. |
| Parse(string) | Parses a robots.txt file from a string. |
| ParseAsync(Stream, Encoding?, CancellationToken) | Asynchronously parses a robots.txt file from a Stream. |
| ParseAsync(TextReader, CancellationToken) | Asynchronously parses a robots.txt file from a TextReader. |
GetCrawlDelay(string)
RobotsFile.csGetGroup(string)
RobotsFile.csReturns the RobotsGroup that best matches the given userAgent.
public RobotsGroup? GetGroup(string userAgent)Parameters
userAgentstring
Returns
Remarks
An exact (case-insensitive) match takes precedence over a catch-all (*) group. Returns null when no group matches.
When several groups declare the selected user-agent token, their rules are merged into a single group, as required by RFC 9309 section 2.2.1. The merged group is not one of the instances in RobotsFile.Groups: it lists the union of the declared tokens, the rules of every contributing group in file order, and the first Crawl-delay that was specified. When exactly one group matches, that instance is returned as-is.
IsAllowed overloads
IsAllowed(string, string)
RobotsFile.csDetermines whether the given userAgent is allowed to access path.
IsAllowed(string, Uri)
RobotsFile.csDetermines whether the given userAgent is allowed to access the URL represented by uri.
Parse overloads
Parse(ReadOnlySpan<char>)
RobotsFile.csParses a robots.txt file from a ReadOnlySpan<T> of characters.
public static RobotsFile Parse(ReadOnlySpan<char> content)Parse(string)
RobotsFile.csParses a robots.txt file from a string.
public static RobotsFile Parse(string content)ParseAsync overloads
ParseAsync(Stream, Encoding?, CancellationToken)
RobotsFile.csAsynchronously parses a robots.txt file from a Stream.
net10.0
public static Task<RobotsFile> ParseAsync(Stream stream, Encoding? encoding = null, CancellationToken cancellationToken = default)net11.0
public static Task<RobotsFile> ParseAsync(Stream stream, Encoding? encoding = null, CancellationToken cancellationToken = default)Parameters
streamStream- The stream to read from.
encodingEncoding?- The text encoding to use. Defaults to
Encoding.UTF8whennull. cancellationTokenCancellationToken- A token to cancel the operation.
Returns
ParseAsync(TextReader, CancellationToken)
RobotsFile.csAsynchronously parses a robots.txt file from a TextReader.
net10.0
public static Task<RobotsFile> ParseAsync(TextReader reader, CancellationToken cancellationToken = default)net11.0
public static Task<RobotsFile> ParseAsync(TextReader reader, CancellationToken cancellationToken = default)Parameters
readerTextReader- The text reader to read from.
cancellationTokenCancellationToken- A token to cancel the operation.