Meziantou.FrameworkRobotsGroup Class
Represents a group of directives in a robots.txt file that applies to one or more user-agents.
public sealed class RobotsGroupRemarks
A group consists of one or more User-agent lines followed by Allow, Disallow, and optional Crawl-delay directives.
Properties
| Name | Description |
|---|---|
| CrawlDelay | Gets the crawl delay specified by a Crawl-delay directive in this group, or null if none was specified. |
| Rules | Gets the ordered list of Allow and Disallow rules in this group. |
| UserAgents | Gets the user-agent tokens this group applies to (e.g. "Googlebot", "*"). |
CrawlDelay
RobotsGroup.csRules
RobotsGroup.csGets the ordered list of Allow and Disallow rules in this group.
public IReadOnlyList<RobotsRule> Rules { get; }Property value
UserAgents
RobotsGroup.csGets the user-agent tokens this group applies to (e.g. "Googlebot", "*").
public IReadOnlyList<string> UserAgents { get; }Property value
Methods
| Name | Description |
|---|---|
| IsAllowed(ReadOnlySpan<char>) | Determines whether the given path is allowed by the rules in this group. |
| IsAllowed(string) | Determines whether the given path is allowed by the rules in this group. |
| Matches(string) | Determines whether this group applies to the given userAgent. |
| ToString() | Returns a string that represents the current object. |
IsAllowed overloads
IsAllowed(ReadOnlySpan<char>)
RobotsGroup.csDetermines whether the given path is allowed by the rules in this group.
public bool IsAllowed(ReadOnlySpan<char> path)IsAllowed(string)
RobotsGroup.csDetermines whether the given path is allowed by the rules in this group.
Parameters
pathstring- The URL path (and optional query string) to check.
Returns
bool true if access is allowed; false if it is disallowed.
Remarks
The most specific (longest RobotsRule.Value) matching rule wins. When two matching rules have equal specificity, RobotsRuleKind.Allow takes precedence.
An empty Disallow path is treated as "allow all" and only matches when no other rule matches. If no rule matches the path, access is allowed.