Parse errors

The parser is lenient: it never throws on the content of a file. A line it cannot use is skipped, and recorded in ParseErrors so you can audit the file:

var robots = RobotsFile.Parse("""
    User-agent: *
    Host: example.com
    this line has no colon
    """);

var errors = robots.ParseErrors.Select(error => $"{error.LineNumber}: {error.Kind} => {error.Line}");
Assert.Equal(["2: UnknownDirective => Host: example.com", "3: MalformedLine => this line has no colon"], errors);

Each RobotsParseError has the 1-based LineNumber, the raw text of the Line (comment and whitespace included), and the Kind of problem:

ParseErrors is empty when every line is valid. Blank lines and comments are not errors.

Invalid crawl delays

A Crawl-delay value is a number of seconds, with an optional decimal part (1.5). A value that is not a non-negative number, or that is too large for a TimeSpan, is reported and ignored:

var robots = RobotsFile.Parse("""
    User-agent: *
    Crawl-delay: soon
    """);

Assert.Null(robots.GetCrawlDelay("MyCrawler"));
var error = Assert.Single(robots.ParseErrors);
Assert.Equal(RobotsParseErrorKind.InvalidCrawlDelay, error.Kind);
Assert.Equal(2, error.LineNumber);

The line still belongs to the group, so a User-agent line after it starts a new group, as after a valid delay.