Network requests

edit

Nothing is fetched unless the browsing context has a DocumentLoader. With a loader, the browsing context loads the documents of the navigations and of the iframe elements, the style sheets (<link rel=stylesheet>, @import), the images, the fonts, and, with a script engine, the scripts and the requests of fetch(), XMLHttpRequest, WebSocket and EventSource.

Load pages over HTTP

DocumentLoader.Create() returns a loader that sends HTTP requests:

using var context = new BrowsingContext(new BrowsingContextOptions
{
    DocumentLoader = DocumentLoader.Create(),  // HTTP GET requests with a shared HttpClient (or pass your own)
    CookieJar = new CookieJar(),               // optional: the cookies (RFC 6265bis, document.cookie, cookieStore)
});

context.Navigate("https://example.com/");
await context.EventLoop.RunUntilIdleAsync(); // runs the tasks until the document and its resources are loaded

var document = context.Document;

Without an argument, the loader uses a shared HttpClient that decompresses the responses. Pass your own HttpClient to DocumentLoader.Create(httpClient) to set a proxy, a timeout or a handler; it should not store cookies itself (UseCookies = false) when the browsing context has a CookieJar.
The redirects are followed (up to 20), the Referer header is sent as a browser sends it, and the cookies of the CookieJar are sent and stored.
Only the http: and https: URLs are requested. The data: and blob: URLs are handled by the library.

Protect your network from untrusted pages

DocumentLoader.Create() requests any URL that a page navigates to or references. With an untrusted page, the requests can reach the services of your private network (server-side request forgery). Use a loader that filters the requests:

internal sealed class SameHostLoader(string host) : DocumentLoader
{
    private static readonly DocumentLoader Http = DocumentLoader.Create();

    public override ValueTask<DocumentResource?> LoadAsync(DocumentRequest request, CancellationToken cancellationToken)
    {
        if (new Uri(request.Url).Host != host)
            return ValueTask.FromResult<DocumentResource?>(null); // network error

        return Http.LoadAsync(request, cancellationToken);
    }
}

A null result is a network error, as is an exception thrown by the loader. DocumentRequest describes the request: its Url, Method, Headers and Body, its Destination (a document, a style sheet, an image, a script...), the Origin of the document that makes it (null when your application navigates), its Referrer, and the BrowsingContext it is for.

The URL keeps the fragment of the reference: <use href="icons.svg#home"> requests https://example.com/icons.svg#home, and <iframe src="frame.html#intro"> requests https://example.com/frame.html#intro. An HTTP request does not send it, so remove it when you look up a resource by URL.

Serve pages from memory or from the disk

A loader can return anything: the files of a folder, the pages of a cache, or test pages. Return a DocumentResource with the content, the status code and the headers:

/// <summary>Serves the documents and the resources from memory, by absolute URL without fragment.</summary>
private sealed class InMemoryLoader(IReadOnlyDictionary<string, (string Content, string ContentType)> files) : DocumentLoader
{
    public override ValueTask<DocumentResource?> LoadAsync(DocumentRequest request, CancellationToken cancellationToken)
    {
        // The URL keeps its fragment (e.g. "frame.html#intro", "icons.svg#home"), which is not sent over HTTP
        var url = request.Url.Split('#', 2)[0];
        if (!files.TryGetValue(url, out var file))
            return ValueTask.FromResult<DocumentResource?>(new DocumentResource("Not found") { StatusCode = 404 });

        return ValueTask.FromResult<DocumentResource?>(new DocumentResource(file.Content)
        {
            Headers = [new("Content-Type", file.ContentType)],
        });
    }
}

var loader = new InMemoryLoader(new Dictionary<string, (string Content, string ContentType)>(StringComparer.Ordinal)
{
    ["https://example.com/"] = ("<link rel=stylesheet href=site.css><h1>Home</h1><iframe src=/frame.html#intro></iframe>", "text/html"),
    ["https://example.com/site.css"] = ("h1 { color: green }", "text/css"),
    ["https://example.com/frame.html"] = ("<p>Inside the frame</p>", "text/html"),
});

using var context = new BrowsingContext(new BrowsingContextOptions { DocumentLoader = loader });
context.Navigate("https://example.com/");
await context.EventLoop.RunUntilIdleAsync(); // the document, its style sheet, the iframe and its document

var document = context.Document;
Assert.Equal("rgb(0, 128, 0)", context.Window.GetComputedStyle(document.QuerySelector("h1")!).Color);
var frame = (HtmlIFrameElement)document.QuerySelector("iframe")!;
Assert.Equal("Inside the frame", frame.ContentDocument!.Body!.TextContent);

Responses

// A resource created from its decoded content
var page = new DocumentResource("<p>Hello</p>");
Assert.Equal("<p>Hello</p>", page.Content);

// A resource created from its bytes, decoded using the charset of the Content-Type header
var bytes = "console.log('Hello');"u8.ToArray();
var script = new DocumentResource(bytes) { Headers = [new("Content-Type", "text/javascript; charset=utf-8")] };
Assert.Null(script.Content);
Assert.Equal(bytes, script.ContentBytes?.ToArray());

The Content-Type header gives the type of the resource (text/html when there is none) and its encoding. The bytes are decoded as a browser decodes them: byte order mark, charset, then, for HTML, <meta charset> and the detection of the encoding from the content.
text/html is parsed as HTML, the XML types (application/xhtml+xml, image/svg+xml, application/xml) as XML, and text/plain and the other text types as a text document.
As in a browser, the type of a page is sniffed from its bytes when the Content-Type is not a valid MIME type, and a text/plain page whose first bytes are binary data is a binary resource (the default type of Apache is wrong for these files), unless the response has X-Content-Type-Options: nosniff. A binary resource (application/octet-stream) does not create a document: it is a download, as a response whose Content-Disposition is attachment, and the page stays.
A document is displayed whatever its status (a 404 page is a page). A script or a style sheet whose status is not in the 200-299 range is a network error.
The library reads the headers that change the document: Content-Security-Policy, Referrer-Policy, Refresh, Link (preloads), X-Frame-Options, Content-Language, Document-Policy... The other headers are ignored.
Document Policy is applied as in Chromium: Document-Policy: sync-xhr=?0 makes the synchronous XMLHttpRequest requests fail, and Document-Policy-Report-Only reports them (document-policy-violation reports). A page requires a policy from the pages of its frames with the Require-Document-Policy header and the policy attribute of its iframe elements: the requests of the frames send it in the Sec-Required-Document-Policy header, and a frame whose Document-Policy header is less strict is blocked. The library supports the sync-xhr and force-load-at-top features. Disable it with DocumentPolicyEnabled.
The Content-DPR header of an image gives its density: its natural size is divided by it (Content-DPR: 2 displays a 100x100 image at 50x50 CSS pixels), in the img elements (where it replaces the density of the srcset candidate), the CSS images, the SVG image elements, the video posters and the image buttons.
RemainingContentBytes streams a body that arrives over time: the HTML of a navigation is parsed as its chunks arrive.

The requests are canceled when the navigation is aborted (another navigation, the removal of the iframe) and when the browsing context is disposed: pass the CancellationToken to your I/O.