TOML

Meziantou.Framework.Language.Toml parses TOML into a tree that keeps every character, so you can change a dependency in a Cargo.toml or a pyproject.toml and keep the comments, the quoting and the layout of the file. It reads TOML 1.1, and TOML 1.0 on request.

dotnet package add Meziantou.Framework.Language.Toml

Two TOML packages

This package and Meziantou.Framework.Toml both read TOML, for different purposes:

The two packages have their own types: a node of one cannot be used with the other.

Parse a document

var tree = TomlSyntaxTree.ParseText("""
    # The server
    [server]
    host = "localhost"
    ports = [ 8000, 8001 ]
    """);

Assert.Empty(tree.GetDiagnostics());

var server = tree.GetRoot().Tables.Single();
var ports = (TomlArraySyntax)server.Properties[1].Value;
Assert.Equal(8000L, ((TomlIntegerSyntax)ports.Elements[0]).Value);

The shape of a tree

A TomlDocumentSyntax holds its Entries in the order of the text:

• a TomlTableSyntax is a table header, [server] or [[products]]; IsArrayOfTables tells them apart;
• a TomlPropertySyntax is a key/value pair, host = "localhost", with its Key and its Value;
• a TomlSkippedTextSyntax is a line the parser could not read.

The entries are flat, as in the text: a table header is followed by its key/value pairs rather than being their parent. TomlTableSyntax.Properties gathers the pairs under a header, and TomlDocumentSyntax.RootProperties the pairs before the first header.

A key is a TomlKeySyntax, with the parts of a dotted key and the dots between them. Its Names are the names without their quotes and escapes, so site."google.com" has the names site and google.com.

Whitespace and comments up to the end of a line belong to the token that ends the line, and the rest to the next token, so a comment on its own line goes with the entry below it.

Read values by key

A table can be written with a header, with dotted keys, or as an inline table, and the three can be mixed. GetValue finds the value of a key whichever way it is written, and GetKeyValues() returns every key/value pair of the document with its full key:

var root = TomlSyntaxTree.ParseText("""
    [dependencies]
    serde = { version = "1.0" }

    [dependencies.tokio]
    version = "1.2"
    """).GetRoot();

var tokio = (TomlStringSyntax)root.GetValue("dependencies", "tokio", "version")!;
Assert.Equal("1.2", tokio.Value);

var keys = new List<string>();
foreach (var pair in root.GetKeyValues())
{
    keys.Add(string.Join('.', pair.Names));
}

Assert.Equal(["dependencies.serde", "dependencies.serde.version", "dependencies.tokio.version"], keys);

Each TomlKeyValue has the Names of its full key, its Value, the Property that defines it, the tokens that write each name in Parts, and the Table header it is under, which tells apart the tables of an array of tables. A pair whose value is an inline table comes before the pairs of that table. Values in arrays are not returned, as they have no key of their own. A list pattern reads the pairs of a table:

var document = TomlSyntaxTree.ParseText("""
    [dependencies]
    serde = "1.0"
    tokio = "1.2"
    """).GetRoot();

var dependencies = new List<string>();
foreach (var pair in document.GetKeyValues())
{
    if (pair.Names is ["dependencies", var name] && pair.Value is TomlStringSyntax version)
    {
        dependencies.Add($"{name} {version.Value}");
    }
}

Assert.Equal(["serde 1.0", "tokio 1.2"], dependencies);

Edit values

Replace a value with WithValue, which writes it the way TOML requires, and keeps the rest of the line:

var root = TomlSyntaxTree.ParseText("port = 8080 # the port\n").GetRoot();
var port = (TomlIntegerSyntax)root.RootProperties[0].Value;

var updated = root.ReplaceNode(port, port.WithValue(9090));
Assert.Equal("port = 9090 # the port\n", updated.ToFullString());

SyntaxFactory builds new nodes. Keys are quoted when they cannot be bare, strings are escaped, and floats always read back as floats:

var document = SyntaxFactory.TomlDocument(
    SyntaxFactory.TomlTable("server"),
    SyntaxFactory.TomlProperty("host name", SyntaxFactory.TomlString("localhost")),
    SyntaxFactory.TomlProperty(SyntaxFactory.Key("limits", "max"), SyntaxFactory.TomlFloat(1)));

Assert.Equal("""
    [server]
    "host name" = "localhost"
    limits.max = 1.0

    """, document.ToFullString());

• Every entry of a document ends its line. TomlDocument, AddEntries, and the edits that put an entry in front of another (ReplaceNode, InsertNodesAfter, InsertNodesBefore, a rewriter) add a line feed to an entry that does not end with one.
• AddElements and AddProperties lay out new items the way the array or the inline table already is: on its line, or one per line.
• SyntaxFactory.ParseValue reads a single value written as text, the easiest way to get a literal or a multi-line string written exactly as you want.
• The factory methods refuse what TOML cannot hold rather than write a document that does not read back: a lone surrogate in a string, whitespace other than spaces and tabs, a comment with a control character, or a key/value pair whose value starts with a line break.

TOML versions

TomlParseOptions.Version chooses the version of the specification. TOML 1.1, the default, adds line breaks and a trailing comma in inline tables, the \e and \xHH escape sequences, and times without seconds. Parsing as TOML 1.0 reports each of them as TOML0013, and reads them the same way otherwise:

// A time without seconds is a TOML 1.1 feature
var text = "start = 07:32\n";

var tree = TomlSyntaxTree.ParseText(text, new TomlParseOptions { Version = TomlVersion.V1_0 });
Assert.Equal("TOML0013", tree.GetDiagnostics().Single().Id);

TomlParseOptions.MaxDepth, 128 by default, limits how deeply arrays and inline tables nest. A deeper one is kept whole as skipped text and reported once. Nesting deeper than the stack can hold is reported the same way, whatever MaxDepth says.

Diagnostics

var messages = new List<string>();
foreach (var diagnostic in TomlSyntaxTree.ParseText("a = 1\na = 2\n").GetDiagnostics())
{
    var line = diagnostic.Location.GetLineSpan().Start;
    messages.Add($"{diagnostic.Id} ({line.Line},{line.Character}): {diagnostic.Message}");
}

Assert.Equal(["TOML0020 (1,0): The key 'a' is already defined."], messages);

The diagnostics from TOML0020 depend on the whole document rather than on one node: the tree works them out from its root when it is first asked. A tree made from an edited root, with WithRoot or Create, parses its text again, so a duplicate an edit removed is no longer reported, and one it added is.

Mistakes

TOML is line-oriented, and so is recovery: a key/value pair or a table header that goes wrong never takes the next line with it. A string without its closing quote ends with its line. An array, or an inline table in TOML 1.1, stops at its closing bracket, or at a line that can only start the next entry, so a missing ] is reported once rather than turning the rest of the document into one array.

What .NET cannot hold

A few valid TOML values do not fit the .NET types: the year 0 and a leap second (:60) are reported as TOML0006, an offset further from UTC than ±14:00 is read as the same instant in UTC, and a fraction of a second is truncated to seven digits. Line breaks in multi-line strings are read as line feeds. The Value of a node in error is a best guess: check ContainsDiagnostics before you rely on it.

Walk a tree

TomlSyntaxWalker, TomlSyntaxVisitor and TomlSyntaxRewriter have a method per node type, such as VisitTomlTable and VisitTomlInteger:

private sealed class DoubleIntegers : TomlSyntaxRewriter
{
    public override SyntaxNode? VisitTomlInteger(TomlIntegerSyntax node) => node.WithValue(node.Value * 2);
}