Implementation Guidelines

Guidelines for implementing TON parsers, validators, and serializers.

Parser Architecture

Recommended implementation approach:

  1. Lexical Analysis: Tokenize the input stream
  2. Syntactic Analysis: Build an abstract syntax tree
  3. Semantic Analysis: Validate and process the structure
  4. Schema Validation: Apply schema rules if present

Core Components

Lexer/Tokenizer

  • Character stream to token stream conversion
  • Handle all token types (literals, operators, keywords)
  • Track line and column positions for error reporting
  • Process comments and whitespace

Parser

  • Recursive descent or parser combinator approach
  • Build document object model
  • Handle nested structures
  • Process multi-line strings with indentation

Validator

  • Schema loading and caching
  • Path-based property traversal
  • Type checking and constraint validation
  • Default value application

Serializer

  • Object model to TON format conversion
  • Formatting options (compact, pretty)
  • Proper escaping and quoting
  • Multi-line string formatting

Error Handling

public class TonParseException : Exception
{
    public int Line { get; }
    public int Column { get; }
    public string Context { get; }

    public TonParseException(string message, int line, int column)
        : base($"Parse error at line {line}, column {column}: {message}")
    {
        Line = line;
        Column = column;
    }
}

Performance Considerations

  • Streaming: Support streaming for large files
  • Caching: Cache compiled schemas
  • Memory: Use string interning for property names
  • Validation: Lazy validation where possible

API Design

// Parsing
var parser = new TonParser();
var document = parser.Parse(tonString);
var document = await parser.ParseFileAsync(filePath);

// Validation
var validator = new TonValidator(schemas);
var result = validator.Validate(document);

// Serialization
var serializer = new TonSerializer();
var tonString = serializer.Serialize(document, options);

// Accessing data
var value = document.RootObject["propertyName"];
var typed = value.To<MyClass>();

Testing Strategy

  1. Unit Tests: Individual components
  2. Integration Tests: End-to-end scenarios
  3. Edge Cases: Boundary conditions
  4. Performance Tests: Large file handling
  5. Compatibility Tests: Conformance levels

Common Pitfalls

  • Not preserving numeric precision
  • Incorrect multi-line string indentation
  • Case sensitivity in property names
  • Not validating GUID formats
  • Improper path traversal in schemas