diff --git a/website/content/docs/cli.md b/website/content/docs/cli.md deleted file mode 100644 index 4cf3932..0000000 --- a/website/content/docs/cli.md +++ /dev/null @@ -1,354 +0,0 @@ -+++ -title = "CLI Reference" -description = "Command-line tool documentation" -weight = 3 -+++ - -## Installation - -Clone and build from source: - -```bash -git clone https://tangled.org/@stavola.xyz/mlf -cd mlf -cargo build --release -``` - -The binary will be at `target/release/mlf`. - -### Install with cargo - -```bash -# Install with all code generators (default: TypeScript, Go, Rust + JSON) -cargo install --path mlf-cli --all-features - -# Install with only specific generators -cargo install --path mlf-cli --no-default-features --features typescript -cargo install --path mlf-cli --no-default-features --features typescript,go - -# Install with JSON generation only (minimal) -cargo install --path mlf-cli --no-default-features -``` - -### Available Features - -The CLI uses cargo features for optional code generators: - -- `typescript` - TypeScript type definitions and interfaces -- `go` - Go structs with JSON tags -- `rust` - Rust structs with serde derive macros -- **default** = `["typescript", "go", "rust"]` - All generators enabled - -The JSON lexicon generator is always available (built into `mlf-codegen`). - -## Commands - -### `mlf check` - -Validate MLF lexicon files for syntax and type errors. - -```bash -mlf check [INPUT]... -``` - -**Arguments:** -- `[INPUT]...` - MLF lexicon file(s) to validate (glob patterns supported) - -**Examples:** - -```bash -# Check a single file -mlf check thread.mlf - -# Check multiple files -mlf check thread.mlf profile.mlf reply.mlf - -# Check with glob patterns -mlf check "lexicons/**/*.mlf" -``` - -**Output:** -- ✓ Success message if valid -- Detailed error messages with source context if invalid - ---- - -### `mlf validate` - -Validate a JSON record against an MLF lexicon. - -```bash -mlf validate -``` - -**Arguments:** -- `` - MLF lexicon file -- `` - JSON record file to validate against the lexicon - -**Example:** - -```bash -mlf validate thread.mlf record.json -``` - -**Output:** -- ✓ Success if record is valid -- Detailed validation errors if invalid - ---- - -### `mlf generate lexicon` - -Generate ATProto JSON lexicons from MLF files. - -```bash -mlf generate lexicon --output [OPTIONS] -``` - -**Options:** -- `-i, --input ` - Input MLF files (glob patterns supported, can be specified multiple times) -- `-o, --output ` - Output directory (required) -- `--flat` - Use flat file structure (e.g., `com.example.thread.json`) - -**Examples:** - -```bash -# Generate with folder structure -mlf generate lexicon -i thread.mlf -o lexicons/ -# Creates: lexicons/com/example/thread.json - -# Generate with flat structure -mlf generate lexicon -i thread.mlf -o lexicons/ --flat -# Creates: lexicons/com.example.thread.json - -# Generate from multiple files -mlf generate lexicon -i thread.mlf -i reply.mlf -o lexicons/ - -# Generate from glob pattern -mlf generate lexicon -i "src/**/*.mlf" -o dist/lexicons/ -``` - ---- - -### `mlf generate code` - -Generate code in various programming languages from MLF files. - -```bash -mlf generate code --generator --output [OPTIONS] -``` - -**Options:** -- `-g, --generator ` - Generator to use: `json`, `typescript`, `go`, or `rust` (required) -- `-i, --input ` - Input MLF files (glob patterns supported, can be specified multiple times) -- `-o, --output ` - Output directory (required) -- `--flat` - Use flat file structure (e.g., `com.example.thread.ts`) - -**Available Generators:** - -| Generator | Output | Description | -|-----------|--------|-------------| -| `json` | `.json` | AT Protocol JSON lexicons (always available) | -| `typescript` | `.ts` | TypeScript interfaces with JSDoc comments | -| `go` | `.go` | Go structs with JSON tags and proper capitalization | -| `rust` | `.rs` | Rust structs with serde derive macros | - -**TypeScript Example:** - -```bash -mlf generate code -g typescript -i thread.mlf -o src/types/ -# Creates: src/types/com/example/thread.ts -``` - -Generated TypeScript: -```typescript -/** - * Generated from com.example.thread - * Do not edit manually - */ - -export interface Thread { - /** Thread title */ - title: string; - /** Creation timestamp */ - createdAt: string; - posts: Post[]; -} -``` - -**Go Example:** - -```bash -mlf generate code -g go -i thread.mlf -o pkg/models/ -# Creates: pkg/models/com/example/thread.go -``` - -Generated Go: -```go -// Generated from com.example.thread -// Do not edit manually - -package thread - -// Thread represents a discussion thread -type Thread struct { - // Thread title - Title string `json:"title"` - // Creation timestamp - CreatedAt string `json:"createdAt"` - Posts []Post `json:"posts"` -} -``` - -**Rust Example:** - -```bash -mlf generate code -g rust -i thread.mlf -o src/models/ -# Creates: src/models/com/example/thread.rs -``` - -Generated Rust: -```rust -// Generated from com.example.thread -// Do not edit manually - -use serde::{Deserialize, Serialize}; - -/// Thread represents a discussion thread -#[derive(Debug, Clone, Serialize, Deserialize)] -pub struct Thread { - /// Thread title - pub title: String, - /// Creation timestamp - #[serde(rename = "createdAt")] - pub created_at: String, - pub posts: Vec, -} -``` - -**Notes:** - -- TypeScript: Uses interfaces for records, type aliases for defs, optional fields use `?` -- Go: Uses structs with JSON tags, pointer types for optional fields, PascalCase for exports -- Rust: Uses structs with serde, `Option` for optional fields, snake_case with `#[serde(rename)]` -- All generators handle doc comments, constraints are preserved in generated code where applicable - ---- - -### `mlf generate mlf` - -Convert ATProto JSON lexicons back to MLF format. - -```bash -mlf generate mlf --output [OPTIONS] -``` - -**Options:** -- `-i, --input ` - Input JSON lexicon files (glob patterns supported, can be specified multiple times) -- `-o, --output ` - Output directory (required) - -**Examples:** - -```bash -# Convert a single JSON lexicon to MLF -mlf generate mlf -i com.example.thread.json -o ./lexicons/ -# Creates: lexicons/com/example/thread.mlf - -# Convert multiple JSON lexicons -mlf generate mlf -i lexicon1.json -i lexicon2.json -o ./mlf/ - -# Convert using glob pattern -mlf generate mlf -i "dist/lexicons/**/*.json" -o ./src/ -``` - -**Features:** - -- **Smart type conversion** - Automatically converts format strings (did, datetime, handle) to prelude types (Did, Datetime, Handle) -- **Proper formatting** - Generates clean, properly indented MLF with correct syntax -- **Reference conversion** - Converts `namespace#name` refs to `namespace.name` notation -- **Complete coverage** - Supports records, queries, procedures, subscriptions, tokens, and type definitions - -**Input format:** - -The JSON lexicon must be a valid ATProto lexicon with: -- `lexicon: 1` version field -- `id` field containing the namespace -- `defs` object with type definitions - -**Example:** - -Given a JSON lexicon: -```json -{ - "lexicon": 1, - "id": "com.example.thread", - "defs": { - "main": { - "type": "record", - "key": "tid", - "record": { - "type": "object", - "required": ["title", "createdAt"], - "properties": { - "title": { - "type": "string", - "maxLength": 200 - }, - "createdAt": { - "type": "string", - "format": "datetime" - } - } - } - } - } -} -``` - -Generates MLF: -```mlf -record main { - title!: string constrained { - maxLength: 200, - }, - createdAt!: Datetime, -}; -``` - -**Use cases:** - -- **Migration** - Convert existing JSON lexicons to MLF format -- **Interoperability** - Work with lexicons from other tools or repositories -- **Comparison** - Generate MLF from JSON to compare with hand-written MLF -- **Learning** - See how JSON lexicons translate to MLF syntax - ---- - -## Error Messages - -MLF provides rich error messages with: - -- Source code context -- Labeled spans showing exact error locations -- Helpful suggestions for fixing errors -- Error codes for categorization - -**Example error:** - -``` - × Undefined reference to 'ProfileView' - ╭─[profile.mlf:5:12] - 5 │ author: ProfileView, - · ^^^^^^^^^^^ 'ProfileView' is not defined - ╰──── - help: Make sure this type is defined in the same file or imported via 'use'. -``` - -## Environment Variables - -None currently used. - -## Exit Codes - -- `0` - Success -- `1` - Error (syntax, validation, or runtime error) diff --git a/website/content/docs/cli/01-installation.md b/website/content/docs/cli/01-installation.md new file mode 100644 index 0000000..8674445 --- /dev/null +++ b/website/content/docs/cli/01-installation.md @@ -0,0 +1,77 @@ ++++ +title = "Installation" +description = "How to install the MLF CLI" +weight = 1 ++++ + +## Building from Source + +Clone and build from source: + +```bash +git clone https://tangled.org/@stavola.xyz/mlf +cd mlf +cargo build --release +``` + +The binary will be at `target/release/mlf`. + +## Installing with Cargo + +### All Features (Recommended) + +Install with all code generators enabled: + +```bash +cargo install --path mlf-cli --all-features +``` + +This includes TypeScript, Go, Rust, and JSON lexicon generators. + +### Selective Installation + +Install only specific generators to reduce compilation time and binary size: + +```bash +# TypeScript only +cargo install --path mlf-cli --no-default-features --features typescript + +# TypeScript and Go +cargo install --path mlf-cli --no-default-features --features typescript,go + +# Minimal (JSON lexicon generation only) +cargo install --path mlf-cli --no-default-features +``` + +## Available Features + +The CLI uses cargo features for optional code generators: + +| Feature | Description | Output Format | +|---------|-------------|---------------| +| `typescript` | TypeScript type definitions and interfaces | `.ts` | +| `go` | Go structs with JSON tags | `.go` | +| `rust` | Rust structs with serde derive macros | `.rs` | +| (built-in) | JSON lexicon generator | `.json` | + +**Default features:** `["typescript", "go", "rust"]` - All generators enabled. + +The JSON lexicon generator is always available (built into `mlf-codegen` core). + +## Verify Installation + +After installation, verify the CLI is working: + +```bash +mlf --version +mlf --help +``` + +## Next Steps + +Once installed, you can: + +1. [Configure your project](02-configuration.md) with an `mlf.toml` file +2. [Check MLF files](03-check.md) for syntax and type errors +3. [Generate code](05-generate.md) in your preferred language +4. [Fetch remote lexicons](06-fetch.md) from ATProto repositories diff --git a/website/content/docs/cli/02-configuration.md b/website/content/docs/cli/02-configuration.md new file mode 100644 index 0000000..26cf8c8 --- /dev/null +++ b/website/content/docs/cli/02-configuration.md @@ -0,0 +1,216 @@ ++++ +title = "Configuration" +description = "Project configuration with mlf.toml" +weight = 2 ++++ + +MLF projects can be configured with an `mlf.toml` file that specifies source directories, output configurations, and dependencies. This allows you to run commands without repeatedly specifying arguments. + +## Basic Structure + +Create an `mlf.toml` file in your project root: + +```toml +[source] +directory = "./lexicons" + +[[output]] +type = "lexicon" +directory = "./dist/lexicons" + +[[output]] +type = "typescript" +directory = "./src/types" + +[dependencies] +dependencies = ["stream.place", "app.bsky"] +``` + +## Configuration Sections + +### Source Directory + +The `[source]` section specifies where your MLF files are located: + +```toml +[source] +directory = "./lexicons" +``` + +**Default:** `"./lexicons"` + +This directory is used by: +- `mlf check` (when run without arguments) +- `mlf generate` (when run without arguments) + +### Output Configurations + +The `[[output]]` section defines code generation targets. You can have multiple output blocks: + +```toml +[[output]] +type = "lexicon" +directory = "./dist/lexicons" + +[[output]] +type = "typescript" +directory = "./src/types" + +[[output]] +type = "go" +directory = "./pkg/lexicons" + +[[output]] +type = "rust" +directory = "./src/lexicons" +``` + +**Supported types:** +- `"lexicon"` - Generate JSON lexicons +- `"typescript"` - Generate TypeScript types +- `"go"` - Generate Go structs +- `"rust"` - Generate Rust structs + +When you run `mlf generate` without arguments, it will generate all configured outputs. + +### Dependencies + +The `[dependencies]` section lists external lexicon namespaces to fetch: + +```toml +[dependencies] +dependencies = [ + "stream.place", + "app.bsky", + "com.atproto" +] +``` + +These dependencies are fetched when you run `mlf fetch` without arguments. + +## Commands Using Configuration + +### `mlf check` + +When run without arguments, checks all files in the source directory: + +```bash +mlf check +# Equivalent to: mlf check "./lexicons/**/*.mlf" +``` + +### `mlf generate` + +When run without a subcommand, generates all configured outputs: + +```bash +mlf generate +# Runs all [[output]] configurations +``` + +### `mlf fetch` + +When run without arguments, fetches all dependencies: + +```bash +mlf fetch +# Fetches: stream.place, app.bsky, com.atproto +``` + +Add `--save` flag to automatically update `mlf.toml`: + +```bash +mlf fetch stream.place --save +# Downloads lexicons AND adds to dependencies array +``` + +## Complete Example + +Here's a complete `mlf.toml` for a TypeScript project using ATProto lexicons: + +```toml +[source] +directory = "./lexicons" + +[[output]] +type = "lexicon" +directory = "./dist/lexicons" + +[[output]] +type = "typescript" +directory = "./src/types" + +[dependencies] +dependencies = [ + "app.bsky", + "com.atproto", + "chat.bsky" +] +``` + +### Workflow + +1. **Fetch dependencies:** + ```bash + mlf fetch + ``` + Downloads `app.bsky`, `com.atproto`, and `chat.bsky` lexicons to `.mlf/lexicons/` + +2. **Write your lexicons:** + Create `.mlf` files in `./lexicons/` that reference the fetched lexicons + +3. **Check for errors:** + ```bash + mlf check + ``` + Validates all MLF files in `./lexicons/` + +4. **Generate outputs:** + ```bash + mlf generate + ``` + Creates: + - JSON lexicons in `./dist/lexicons/` + - TypeScript types in `./src/types/` + +## Fetched Lexicons Cache + +When you fetch lexicons, they're stored in `.mlf/lexicons/`: + +``` +.mlf/ +├── .gitignore # Automatically created +├── .lexicon-cache.toml # Metadata about fetched lexicons +└── lexicons/ + ├── json/ # Original JSON lexicons + │ ├── app.bsky.actor.profile.json + │ └── ... + └── mlf/ # Converted MLF format + ├── app.bsky.actor.profile.mlf + └── ... +``` + +The `.mlf` directory is automatically added to `.gitignore`, so fetched lexicons won't be committed to your repository. + +## Best Practices + +1. **Commit `mlf.toml`** - Version control your configuration +2. **Don't commit `.mlf/`** - Let each developer fetch dependencies +3. **Use semantic namespaces** - Organize lexicons by domain +4. **Multiple outputs** - Generate both lexicons and code simultaneously +5. **CI/CD integration** - Run `mlf check` in your CI pipeline + +## Override Configuration + +You can always override configuration with explicit arguments: + +```bash +# Override source directory +mlf check "./other-lexicons/**/*.mlf" + +# Override output +mlf generate lexicon -i custom.mlf -o ./custom-output/ + +# Fetch specific namespace (ignoring dependencies list) +mlf fetch stream.place +``` diff --git a/website/content/docs/cli/03-check.md b/website/content/docs/cli/03-check.md new file mode 100644 index 0000000..fdf338f --- /dev/null +++ b/website/content/docs/cli/03-check.md @@ -0,0 +1,191 @@ ++++ +title = "Check Command" +description = "Validate MLF lexicon files" +weight = 3 ++++ + +The `mlf check` command validates MLF lexicon files for syntax and type errors. + +## Usage + +```bash +mlf check [INPUT]... +``` + +**Arguments:** +- `[INPUT]...` - MLF lexicon file(s) to validate (glob patterns supported) + +If no input files are provided, `mlf check` will use the source directory from your `mlf.toml` configuration. + +## Examples + +### Check with Configuration + +If you have an `mlf.toml` file: + +```toml +[source] +directory = "./lexicons" +``` + +Simply run: + +```bash +mlf check +``` + +This automatically checks all `.mlf` files in `./lexicons/`. + +### Check Specific Files + +```bash +# Check a single file +mlf check thread.mlf + +# Check multiple files +mlf check thread.mlf profile.mlf reply.mlf +``` + +### Check with Glob Patterns + +```bash +# Check all MLF files in a directory +mlf check "lexicons/**/*.mlf" + +# Check files matching a pattern +mlf check "src/*/schema.mlf" + +# Check all MLF files recursively +mlf check "**/*.mlf" +``` + +## Validation Checks + +The check command performs comprehensive validation: + +### 1. **Syntax Validation** +- Correct MLF syntax +- Proper use of keywords +- Valid identifiers and namespaces +- Correct constraint syntax + +### 2. **Type Validation** +- All referenced types exist +- Type constraints are valid +- Required fields are properly marked +- Array and object structures are correct + +### 3. **Semantic Validation** +- No duplicate definitions +- Valid record keys +- Proper XRPC method signatures +- Correct union type usage + +### 4. **Cross-Reference Validation** +- External references resolve correctly +- Import statements are valid +- Prelude types are accessible +- Fetched dependencies are available + +## Output + +### Success + +When all files are valid: + +``` +✓ thread.mlf: Parsed successfully +✓ profile.mlf: Parsed successfully +✓ reply.mlf: Parsed successfully + +✓ All lexicons are valid +``` + +### Errors + +When errors are found, detailed diagnostics are shown: + +``` + × Undefined reference to 'ProfileView' + ╭─[profile.mlf:5:12] + 5 │ author: ProfileView, + · ^^^^^^^^^^^ 'ProfileView' is not defined + ╰──── + help: Make sure this type is defined in the same file or imported via 'use'. +``` + +Each error includes: +- Error message with context +- Source code snippet +- Exact location (line and column) +- Helpful suggestions for fixing + +## Working with Dependencies + +If your lexicons reference external types (e.g., from `app.bsky` or `com.atproto`), make sure to fetch them first: + +```bash +# Fetch dependencies +mlf fetch + +# Then check your lexicons +mlf check +``` + +The check command automatically loads lexicons from `.mlf/lexicons/mlf/` if they exist. + +## Exit Codes + +- `0` - All files are valid +- `1` - Validation errors found + +## Common Issues + +### Undefined Reference + +``` +× Undefined reference to 'SomeType' +``` + +**Solution:** +- Define `SomeType` in the same file, or +- Fetch the lexicon containing `SomeType` using `mlf fetch`, or +- Add an import statement if needed + +### Parse Error + +``` +× Expected 'constrained', found 'contsrained' +``` + +**Solution:** Fix the typo in your MLF file + +### Type Mismatch + +``` +× Field 'count' expects integer, found string +``` + +**Solution:** Correct the field type to match the schema + +## Integration with CI/CD + +Use `mlf check` in your continuous integration pipeline: + +```yaml +# GitHub Actions example +- name: Validate MLF Lexicons + run: | + mlf fetch + mlf check +``` + +This ensures all lexicons remain valid as your project evolves. + +## Tips + +1. **Use configuration** - Set up `mlf.toml` to avoid typing paths repeatedly +2. **Check often** - Run `mlf check` frequently during development +3. **Version control** - Commit valid lexicons only +4. **Fetch first** - Always fetch dependencies before checking +5. **Read errors carefully** - MLF provides detailed error messages with helpful suggestions diff --git a/website/content/docs/cli/04-validate.md b/website/content/docs/cli/04-validate.md new file mode 100644 index 0000000..e8ce37c --- /dev/null +++ b/website/content/docs/cli/04-validate.md @@ -0,0 +1,211 @@ ++++ +title = "Validate Command" +description = "Validate JSON records against lexicons" +weight = 4 ++++ + +The `mlf validate` command validates JSON record data against an MLF lexicon schema. + +## Usage + +```bash +mlf validate +``` + +**Arguments:** +- `` - MLF lexicon file defining the schema +- `` - JSON file containing the record to validate + +## Example + +Given a lexicon file `thread.mlf`: + +```mlf +record main { + title!: string constrained { + maxLength: 200, + }, + createdAt!: Datetime, + posts: Post[], +}; + +def Post = { + text!: string, + createdAt!: Datetime, +}; +``` + +And a JSON record file `my-thread.json`: + +```json +{ + "title": "My Discussion Thread", + "createdAt": "2024-01-15T10:30:00Z", + "posts": [ + { + "text": "First post!", + "createdAt": "2024-01-15T10:30:00Z" + } + ] +} +``` + +Validate the record: + +```bash +mlf validate thread.mlf my-thread.json +``` + +## Output + +### Valid Record + +``` +✓ Lexicon parsed successfully +✓ JSON record parsed successfully +✓ Record is valid according to the lexicon schema +``` + +### Invalid Record + +If the record doesn't match the schema: + +``` +✗ Record validation failed with 2 error(s): + • Field 'title' is required but missing + • Field 'createdAt': expected string in datetime format, found "not-a-date" +``` + +## Validation Rules + +The validate command checks: + +### 1. **Required Fields** +- All fields marked with `!` must be present +- Optional fields can be omitted + +### 2. **Type Matching** +- String fields must be strings +- Integer fields must be numbers +- Arrays must be arrays +- Objects must have correct structure + +### 3. **Constraints** +- String length constraints (`maxLength`, `minLength`) +- String grapheme constraints (`maxGraphemes`, `minGraphemes`) +- Integer range constraints (`minimum`, `maximum`) +- Array length constraints +- Format validation (`datetime`, `uri`, `at-uri`, etc.) + +### 4. **Nested Structures** +- Object fields are recursively validated +- Array items match their item schema +- References to other defs are resolved + +## Use Cases + +### 1. **Test Data Validation** + +Validate test fixtures before using them: + +```bash +mlf validate profile.mlf test-data/profile-1.json +``` + +### 2. **API Response Validation** + +Check that API responses match your schema: + +```bash +curl https://api.example.com/profile > response.json +mlf validate profile.mlf response.json +``` + +### 3. **Schema Migration** + +When updating schemas, validate existing records against the new schema: + +```bash +# Validate all records +for file in records/*.json; do + mlf validate schema.mlf "$file" || echo "Failed: $file" +done +``` + +### 4. **Development Workflow** + +Validate records during development: + +```bash +# Write some test data +echo '{"title": "Test", "createdAt": "2024-01-15T10:30:00Z"}' > test.json + +# Validate it +mlf validate thread.mlf test.json +``` + +## Common Validation Errors + +### Missing Required Field + +``` +✗ Field 'title' is required but missing +``` + +**Solution:** Add the required field to your JSON + +### Type Mismatch + +``` +✗ Field 'count': expected integer, found "123" +``` + +**Solution:** Use a number instead of a string: `"count": 123` + +### Constraint Violation + +``` +✗ Field 'title': string length 250 exceeds maxLength 200 +``` + +**Solution:** Shorten the string to meet the constraint + +### Invalid Format + +``` +✗ Field 'createdAt': expected datetime format, found "2024-01-15" +``` + +**Solution:** Use full ISO 8601 datetime format: `"2024-01-15T10:30:00Z"` + +### Invalid Array Item + +``` +✗ Array item at index 0: Field 'text' is required but missing +``` + +**Solution:** Ensure all array items match the schema + +## Exit Codes + +- `0` - Record is valid +- `1` - Validation failed or error occurred + +## Limitations + +The validate command currently validates against the lexicon structure but does not: + +- Validate CID references +- Validate blob content types +- Check external references +- Verify cryptographic signatures + +These checks would typically be performed by the PDS or other ATProto infrastructure. + +## Tips + +1. **Start simple** - Validate basic records first, then add complexity +2. **Use test data** - Create JSON fixtures for your schemas +3. **Automate validation** - Add validation to your test suite +4. **Check constraints** - Pay attention to string lengths and ranges +5. **Format matters** - Use proper datetime/URI formats as specified by ATProto diff --git a/website/content/docs/cli/05-generate.md b/website/content/docs/cli/05-generate.md new file mode 100644 index 0000000..71a300d --- /dev/null +++ b/website/content/docs/cli/05-generate.md @@ -0,0 +1,355 @@ ++++ +title = "Generate Commands" +description = "Generate code and lexicons from MLF" +weight = 5 ++++ + +The `mlf generate` command converts MLF files to various output formats including JSON lexicons and code in multiple programming languages. + +## Overview + +```bash +# Generate all configured outputs +mlf generate + +# Generate JSON lexicons +mlf generate lexicon -i -o + +# Generate code in a specific language +mlf generate code -g -i -o + +# Convert JSON lexicons to MLF +mlf generate mlf -i -o +``` + +## Generate All Outputs + +When run without a subcommand, `mlf generate` uses your `mlf.toml` configuration to generate all specified outputs: + +```toml +[[output]] +type = "lexicon" +directory = "./dist/lexicons" + +[[output]] +type = "typescript" +directory = "./src/types" +``` + +```bash +mlf generate +``` + +This will: +1. Read the source directory from configuration +2. Process all `.mlf` files +3. Generate each configured output type + +**Output:** +``` +Running 2 output configuration(s)... + +Generating lexicon output to ./dist/lexicons... +Generated: ./dist/lexicons/com/example/thread.json + ✓ Generated lexicon output successfully + +Generating typescript output to ./src/types... +Generated: ./src/types/com/example/thread.ts + ✓ Generated typescript output successfully + +✓ Successfully generated all 2 output(s) +``` + +--- + +## Generate Lexicon (JSON) + +Generate ATProto JSON lexicons from MLF files. + +```bash +mlf generate lexicon -i -o [OPTIONS] +``` + +**Options:** +- `-i, --input ` - Input MLF files (glob patterns supported, can be specified multiple times) +- `-o, --output ` - Output directory (required) +- `--flat` - Use flat file structure (e.g., `com.example.thread.json`) + +**Examples:** + +```bash +# Generate with folder structure +mlf generate lexicon -i thread.mlf -o lexicons/ +# Creates: lexicons/com/example/thread.json + +# Generate with flat structure +mlf generate lexicon -i thread.mlf -o lexicons/ --flat +# Creates: lexicons/com.example.thread.json + +# Generate from multiple files +mlf generate lexicon -i thread.mlf -i reply.mlf -o lexicons/ + +# Generate from glob pattern +mlf generate lexicon -i "src/**/*.mlf" -o dist/lexicons/ +``` + +--- + +## Generate Code + +Generate code in various programming languages from MLF files. + +```bash +mlf generate code -g -i -o [OPTIONS] +``` + +**Options:** +- `-g, --generator ` - Generator to use (required): `json`, `typescript`, `go`, or `rust` +- `-i, --input ` - Input MLF files (glob patterns supported, can be specified multiple times) +- `-o, --output ` - Output directory (required) +- `--flat` - Use flat file structure + +**Available Generators:** + +| Generator | Output | Features | +|-----------|--------|----------| +| `json` | `.json` | AT Protocol JSON lexicons (always available) | +| `typescript` | `.ts` | TypeScript interfaces with JSDoc, optional fields with `?` | +| `go` | `.go` | Go structs with JSON tags, proper capitalization | +| `rust` | `.rs` | Rust structs with serde, `Option` for optional fields | + +### TypeScript Example + +```bash +mlf generate code -g typescript -i thread.mlf -o src/types/ +``` + +**Input MLF:** +```mlf +/// A discussion thread +record main { + /// Thread title + title!: string constrained { + maxLength: 200, + }, + /// Creation timestamp + createdAt!: Datetime, + posts: Post[], +}; +``` + +**Generated TypeScript:** +```typescript +/** + * Generated from com.example.thread + * Do not edit manually + */ + +/** + * A discussion thread + */ +export interface Main { + /** Thread title */ + title: string; + /** Creation timestamp */ + createdAt: string; + posts?: Post[]; +} +``` + +### Go Example + +```bash +mlf generate code -g go -i thread.mlf -o pkg/models/ +``` + +**Generated Go:** +```go +// Generated from com.example.thread +// Do not edit manually + +package thread + +// Main represents a discussion thread +type Main struct { + // Thread title + Title string `json:"title"` + // Creation timestamp + CreatedAt string `json:"createdAt"` + // Posts (optional) + Posts []Post `json:"posts,omitempty"` +} +``` + +### Rust Example + +```bash +mlf generate code -g rust -i thread.mlf -o src/models/ +``` + +**Generated Rust:** +```rust +// Generated from com.example.thread +// Do not edit manually + +use serde::{Deserialize, Serialize}; + +/// A discussion thread +#[derive(Debug, Clone, Serialize, Deserialize)] +pub struct Main { + /// Thread title + pub title: String, + /// Creation timestamp + #[serde(rename = "createdAt")] + pub created_at: String, + /// Posts (optional) + #[serde(skip_serializing_if = "Option::is_none")] + pub posts: Option>, +} +``` + +--- + +## Generate MLF + +Convert ATProto JSON lexicons back to MLF format. + +```bash +mlf generate mlf -i -o +``` + +**Options:** +- `-i, --input ` - Input JSON lexicon files (glob patterns supported, can be specified multiple times) +- `-o, --output ` - Output directory (required) + +**Examples:** + +```bash +# Convert a single JSON lexicon to MLF +mlf generate mlf -i com.example.thread.json -o ./lexicons/ +# Creates: lexicons/com/example/thread.mlf + +# Convert multiple JSON lexicons +mlf generate mlf -i lexicon1.json -i lexicon2.json -o ./mlf/ + +# Convert using glob pattern +mlf generate mlf -i "dist/lexicons/**/*.json" -o ./src/ +``` + +**Features:** + +- **Smart type conversion** - Automatically converts format strings to prelude types + - `"format": "did"` → `Did` + - `"format": "datetime"` → `Datetime` + - `"format": "handle"` → `Handle` +- **Proper formatting** - Generates clean, properly indented MLF +- **Reference conversion** - Converts `namespace#name` to `namespace.name` +- **Complete coverage** - Supports all ATProto lexicon types + +**Input JSON:** +```json +{ + "lexicon": 1, + "id": "com.example.thread", + "defs": { + "main": { + "type": "record", + "record": { + "type": "object", + "required": ["title", "createdAt"], + "properties": { + "title": { + "type": "string", + "maxLength": 200 + }, + "createdAt": { + "type": "string", + "format": "datetime" + } + } + } + } + } +} +``` + +**Generated MLF:** +```mlf +record main { + title!: string constrained { + maxLength: 200, + }, + createdAt!: Datetime, +}; +``` + +--- + +## Code Generator Features + +### Documentation Comments + +All generators preserve documentation comments from MLF: + +```mlf +/// This is a user profile +def Profile = { + /// The user's display name + displayName: string, +}; +``` + +Generates appropriate doc comments for each language (JSDoc, Go comments, Rust doc comments). + +### Optional Fields + +Optional fields are handled idiomatically in each language: + +- **TypeScript**: `field?: Type` +- **Go**: `Field *Type` or `json:",omitempty"` +- **Rust**: `field: Option` with `#[serde(skip_serializing_if = "Option::is_none")]` + +### Field Naming Conventions + +Each generator follows language conventions: + +- **TypeScript**: camelCase (matches JSON) +- **Go**: PascalCase with JSON tags +- **Rust**: snake_case with `#[serde(rename)]` attributes + +### Type Mapping + +MLF types are mapped to appropriate language types: + +| MLF Type | TypeScript | Go | Rust | +|----------|------------|-------|------| +| `string` | `string` | `string` | `String` | +| `integer` | `number` | `int64` | `i64` | +| `boolean` | `boolean` | `bool` | `bool` | +| `bytes` | `Uint8Array` | `[]byte` | `Vec` | +| `array` | `T[]` | `[]T` | `Vec` | +| `Did` | `string` | `string` | `String` | +| `Datetime` | `string` | `string` | `String` | + +--- + +## Tips + +1. **Use configuration** - Set up `mlf.toml` for multi-output generation +2. **Commit generated code** - If it's part of your build artifacts +3. **Regenerate often** - Run `mlf generate` after any lexicon changes +4. **Use flat mode** - For simpler directory structures +5. **Multiple generators** - Generate multiple languages from the same MLF files +6. **Version control** - Track both MLF source and generated code + +## Error Handling + +If generation fails for some files, you'll see detailed errors: + +``` +3 file(s) generated successfully, 1 error(s) encountered: + + thread.mlf - Type resolution error: undefined reference to 'Post' +``` + +The command exits with status `1` if any errors occur. diff --git a/website/content/docs/cli/06-fetch.md b/website/content/docs/cli/06-fetch.md new file mode 100644 index 0000000..c30d360 --- /dev/null +++ b/website/content/docs/cli/06-fetch.md @@ -0,0 +1,307 @@ ++++ +title = "Fetch Command" +description = "Download lexicons from remote repositories" +weight = 6 ++++ + +The `mlf fetch` command downloads ATProto lexicons from remote repositories and converts them to MLF format. + +## Usage + +```bash +# Fetch all dependencies from mlf.toml +mlf fetch + +# Fetch a specific namespace +mlf fetch + +# Fetch and save to dependencies +mlf fetch --save +``` + +**Arguments:** +- `[NAMESPACE]` - Optional namespace to fetch (e.g., `stream.place`, `app.bsky`) + +**Options:** +- `--save` - Add the namespace to dependencies in `mlf.toml` + +## How It Works + +The fetch command follows the ATProto lexicon discovery protocol: + +1. **DNS Lookup** - Queries `_lexicon.` TXT record +2. **DID Resolution** - Resolves the DID to a PDS endpoint +3. **Fetch Records** - Queries `com.atproto.repo.listRecords` for lexicon schemas +4. **Save & Convert** - Saves JSON and converts to MLF format + +## Examples + +### Fetch All Dependencies + +With an `mlf.toml` file: + +```toml +[dependencies] +dependencies = ["stream.place", "app.bsky"] +``` + +Run: + +```bash +mlf fetch +``` + +**Output:** +``` +Fetching 2 dependencies... + +Fetching: stream.place +Fetching lexicons for authority: stream.place + → Resolved DID: did:web:stream.place + → Using PDS: https://stream.place + → Found 5 lexicon record(s) + Processing: stream.place.thread + → Saved JSON to .mlf/lexicons/json/stream.place.thread.json + → Converted to MLF at .mlf/lexicons/mlf/stream.place.thread.mlf +✓ Successfully fetched lexicons for stream.place + +Fetching: app.bsky +... + +✓ Successfully fetched all 2 dependencies +``` + +### Fetch Specific Namespace + +```bash +mlf fetch stream.place +``` + +This downloads all lexicons under the `stream.place` authority. + +### Fetch and Save + +```bash +mlf fetch stream.place --save +``` + +This: +1. Downloads the lexicons +2. Adds `"stream.place"` to the dependencies array in `mlf.toml` +3. Creates `mlf.toml` if it doesn't exist + +## Storage Structure + +Fetched lexicons are stored in `.mlf/lexicons/`: + +``` +.mlf/ +├── .gitignore # Auto-generated +├── .lexicon-cache.toml # Cache metadata +└── lexicons/ + ├── json/ # Original JSON lexicons + │ ├── stream.place.thread.json + │ └── app.bsky.actor.profile.json + └── mlf/ # Converted MLF format + ├── stream.place.thread.mlf + └── app.bsky.actor.profile.mlf +``` + +### Cache File + +The `.lexicon-cache.toml` tracks what's been fetched: + +```toml +[[lexicons.stream.place.thread]] +nsid = "stream.place.thread" +fetched_at = "2024-01-15T10:30:00Z" +did = "did:web:stream.place" +``` + +## DNS Resolution + +For a namespace like `stream.place.thread`: + +1. Extract authority: `stream.place` +2. Reverse for DNS: `place.stream` +3. Query TXT record: `_lexicon.place.stream` +4. Parse `did=did:web:...` or `did=did:plc:...` + +**Example DNS record:** +``` +_lexicon.place.stream. 300 IN TXT "did=did:web:stream.place" +``` + +## DID Resolution + +### did:web + +For `did:web:stream.place`, the PDS is `https://stream.place` + +### did:plc + +For `did:plc:abc123...`, query `https://plc.directory/did:plc:abc123...` to get the PDS endpoint from the DID document. + +## Fetched Lexicons in Your Code + +Once fetched, lexicons in `.mlf/lexicons/mlf/` are automatically available for: + +### Type References + +```mlf +use stream.place.thread; + +def Reply = { + thread!: stream.place.thread, + text!: string, +}; +``` + +### Code Generation + +```bash +mlf generate code -g typescript -i my-lexicon.mlf -o src/types/ +``` + +The generator can resolve references to fetched lexicons. + +### Validation + +```bash +mlf check my-lexicon.mlf +``` + +The check command loads fetched lexicons for type resolution. + +## Working Without mlf.toml + +If you don't have an `mlf.toml`, the fetch command will offer to create one: + +```bash +$ mlf fetch stream.place +No mlf.toml found in current or parent directories. +Would you like to create one in the current directory? (y/n) +y +Created mlf.toml in /path/to/current/dir +... +``` + +## Re-fetching + +If a namespace is already cached, fetch skips it: + +```bash +$ mlf fetch stream.place +Lexicon 'stream.place.thread' is already cached. Skipping fetch. + (Use --force to re-fetch) +``` + +To re-fetch: + +```bash +mlf fetch stream.place --force # Not yet implemented +``` + +## Error Handling + +### DNS Errors + +``` +✗ DNS lookup failed: No TXT record found for _lexicon.place.stream +``` + +**Causes:** +- Domain doesn't have a lexicon TXT record +- DNS propagation delay +- Network issues + +### DID Resolution Errors + +``` +✗ Failed to resolve DID: No PDS endpoint found in DID document +``` + +**Causes:** +- Invalid DID format +- PLC directory unreachable +- DID document missing PDS service + +### No Records Found + +``` +✗ No lexicon records found for stream.place +``` + +**Causes:** +- Namespace exists but has no published lexicons +- Wrong namespace (typo) +- PDS doesn't support lexicon publishing + +## Best Practices + +1. **Fetch before work** - Always fetch dependencies before coding +2. **Use --save** - Keep `mlf.toml` up to date with dependencies +3. **Don't commit `.mlf/`** - Let each developer fetch independently +4. **Check DNS** - Verify TXT records before fetching +5. **Version dependencies** - Consider tracking lexicon versions (future feature) + +## CI/CD Integration + +In your CI pipeline: + +```yaml +# GitHub Actions example +- name: Fetch ATProto Lexicons + run: | + mlf fetch + +- name: Generate Code + run: | + mlf generate +``` + +This ensures builds have access to the latest lexicons. + +## Comparison with npm/cargo + +The fetch command is similar to package managers: + +| Command | npm | cargo | mlf | +|---------|-----|-------|-----| +| Install deps | `npm install` | `cargo fetch` | `mlf fetch` | +| Add dep | `npm install pkg --save` | `cargo add pkg` | `mlf fetch ns --save` | +| Config file | `package.json` | `Cargo.toml` | `mlf.toml` | +| Cache | `node_modules/` | `~/.cargo/` | `.mlf/` | + +## Troubleshooting + +### Network Issues + +```bash +# Check DNS resolution +dig TXT _lexicon.place.stream + +# Test DID resolution +curl https://plc.directory/did:plc:abc123 +``` + +### Invalid Namespace + +Make sure you're using the correct namespace format: +- ✓ `stream.place` +- ✓ `app.bsky` +- ✗ `stream.place.thread` (too specific) + +### Permission Errors + +Ensure you have write permissions for the project directory to create `.mlf/`. + +## Future Features + +Planned enhancements: + +- `--force` flag to re-fetch cached lexicons +- Version pinning (fetch specific lexicon versions) +- Private/authenticated repositories +- Offline mode with local cache +- Fetch from local directories diff --git a/website/content/docs/cli/07-errors.md b/website/content/docs/cli/07-errors.md new file mode 100644 index 0000000..b18f865 --- /dev/null +++ b/website/content/docs/cli/07-errors.md @@ -0,0 +1,335 @@ ++++ +title = "Error Messages" +description = "Understanding MLF error diagnostics" +weight = 7 ++++ + +MLF provides rich, helpful error messages with source code context and suggestions for fixing issues. + +## Error Format + +All MLF errors follow this format: + +``` + × Error title + ╭─[file.mlf:5:12] + 5 │ author: ProfileView, + · ^^^^^^^^^^^ error message + ╰──── + help: Suggestion for fixing the issue +``` + +**Components:** +- **× Error title** - Brief description of the error +- **Source location** - File name, line, and column +- **Source context** - The relevant code snippet +- **Error span** - Highlighted location with `^^^` +- **help:** - Actionable suggestion for fixing + +## Common Errors + +### Parse Errors + +#### Expected Token + +``` + × Expected '}', found ',' + ╭─[thread.mlf:10:5] +10 │ title: string, + · ^ expected '}' + ╰──── + help: Check for missing closing braces or extra commas +``` + +**Cause:** Syntax error in MLF code + +**Fix:** Correct the syntax according to MLF grammar + +#### Invalid Identifier + +``` + × Invalid identifier: '123invalid' + ╭─[thread.mlf:5:5] + 5 │ 123invalid: string, + · ^^^^^^^^^^ identifiers cannot start with numbers + ╰──── + help: Identifiers must start with a letter or underscore +``` + +**Cause:** Identifier doesn't follow naming rules + +**Fix:** Start identifiers with letters or underscores + +### Type Errors + +#### Undefined Reference + +``` + × Undefined reference to 'ProfileView' + ╭─[thread.mlf:8:12] + 8 │ author: ProfileView, + · ^^^^^^^^^^^ 'ProfileView' is not defined + ╰──── + help: Make sure this type is defined in the same file or imported via 'use'. +``` + +**Causes:** +- Type is not defined +- Type is in another file and not imported +- Typo in type name + +**Fixes:** +- Define the type in the same file +- Use `mlf fetch` to download external lexicons +- Add a `use` statement (if MLF supports imports) +- Check spelling + +#### Type Mismatch + +``` + × Type mismatch: expected integer, found string + ╭─[thread.mlf:12:15] +12 │ count: string, + · ^^^^^^ expected integer type + ╰──── + help: Change this to 'integer' or update the constraint +``` + +**Cause:** Field type doesn't match its definition + +**Fix:** Update the type to match the schema + +### Constraint Errors + +#### Invalid Constraint + +``` + × Invalid constraint for type 'integer': 'maxLength' + ╭─[thread.mlf:15:9] +15 │ maxLength: 100, + · ^^^^^^^^^ 'maxLength' is only valid for string types + ╰──── + help: Use 'maximum' for integer constraints +``` + +**Cause:** Constraint doesn't apply to the type + +**Fix:** Use the correct constraint for the type: +- String: `maxLength`, `minLength`, `maxGraphemes`, `minGraphemes` +- Integer: `minimum`, `maximum` +- Array: `maxLength`, `minLength` + +#### Constraint Value Error + +``` + × Constraint value must be positive + ╭─[thread.mlf:18:20] +18 │ maxLength: -10, + · ^^^ negative value not allowed + ╰──── + help: Use a positive integer value +``` + +**Cause:** Constraint value is invalid + +**Fix:** Use a valid value according to the constraint rules + +### Record Errors + +#### Missing Record Key + +``` + × Record definition must specify a key type + ╭─[thread.mlf:3:1] + 3 │ record main { + · ^^^^^^^^^^^ missing key specification + ╰──── + help: Add 'key: "tid"' or another valid key type to the record +``` + +**Cause:** Record doesn't specify how records are keyed + +**Fix:** Add a key specification to the record definition + +### XRPC Errors + +#### Invalid Response Code + +``` + × Invalid HTTP response code: 999 + ╭─[api.mlf:25:5] +25 │ 999: error, + · ^^^ response code must be between 200-599 + ╰──── + help: Use a valid HTTP status code +``` + +**Cause:** Invalid HTTP status code in query/procedure + +**Fix:** Use standard HTTP status codes (200, 400, 401, 404, 500, etc.) + +#### Missing Required Response + +``` + × Query/procedure must define at least one success response + ╭─[api.mlf:20:1] +20 │ query getProfile(...) { + · ^^^^^^^^^^^^^^^^^^^^^ no 200-level responses defined + ╰──── + help: Add at least one 2xx response (typically 200) +``` + +**Cause:** XRPC method has no success responses + +**Fix:** Add a 200-level response + +### Union Errors + +#### Empty Union + +``` + × Union must have at least one type + ╭─[thread.mlf:30:15] +30 │ data: unit | , + · ^ empty union + ╰──── + help: Add at least one type to the union +``` + +**Cause:** Union has no types + +**Fix:** Add types to the union: `string | integer` + +#### Duplicate Union Types + +``` + × Duplicate type in union: 'string' + ╭─[thread.mlf:32:20] +32 │ data: string | string, + · ^^^^^^ duplicate type + ╰──── + help: Remove duplicate types from the union +``` + +**Cause:** Same type appears multiple times in union + +**Fix:** Remove duplicates + +## Validation Errors + +### Record Validation + +``` +✗ Record validation failed with 2 error(s): + • Field 'title' is required but missing + • Field 'count': expected integer, found "123" +``` + +**Cause:** JSON record doesn't match lexicon schema + +**Fix:** Update the JSON to match the schema + +## Generation Errors + +### File Read Error + +``` +Failed to read file: permission denied +``` + +**Cause:** Cannot read input file + +**Fix:** Check file permissions and path + +### Type Resolution Error + +``` +Type resolution error: circular reference detected +``` + +**Cause:** Types reference each other in a cycle + +**Fix:** Break the circular dependency + +### Code Generation Error + +``` +Generator 'typescript' not found +``` + +**Cause:** Generator feature not enabled + +**Fix:** Install with `--features typescript` or use `--all-features` + +## Fetch Errors + +### DNS Error + +``` +✗ DNS lookup failed: No TXT record found for _lexicon.place.stream +``` + +**Cause:** Domain has no lexicon TXT record + +**Fix:** Verify the namespace is correct and has published lexicons + +### Network Error + +``` +✗ Failed to fetch lexicon records: connection timeout +``` + +**Cause:** Network connectivity issues + +**Fix:** Check internet connection and try again + +### Parse Error + +``` +✗ Failed to parse lexicon JSON: unexpected end of input +``` + +**Cause:** Malformed JSON from remote repository + +**Fix:** Report issue to namespace maintainer + +## Exit Codes + +All MLF commands use standard exit codes: + +- `0` - Success +- `1` - Error occurred + +Use in scripts: + +```bash +if mlf check; then + echo "✓ Valid" +else + echo "✗ Invalid" + exit 1 +fi +``` + +## Tips for Reading Errors + +1. **Read the title** - Gives you the high-level issue +2. **Check the location** - Find the exact line and column +3. **Examine the span** - See what code is problematic +4. **Follow the help** - Actionable suggestions for fixes +5. **Look for patterns** - Similar errors often have similar fixes + +## Reporting Bugs + +If you encounter an error that seems like a bug: + +1. Note the exact error message +2. Create a minimal reproduction case +3. Check if it's a known issue +4. Report at: https://github.com/anthropics/claude-code/issues + +Include: +- MLF version (`mlf --version`) +- Complete error message +- Minimal MLF code that reproduces the issue +- Expected behavior vs actual behavior diff --git a/website/content/docs/cli/_index.md b/website/content/docs/cli/_index.md new file mode 100644 index 0000000..b77bdf7 --- /dev/null +++ b/website/content/docs/cli/_index.md @@ -0,0 +1,60 @@ ++++ +title = "CLI Reference" +description = "Command-line tool documentation" +weight = 3 +sort_by = "weight" +template = "section.html" ++++ + +The MLF CLI is a powerful command-line tool for working with Matt's Lexicon Format and ATProto lexicons. + +## Overview + +The `mlf` command-line tool provides: + +- **Validation** - Check MLF files for syntax and type errors +- **Code Generation** - Generate TypeScript, Go, Rust, and JSON from MLF +- **Lexicon Fetching** - Download lexicons from remote ATProto repositories +- **Format Conversion** - Convert between MLF and JSON lexicon formats +- **Project Management** - Configure projects with `mlf.toml` + +## Quick Start + +```bash +# Check MLF files +mlf check + +# Generate TypeScript types +mlf generate code -g typescript -i "lexicons/**/*.mlf" -o src/types/ + +# Fetch remote lexicons +mlf fetch stream.place --save + +# Generate all configured outputs +mlf generate +``` + +## Project Configuration + +Projects can be configured with an `mlf.toml` file that specifies source directories, output configurations, and dependencies. This allows you to run commands without repeatedly specifying arguments. + +See the [Configuration](02-configuration.md) section for details. + +## Command Categories + +### Validation Commands +- `mlf check` - Validate MLF syntax and types +- `mlf validate` - Validate JSON records against lexicons + +### Generation Commands +- `mlf generate lexicon` - Generate JSON lexicons +- `mlf generate code` - Generate code in various languages +- `mlf generate mlf` - Convert JSON lexicons to MLF +- `mlf generate` (no subcommand) - Generate all configured outputs + +### Fetching Commands +- `mlf fetch` - Download lexicons from remote repositories +- `mlf fetch ` - Fetch specific namespace +- `mlf fetch --save` - Fetch and add to dependencies + +This guide walks through each command in detail, with examples and usage patterns.