diff --git a/docs/implementation-plans/2026-02-23-dataflow-renderer/phase_01.md b/docs/implementation-plans/2026-02-23-dataflow-renderer/phase_01.md new file mode 100644 index 0000000..a1812d2 --- /dev/null +++ b/docs/implementation-plans/2026-02-23-dataflow-renderer/phase_01.md @@ -0,0 +1,375 @@ +# Dataflow Graph Renderer Implementation Plan — Phase 1: Project Scaffolding + +**Goal:** Create the `dfgraph/` package structure and install all dependencies (Python + JS). + +**Architecture:** Two-process system — Python backend (FastAPI + WebSocket) serves graph data, browser frontend (cytoscape.js) renders it. This phase creates the skeleton for both. + +**Tech Stack:** Python 3.12, FastAPI, uvicorn, watchdog, Node.js, cytoscape.js, cytoscape-dagre, esbuild + +**Scope:** 8 phases from original design (this is phase 1 of 8) + +**Codebase verified:** 2026-02-23 + +--- + +## Acceptance Criteria Coverage + +This phase is infrastructure scaffolding. **Verifies: None** — verified operationally (CLI prints help, esbuild bundles, imports resolve). + +--- + + +### Task 1: Add Python dependencies to flake.nix + +**Files:** +- Modify: `/home/orual/Projects/or1-design/flake.nix:20-31` (pythonPackages list) + +**Step 1: Add fastapi, uvicorn, websockets, and watchdog to the pythonPackages list** + +In `/home/orual/Projects/or1-design/flake.nix`, modify the `pythonPackages` function (lines 20–31) to include the new dependencies. Add them after `typing-extensions`: + +```nix +pythonPackages = ps: + with ps; [ + ipykernel + jupyterlab + numpy + matplotlib + pip + simpy + pytest + hypothesis + typing-extensions + lark + fastapi + uvicorn + websockets + watchdog + ]; +``` + +**Step 2: Verify the flake evaluates** + +```bash +cd /home/orual/Projects/or1-design +nix flake check 2>&1 | head -20 +``` + +Expected: No errors. + +**Step 3: Re-enter dev shell and verify imports** + +```bash +# Exit and re-enter the shell (or direnv reload if using direnv) +nix develop +python -c "import lark; import fastapi; import uvicorn; import watchdog; print('OK')" +``` + +Expected: Prints `OK`. + +**Step 4: Commit** + +```bash +jj commit -m "chore: add lark, fastapi, uvicorn, websockets, watchdog to flake.nix" +``` + + + +### Task 2: Create dfgraph package skeleton + +**Files:** +- Create: `/home/orual/Projects/or1-design/dfgraph/__init__.py` +- Create: `/home/orual/Projects/or1-design/dfgraph/__main__.py` + +**Step 1: Create the dfgraph directory and __init__.py** + +Create `/home/orual/Projects/or1-design/dfgraph/__init__.py`: + +```python +"""Dataflow graph renderer for OR1 dfasm programs.""" +``` + +**Step 2: Create __main__.py with CLI argument parsing** + +Create `/home/orual/Projects/or1-design/dfgraph/__main__.py`: + +```python +"""CLI entry point: python -m dfgraph path/to/file.dfasm [--port 8420]""" + +import argparse +import sys +from pathlib import Path + + +def main() -> None: + parser = argparse.ArgumentParser( + prog="dfgraph", + description="Visualise a dfasm dataflow program as an interactive graph.", + ) + parser.add_argument( + "file", + type=Path, + help="Path to the .dfasm source file to visualise", + ) + parser.add_argument( + "--port", + type=int, + default=8420, + help="Port for the web server (default: 8420)", + ) + + args = parser.parse_args() + + if not args.file.exists(): + print(f"Error: file not found: {args.file}", file=sys.stderr) + sys.exit(1) + + if not args.file.suffix == ".dfasm": + print(f"Warning: expected .dfasm file, got: {args.file.suffix}", file=sys.stderr) + + # Server startup will be added in Phase 4 + print(f"dfgraph: would serve {args.file} on port {args.port}") + + +if __name__ == "__main__": + main() +``` + +**Step 3: Verify CLI works** + +```bash +cd /home/orual/Projects/or1-design +python -m dfgraph --help +``` + +Expected output includes: +``` +usage: dfgraph [-h] [--port PORT] file +``` + +**Step 4: Commit** + +```bash +jj commit -m "feat: add dfgraph package skeleton with CLI entry point" +``` + + + +### Task 3: Create frontend project with cytoscape.js + +**Files:** +- Create: `/home/orual/Projects/or1-design/dfgraph/frontend/package.json` +- Create: `/home/orual/Projects/or1-design/dfgraph/frontend/tsconfig.json` +- Create: `/home/orual/Projects/or1-design/dfgraph/frontend/src/main.ts` +- Create: `/home/orual/Projects/or1-design/dfgraph/frontend/index.html` + +**Step 1: Create the frontend directory structure** + +```bash +mkdir -p /home/orual/Projects/or1-design/dfgraph/frontend/src +mkdir -p /home/orual/Projects/or1-design/dfgraph/frontend/dist +``` + +**Step 2: Create package.json** + +Create `/home/orual/Projects/or1-design/dfgraph/frontend/package.json`: + +```json +{ + "name": "dfgraph-frontend", + "version": "0.1.0", + "private": true, + "type": "module", + "scripts": { + "build": "esbuild src/main.ts --bundle --outfile=dist/bundle.js --format=esm --target=es2020", + "watch": "esbuild src/main.ts --bundle --outfile=dist/bundle.js --format=esm --target=es2020 --watch" + }, + "dependencies": { + "cytoscape": "^3.30.0", + "cytoscape-dagre": "^2.5.0" + }, + "devDependencies": { + "esbuild": "^0.24.0", + "@types/cytoscape": "^3.21.0" + } +} +``` + +**Step 3: Create tsconfig.json** + +Create `/home/orual/Projects/or1-design/dfgraph/frontend/tsconfig.json`: + +```json +{ + "compilerOptions": { + "target": "ES2020", + "module": "ES2020", + "moduleResolution": "bundler", + "strict": true, + "esModuleInterop": true, + "skipLibCheck": true, + "outDir": "dist", + "rootDir": "src", + "declaration": false + }, + "include": ["src/**/*.ts"] +} +``` + +**Step 4: Create minimal main.ts entry point** + +Create `/home/orual/Projects/or1-design/dfgraph/frontend/src/main.ts`: + +```typescript +import cytoscape from "cytoscape"; +import dagre from "cytoscape-dagre"; + +cytoscape.use(dagre); + +const cy = cytoscape({ + container: document.getElementById("graph"), + style: [ + { + selector: "node", + style: { + label: "data(label)", + "text-valign": "center", + "text-halign": "center", + }, + }, + { + selector: "edge", + style: { + "curve-style": "bezier", + "target-arrow-shape": "triangle", + }, + }, + ], + elements: [], +}); + +console.log("dfgraph frontend initialized", cy); +``` + +**Step 5: Create index.html** + +Create `/home/orual/Projects/or1-design/dfgraph/frontend/index.html`: + +```html + + + + + + dfgraph — Dataflow Graph Renderer + + + +
+

dfgraph

+
+
+ + + +``` + +**Step 6: Install npm dependencies and build** + +```bash +cd /home/orual/Projects/or1-design/dfgraph/frontend +npm install +npm run build +``` + +Expected: `dist/bundle.js` is created without errors. + +**Step 7: Commit** + +```bash +cd /home/orual/Projects/or1-design +jj commit -m "feat: add dfgraph frontend skeleton with cytoscape.js" +``` + + + +### Task 4: Add .gitignore entries for frontend build artifacts + +**Files:** +- Modify or create: `/home/orual/Projects/or1-design/.gitignore` + +**Step 1: Add ignore entries** + +Ensure the following entries are in `.gitignore` (create the file if it doesn't exist, or append if it does): + +``` +# dfgraph frontend +dfgraph/frontend/node_modules/ +dfgraph/frontend/dist/ +``` + +Note: The root `node_modules/` may already be gitignored. Check first — if a root `.gitignore` already covers `node_modules/`, the `dfgraph/frontend/node_modules/` entry is still useful for explicitness. + +**Step 2: Verify** + +```bash +cd /home/orual/Projects/or1-design +jj status +``` + +Expected: `dfgraph/frontend/node_modules/` and `dfgraph/frontend/dist/` do not appear as untracked. + +**Step 3: Commit** + +```bash +jj commit -m "chore: add gitignore entries for dfgraph frontend artifacts" +``` + + + +### Task 5: Verify full scaffolding works end-to-end + +**Files:** None (verification only) + +**Step 1: Verify Python CLI** + +```bash +cd /home/orual/Projects/or1-design +python -m dfgraph --help +``` + +Expected: Usage message with `file` and `--port` arguments. + +**Step 2: Verify Python imports** + +```bash +python -c "import lark; import fastapi; import uvicorn; import watchdog; import dfgraph; print('All imports OK')" +``` + +Expected: Prints `All imports OK`. + +**Step 3: Verify frontend builds** + +```bash +cd /home/orual/Projects/or1-design/dfgraph/frontend +npm run build +ls -la dist/bundle.js +``` + +Expected: `dist/bundle.js` exists. + +**Step 4: Run existing tests to ensure no regressions** + +```bash +cd /home/orual/Projects/or1-design +python -m pytest tests/ -v +``` + +Expected: All existing tests pass. + diff --git a/docs/implementation-plans/2026-02-23-dataflow-renderer/phase_02.md b/docs/implementation-plans/2026-02-23-dataflow-renderer/phase_02.md new file mode 100644 index 0000000..eb96659 --- /dev/null +++ b/docs/implementation-plans/2026-02-23-dataflow-renderer/phase_02.md @@ -0,0 +1,344 @@ +# Dataflow Graph Renderer Implementation Plan — Phase 2: Progressive Pipeline Runner + +**Goal:** Build the pipeline module that runs assembler passes individually with error capture, and the opcode-to-category mapping for visual colouring. + +**Architecture:** `dfgraph/pipeline.py` wraps the existing `lower → resolve → place → allocate` pipeline but catches errors at each stage instead of raising. `dfgraph/categories.py` maps each opcode to a visual category using isinstance dispatch on the ALUOp/MemOp/CfgOp hierarchy. + +**Tech Stack:** Python 3.12, existing asm/ package internals + +**Scope:** 8 phases from original design (this is phase 2 of 8) + +**Codebase verified:** 2026-02-23 + +--- + +## Acceptance Criteria Coverage + +This phase implements and tests: + +### dataflow-renderer.AC2: Progressive pipeline produces partial graphs +- **dataflow-renderer.AC2.1 Success:** Clean source produces allocate-stage graph with zero errors +- **dataflow-renderer.AC2.2 Success:** Source with errors at any stage still returns a graph with valid nodes + +### dataflow-renderer.AC5: Error display (partial — error data production) +- **dataflow-renderer.AC5.1 Success:** Nodes/edges with errors render with red highlight (this phase provides the error data; frontend styling is Phase 7) +- **dataflow-renderer.AC5.2 Success:** Error panel shows error line, category, and message (this phase provides error data with line, category, message fields) +- **dataflow-renderer.AC5.3 Success:** Pipeline errors include suggestions when available (this phase preserves suggestion data from AssemblyError) + +--- + + + +### Task 1: Create dfgraph/pipeline.py — progressive pipeline runner + +**Verifies:** dataflow-renderer.AC2.1, dataflow-renderer.AC2.2, dataflow-renderer.AC5.1, dataflow-renderer.AC5.2, dataflow-renderer.AC5.3 + +**Files:** +- Create: `/home/orual/Projects/or1-design/dfgraph/pipeline.py` + +**Implementation:** + +The progressive pipeline runner calls each assembler pass individually, capturing the `IRGraph` at the deepest successful stage. Unlike `asm._run_pipeline()` which raises `ValueError` on first error, this function always returns a result. + +Key design decisions based on codebase investigation: +- `lower()` takes a Lark parse tree, not raw source. The pipeline module needs to handle parsing internally using `asm._get_parser()` or creating its own parser. +- Each pass (`resolve`, `place`, `allocate`) may add errors to `graph.errors` but still returns a valid `IRGraph`. The existing passes accumulate errors — they don't raise. +- `_run_pipeline()` in `asm/__init__.py` checks `graph.errors` after each pass and raises if non-empty. Our progressive runner skips those checks and continues. +- **Important**: The parse step can raise `lark.exceptions.UnexpectedInput` (or subclasses) if the source has syntax errors — this is the only stage that truly fails rather than accumulating errors. + +Create a `PipelineStage` enum and a `PipelineResult` dataclass: + +```python +"""Progressive pipeline runner for dfasm assembly. + +Runs assembler passes individually, capturing the deepest successful IRGraph +even when later passes fail. This enables partial graph visualisation. +""" + +from __future__ import annotations + +from dataclasses import dataclass +from enum import Enum +from typing import Optional + +from lark import Lark +from pathlib import Path + +from asm.ir import IRGraph +from asm.lower import lower +from asm.resolve import resolve +from asm.place import place +from asm.allocate import allocate +from asm.errors import AssemblyError + + +_GRAMMAR_PATH = Path(__file__).parent.parent / "dfasm.lark" +_parser: Optional[Lark] = None + + +def _get_parser() -> Lark: + global _parser + if _parser is None: + _parser = Lark( + _GRAMMAR_PATH.read_text(), + parser="earley", + propagate_positions=True, + ) + return _parser + + +class PipelineStage(Enum): + PARSE_ERROR = "parse_error" + LOWER = "lower" + RESOLVE = "resolve" + PLACE = "place" + ALLOCATE = "allocate" + + +@dataclass(frozen=True) +class PipelineResult: + graph: Optional[IRGraph] + stage: PipelineStage + errors: list[AssemblyError] + parse_error: Optional[str] = None + + +def run_progressive(source: str) -> PipelineResult: + try: + tree = _get_parser().parse(source) + except Exception as exc: + return PipelineResult( + graph=None, + stage=PipelineStage.PARSE_ERROR, + errors=[], + parse_error=str(exc), + ) + + graph = lower(tree) + stage = PipelineStage.LOWER + + # lower → resolve runs unconditionally (matches asm._run_pipeline behaviour). + # resolve() handles structurally incomplete graphs gracefully. + graph = resolve(graph) + stage = PipelineStage.RESOLVE + + if not graph.errors: + graph = place(graph) + stage = PipelineStage.PLACE + + if not graph.errors: + graph = allocate(graph) + stage = PipelineStage.ALLOCATE + + return PipelineResult( + graph=graph, + stage=stage, + errors=list(graph.errors), + ) +``` + +**Note on error-stop behaviour**: The actual `_run_pipeline()` in `asm/__init__.py:52-54` passes `lower()` output to `resolve()` unconditionally (no error check between them). We match this behaviour. Error checks happen after `resolve()`, `place()`, and `allocate()` — if any stage accumulates errors, subsequent stages are skipped. + +**Verification:** + +```bash +cd /home/orual/Projects/or1-design +python -c "from dfgraph.pipeline import run_progressive, PipelineStage; print('OK')" +``` + +Expected: Prints `OK`. + +**Commit:** `feat: add progressive pipeline runner for dfgraph` + + + +### Task 2: Tests for progressive pipeline runner + +**Verifies:** dataflow-renderer.AC2.1, dataflow-renderer.AC2.2, dataflow-renderer.AC5.2, dataflow-renderer.AC5.3 + +**Files:** +- Create: `/home/orual/Projects/or1-design/tests/test_dfgraph_pipeline.py` + +**Testing:** + +Tests must verify each AC listed above: +- **dataflow-renderer.AC2.1:** Clean dfasm source (with @system pragma, function, nodes, edges) produces `PipelineStage.ALLOCATE` with zero errors +- **dataflow-renderer.AC2.2:** Source with undefined name references produces a graph with valid nodes at `PipelineStage.RESOLVE` stage (resolve runs, accumulates NAME errors, pipeline stops before place) +- **dataflow-renderer.AC2.2:** Source with placement errors (e.g., too many nodes for PE count) produces a graph at `PipelineStage.PLACE` stage (place runs, accumulates PLACEMENT errors, pipeline stops before allocate) +- **dataflow-renderer.AC5.2:** Error results contain `AssemblyError` objects with `loc.line`, `category`, and `message` fields populated +- **dataflow-renderer.AC5.3:** Name resolution errors include suggestions (Levenshtein "did you mean" suggestions from resolve pass) +- **Parse error case:** Syntactically invalid source returns `PipelineStage.PARSE_ERROR` with `graph=None` and `parse_error` string populated + +Follow existing test patterns: +- Use pytest classes to group tests (e.g., `TestProgressivePipeline`) +- Document AC references in docstrings +- Use the Lark parser fixture from `tests/conftest.py` if needed, or rely on `run_progressive` which handles parsing internally +- For valid dfasm source examples, copy working programs from `tests/test_lower.py` and `tests/test_e2e.py` — do NOT invent new dfasm syntax. The grammar (`dfasm.lark`) has specific rules for pragmas, function blocks, node declarations, and edge syntax that are easy to get wrong. + +**Verification:** + +```bash +cd /home/orual/Projects/or1-design +python -m pytest tests/test_dfgraph_pipeline.py -v +``` + +Expected: All tests pass. + +**Commit:** `test: add tests for progressive pipeline runner` + + + + + +### Task 3: Create dfgraph/categories.py — opcode-to-category mapping + +**Verifies:** (supports dataflow-renderer.AC1.3 in Phase 5) + +**Files:** +- Create: `/home/orual/Projects/or1-design/dfgraph/categories.py` + +**Implementation:** + +Map each opcode to a visual category for the graph renderer. The design specifies these categories with colours: + +| Category | Colour | Opcodes | +|----------|--------|---------| +| arithmetic | #4a90d9 (blue) | ArithOp.ADD, SUB, INC, DEC, SHIFT_L, SHIFT_R, ASHFT_R | +| logic | #4caf50 (green) | LogicOp.AND, OR, XOR, NOT | +| comparison | #ff9800 (amber) | LogicOp.EQ, LT, LTE, GT, GTE | +| routing | #9c27b0 (purple) | RoutingOp.* EXCEPT CONST/FREE_CTX (BREQ, BRGT, BRGE, BROF, SWEQ, SWGT, SWGE, SWOF, GATE, PASS, SEL, MRGE) | +| memory | #ff5722 (orange) | MemOp.* (READ, WRITE, CLEAR, ALLOC, FREE, RD_INC, RD_DEC, CMP_SW) | +| config | #9e9e9e (grey) | CfgOp.* (LOAD_INST, ROUTE_SET) + RoutingOp.CONST, RoutingOp.FREE_CTX | + +**Important codebase finding**: `LogicOp` contains both pure logic ops (AND, OR, XOR, NOT) and comparison ops (EQ, LT, LTE, GT, GTE). The category mapping must split these within the same enum class by checking specific enum values. + +**Note**: The design mentions an "I/O" category (ior, iow, iorw = teal), but these opcodes do NOT exist in `asm/opcodes.py:MNEMONIC_TO_OP`. They are in the grammar and design notes but not implemented in the assembler. The I/O category should be defined in the mapping but won't be tested against `MNEMONIC_TO_OP` until those ops are added. + +**Note on IntEnum collision handling**: The existing codebase uses `TypeAwareOpToMnemonicDict` in `asm/opcodes.py` because `ArithOp.ADD == 0 == MemOp.READ` (IntEnum value collisions). The category mapping should use `isinstance` dispatch on the type hierarchy rather than plain dict lookup, which avoids this problem entirely. + +```python +"""Opcode-to-category mapping for visual graph rendering. + +Maps each ALUOp/MemOp/CfgOp to a visual category and colour +for the dataflow graph renderer. +""" + +from __future__ import annotations + +from enum import Enum +from typing import Union + +from cm_inst import ArithOp, CfgOp, LogicOp, MemOp, RoutingOp + + +class OpcodeCategory(Enum): + ARITHMETIC = "arithmetic" + LOGIC = "logic" + COMPARISON = "comparison" + ROUTING = "routing" + MEMORY = "memory" + IO = "io" # reserved for future I/O ops (ior, iow, iorw) — not yet in asm/opcodes.py + CONFIG = "config" + + +CATEGORY_COLOURS: dict[OpcodeCategory, str] = { + OpcodeCategory.ARITHMETIC: "#4a90d9", + OpcodeCategory.LOGIC: "#4caf50", + OpcodeCategory.COMPARISON: "#ff9800", + OpcodeCategory.ROUTING: "#9c27b0", + OpcodeCategory.MEMORY: "#ff5722", + OpcodeCategory.IO: "#009688", + OpcodeCategory.CONFIG: "#9e9e9e", +} + + +_COMPARISON_OPS: frozenset[LogicOp] = frozenset({ + LogicOp.EQ, LogicOp.LT, LogicOp.LTE, LogicOp.GT, LogicOp.GTE, +}) + +_CONFIG_ROUTING_OPS: frozenset[RoutingOp] = frozenset({ + RoutingOp.CONST, RoutingOp.FREE_CTX, +}) + + +def categorise(op: Union[ArithOp, LogicOp, RoutingOp, MemOp, CfgOp]) -> OpcodeCategory: + if isinstance(op, ArithOp): + return OpcodeCategory.ARITHMETIC + if isinstance(op, LogicOp): + if op in _COMPARISON_OPS: + return OpcodeCategory.COMPARISON + return OpcodeCategory.LOGIC + if isinstance(op, RoutingOp): + if op in _CONFIG_ROUTING_OPS: + return OpcodeCategory.CONFIG + return OpcodeCategory.ROUTING + if isinstance(op, MemOp): + return OpcodeCategory.MEMORY + if isinstance(op, CfgOp): + return OpcodeCategory.CONFIG + raise ValueError(f"Unknown opcode type: {type(op).__name__}") +``` + +**Verification:** + +```bash +cd /home/orual/Projects/or1-design +python -c "from dfgraph.categories import categorise, OpcodeCategory; from cm_inst import ArithOp, LogicOp; print(categorise(ArithOp.ADD), categorise(LogicOp.EQ))" +``` + +Expected: `OpcodeCategory.ARITHMETIC OpcodeCategory.COMPARISON` + +**Commit:** `feat: add opcode-to-category mapping for dfgraph` + + + +### Task 4: Tests for opcode category mapping + +**Verifies:** (supports dataflow-renderer.AC1.3) + +**Files:** +- Create: `/home/orual/Projects/or1-design/tests/test_dfgraph_categories.py` + +**Testing:** + +Tests must verify: +- Every opcode in `MNEMONIC_TO_OP` (from `asm/opcodes.py`) has a category assignment via `categorise()` — iterate the mapping and assert no `ValueError` is raised +- ArithOp members map to `OpcodeCategory.ARITHMETIC` +- LogicOp.AND, OR, XOR, NOT map to `OpcodeCategory.LOGIC` +- LogicOp.EQ, LT, LTE, GT, GTE map to `OpcodeCategory.COMPARISON` +- RoutingOp members (except CONST and FREE_CTX) map to `OpcodeCategory.ROUTING`; RoutingOp.CONST and RoutingOp.FREE_CTX map to `OpcodeCategory.CONFIG` +- MemOp members map to `OpcodeCategory.MEMORY` +- CfgOp members map to `OpcodeCategory.CONFIG` +- Every `OpcodeCategory` has a colour in `CATEGORY_COLOURS` + +Follow existing test patterns: +- Use pytest parametrize for iterating over `MNEMONIC_TO_OP` items +- Reference the `TypeAwareOpToMnemonicDict.items()` method for collision-safe iteration + +**Verification:** + +```bash +cd /home/orual/Projects/or1-design +python -m pytest tests/test_dfgraph_categories.py -v +``` + +Expected: All tests pass. + +**Commit:** `test: add tests for opcode category mapping` + + + + +### Task 5: Run full test suite to verify no regressions + +**Files:** None (verification only) + +**Step 1: Run all tests** + +```bash +cd /home/orual/Projects/or1-design +python -m pytest tests/ -v +``` + +Expected: All tests pass, including the new `test_dfgraph_pipeline.py` and `test_dfgraph_categories.py`. + diff --git a/docs/implementation-plans/2026-02-23-dataflow-renderer/phase_03.md b/docs/implementation-plans/2026-02-23-dataflow-renderer/phase_03.md new file mode 100644 index 0000000..6fbed84 --- /dev/null +++ b/docs/implementation-plans/2026-02-23-dataflow-renderer/phase_03.md @@ -0,0 +1,290 @@ +# Dataflow Graph Renderer Implementation Plan — Phase 3: IR-to-JSON Conversion + +**Goal:** Convert an `IRGraph` at any pipeline stage into a JSON-serialisable structure for the frontend. + +**Architecture:** `dfgraph/graph_json.py` traverses the IRGraph (including nested regions) and produces a typed dict structure containing nodes, edges, regions, errors, and metadata. Uses `collect_all_nodes_and_edges()` from `asm/ir.py` for flattening. Uses `categorise()` from `dfgraph/categories.py` for opcode colour mapping. Handles both pre-allocation graphs (no PE/IRAM data) and post-allocation graphs (full ResolvedDest/Addr data). + +**Tech Stack:** Python 3.12, existing asm/ir types, dfgraph/pipeline and dfgraph/categories from Phase 2 + +**Scope:** 8 phases from original design (this is phase 3 of 8) + +**Codebase verified:** 2026-02-23 + +--- + +## Acceptance Criteria Coverage + +This phase implements and tests: + +### dataflow-renderer.AC1: Logical view renders dataflow graph (partial — data production) +- **dataflow-renderer.AC1.1 Success:** Valid dfasm source renders as a directed graph with nodes and edges (this phase produces the JSON data that the frontend will render) + +--- + + + +### Task 1: Create dfgraph/graph_json.py — IRGraph to JSON conversion + +**Verifies:** dataflow-renderer.AC1.1 + +**Files:** +- Create: `/home/orual/Projects/or1-design/dfgraph/graph_json.py` + +**Implementation:** + +Convert an `IRGraph` and `PipelineResult` into a JSON-serialisable dict for WebSocket transmission to the frontend. + +Key design decisions based on codebase investigation: +- `IRGraph.nodes` is `dict[str, IRNode]` and `IRGraph.edges` is `list[IREdge]` +- Nodes and edges can be nested inside `IRRegion.body` (which is itself an `IRGraph`). Use `collect_all_nodes_and_edges()` from `asm/ir.py` to flatten. +- `IRNode.opcode` is `Union[ALUOp, MemOp]` — use `categorise()` from `dfgraph/categories.py` for the category. Note: `CfgOp` is NOT an IRNode opcode (config ops are system-level, not graph nodes). +- `IRNode.dest_l` / `dest_r` are `Optional[Union[NameRef, ResolvedDest]]`. Before allocation: `NameRef(name, port)`. After allocation: `ResolvedDest(name, addr)` where `addr` is `Addr(a, port, pe)`. +- `IREdge` has `source`, `dest`, `port` (Port enum: L=0, R=1), `source_port` (Optional[Port]). +- `IRRegion` has `tag`, `kind` (RegionKind: FUNCTION or LOCATION), `body` (IRGraph). +- `AssemblyError` has `loc` (SourceLoc with line, column), `category` (ErrorCategory enum), `message` (str), `suggestions` (list[str]). +- Nodes involved in errors are identified by matching `AssemblyError.loc.line` against `IRNode.loc.line`. This is reliable because each node declaration occupies a distinct source line. Do NOT use substring matching on error messages — node names like `&a` would false-positive match any error mentioning words containing "a". +- Use `OP_TO_MNEMONIC` from `asm/opcodes.py` for mnemonic labels (collision-safe via TypeAwareOpToMnemonicDict). + +```python +"""Convert IRGraph to JSON-serialisable structure for the frontend. + +Produces a flat graph representation with all nodes, edges, regions, +errors, and metadata needed for both logical and physical views. +""" + +from __future__ import annotations + +from typing import Any, Union + +from cm_inst import Addr, Port +from asm.ir import ( + IRGraph, IRNode, IREdge, IRRegion, RegionKind, + SourceLoc, NameRef, ResolvedDest, + collect_all_nodes_and_edges, +) +from asm.errors import AssemblyError +from asm.opcodes import OP_TO_MNEMONIC +from dfgraph.pipeline import PipelineResult, PipelineStage +from dfgraph.categories import categorise, CATEGORY_COLOURS, OpcodeCategory + + +def _serialise_loc(loc: SourceLoc) -> dict[str, Any]: + return { + "line": loc.line, + "column": loc.column, + "end_line": loc.end_line, + "end_column": loc.end_column, + } + + +def _serialise_addr(addr: Addr) -> dict[str, Any]: + return { + "offset": addr.a, + "port": addr.port.name, + "pe": addr.pe, + } + + +def _serialise_node(node: IRNode, error_node_names: set[str]) -> dict[str, Any]: + category = categorise(node.opcode) + mnemonic = OP_TO_MNEMONIC[node.opcode] + + result: dict[str, Any] = { + "id": node.name, + "opcode": mnemonic, + "category": category.value, + "colour": CATEGORY_COLOURS[category], + "const": node.const, + "pe": node.pe, + "iram_offset": node.iram_offset, + "ctx": node.ctx, + "has_error": node.name in error_node_names, + "loc": _serialise_loc(node.loc), + } + + return result + + +def _serialise_edge(edge: IREdge, all_nodes: dict[str, IRNode], + error_lines: set[int]) -> dict[str, Any]: + result: dict[str, Any] = { + "source": edge.source, + "target": edge.dest, + "port": edge.port.name, + "source_port": edge.source_port.name if edge.source_port else None, + "has_error": edge.loc.line in error_lines, + } + + source_node = all_nodes.get(edge.source) + if source_node: + if (isinstance(source_node.dest_l, ResolvedDest) + and source_node.dest_l.name == edge.dest): + result["addr"] = _serialise_addr(source_node.dest_l.addr) + elif (isinstance(source_node.dest_r, ResolvedDest) + and source_node.dest_r.name == edge.dest): + result["addr"] = _serialise_addr(source_node.dest_r.addr) + + return result + + +def _serialise_error(error: AssemblyError) -> dict[str, Any]: + return { + "line": error.loc.line, + "column": error.loc.column, + "category": error.category.value, + "message": error.message, + "suggestions": error.suggestions, + } + + +def _serialise_region(region: IRRegion) -> dict[str, Any]: + node_ids = list(region.body.nodes.keys()) + for sub_region in region.body.regions: + node_ids.extend(sub_region.body.nodes.keys()) + + return { + "tag": region.tag, + "kind": region.kind.value, + "node_ids": node_ids, + } + + +def _collect_error_node_names(errors: list[AssemblyError], + all_nodes: dict[str, IRNode]) -> set[str]: + error_lines: set[int] = {e.loc.line for e in errors} + return { + name for name, node in all_nodes.items() + if node.loc.line in error_lines + } + + +def graph_to_json(result: PipelineResult) -> dict[str, Any]: + if result.graph is None: + return { + "type": "graph_update", + "stage": result.stage.value, + "nodes": [], + "edges": [], + "regions": [], + "errors": [], + "parse_error": result.parse_error, + "metadata": { + "stage": result.stage.value, + "pe_count": 0, + "sm_count": 0, + }, + } + + graph = result.graph + all_nodes, all_edges = collect_all_nodes_and_edges(graph) + error_lines: set[int] = {e.loc.line for e in result.errors} + error_node_names = _collect_error_node_names(result.errors, all_nodes) + + nodes_json = [ + _serialise_node(node, error_node_names) + for node in all_nodes.values() + ] + + edges_json = [ + _serialise_edge(edge, all_nodes, error_lines) + for edge in all_edges + ] + + regions_json = [] + for subgraph_regions in [graph.regions]: + for region in subgraph_regions: + if region.kind == RegionKind.FUNCTION: + regions_json.append(_serialise_region(region)) + + errors_json = [_serialise_error(e) for e in result.errors] + + pe_count = graph.system.pe_count if graph.system else 0 + sm_count = graph.system.sm_count if graph.system else 0 + + return { + "type": "graph_update", + "stage": result.stage.value, + "nodes": nodes_json, + "edges": edges_json, + "regions": regions_json, + "errors": errors_json, + "parse_error": None, + "metadata": { + "stage": result.stage.value, + "pe_count": pe_count, + "sm_count": sm_count, + }, + } +``` + +**Verification:** + +```bash +cd /home/orual/Projects/or1-design +python -c "from dfgraph.graph_json import graph_to_json; print('OK')" +``` + +Expected: Prints `OK`. + +**Commit:** `feat: add IR-to-JSON conversion for dfgraph` + + + +### Task 2: Tests for IR-to-JSON conversion + +**Verifies:** dataflow-renderer.AC1.1 + +**Files:** +- Create: `/home/orual/Projects/or1-design/tests/test_dfgraph_json.py` + +**Testing:** + +Tests must verify the AC listed above: +- **dataflow-renderer.AC1.1 (fully allocated graph):** Run `run_progressive()` on a clean dfasm program (one with @system, function, nodes, edges — use examples from `tests/test_e2e.py`). Call `graph_to_json()` on the result. Assert: + - `stage` is `"allocate"` + - `nodes` list is non-empty, each node has `id`, `opcode`, `category`, `colour`, `pe` (not None), `iram_offset` (not None), `ctx` (not None) + - `edges` list is non-empty, each edge has `source`, `target`, `port` + - `errors` list is empty + - `metadata.pe_count` and `metadata.sm_count` match @system pragma values + +- **dataflow-renderer.AC1.1 (partially resolved graph):** Run `run_progressive()` on source with undefined name references. Assert: + - `stage` is `"lower"` (errors stop at resolve) + - `nodes` list contains valid nodes from lowering + - `pe`, `iram_offset`, `ctx` are all `None` (not yet allocated) + - `errors` list is non-empty + +- **Error nodes flagged:** Run on source with errors. Assert nodes whose source line matches an error's `loc.line` have `has_error: true`. + +- **Function regions:** Run on source with `$func { ... }` blocks. Assert `regions` list contains entries with `tag`, `kind: "function"`, and `node_ids` listing child node names. + +- **Parse error case:** Call `graph_to_json()` on a `PipelineResult` with `stage=PARSE_ERROR` and `graph=None`. Assert the JSON has empty nodes/edges and a `parse_error` string. + +Follow existing test patterns from `tests/test_e2e.py` and `tests/test_lower.py` for valid dfasm source examples. + +**Verification:** + +```bash +cd /home/orual/Projects/or1-design +python -m pytest tests/test_dfgraph_json.py -v +``` + +Expected: All tests pass. + +**Commit:** `test: add tests for IR-to-JSON conversion` + + + + +### Task 3: Run full test suite to verify no regressions + +**Files:** None (verification only) + +**Step 1: Run all tests** + +```bash +cd /home/orual/Projects/or1-design +python -m pytest tests/ -v +``` + +Expected: All tests pass. + diff --git a/docs/implementation-plans/2026-02-23-dataflow-renderer/phase_04.md b/docs/implementation-plans/2026-02-23-dataflow-renderer/phase_04.md new file mode 100644 index 0000000..92478af --- /dev/null +++ b/docs/implementation-plans/2026-02-23-dataflow-renderer/phase_04.md @@ -0,0 +1,299 @@ +# Dataflow Graph Renderer Implementation Plan — Phase 4: Backend Server with WebSocket + +**Goal:** FastAPI application that serves the frontend and pushes graph updates over WebSocket when the source file changes. + +**Architecture:** `dfgraph/server.py` creates a FastAPI app with: (1) static file serving for the frontend, (2) a WebSocket endpoint that broadcasts graph JSON to all connected clients, (3) a watchdog file observer with 300ms debounce that re-assembles the source and pushes updates. `dfgraph/__main__.py` is expanded to start uvicorn and open the browser. + +**Tech Stack:** Python 3.12, FastAPI 0.121.x, uvicorn, watchdog 6.x, websockets + +**Scope:** 8 phases from original design (this is phase 4 of 8) + +**Codebase verified:** 2026-02-23 + +--- + +## Acceptance Criteria Coverage + +This phase implements and tests: + +### dataflow-renderer.AC4: Live reload on file change +- **dataflow-renderer.AC4.1 Success:** Saving the dfasm file triggers re-render within 1 second +- **dataflow-renderer.AC4.2 Success:** Initial load renders the graph without manual refresh + +--- + + + +### Task 1: Create dfgraph/server.py — FastAPI app with WebSocket and file watcher + +**Verifies:** dataflow-renderer.AC4.1, dataflow-renderer.AC4.2 + +**Files:** +- Create: `/home/orual/Projects/or1-design/dfgraph/server.py` + +**Implementation:** + +The server module creates a FastAPI app with three responsibilities: +1. Serve the frontend static files (index.html, dist/bundle.js) +2. WebSocket endpoint at `/ws` that sends current graph JSON on connect and on file changes +3. File watcher that monitors the .dfasm source file and triggers re-assembly with 300ms debounce + +Key design decisions based on research: +- Use `ConnectionManager` pattern for broadcasting to multiple WebSocket clients +- Use `StaticFiles(directory=..., html=True)` for serving index.html at root +- Use watchdog `Observer` in a background thread with `threading.Timer` for debounce +- The file watcher callback needs to run `run_progressive()` and `graph_to_json()`, then broadcast. Since watchdog callbacks run in a thread, use `asyncio.run_coroutine_threadsafe()` to schedule the broadcast on the FastAPI event loop. +- Use the `lifespan` async context manager (not the deprecated `@app.on_event("startup")`) for startup/shutdown. The lifespan teardown stops the watchdog observer cleanly. +- Mount static files AFTER defining the WebSocket route (mount order matters in FastAPI). + +```python +"""FastAPI server for the dataflow graph renderer. + +Serves the frontend static files and pushes graph updates over WebSocket +when the source dfasm file changes. +""" + +from __future__ import annotations + +import asyncio +import json +import os +import threading +import time +from pathlib import Path +from typing import Optional + +from contextlib import asynccontextmanager + +from fastapi import FastAPI, WebSocket, WebSocketDisconnect +from fastapi.staticfiles import StaticFiles +from watchdog.observers import Observer +from watchdog.events import FileSystemEventHandler + +from dfgraph.pipeline import run_progressive, PipelineResult +from dfgraph.graph_json import graph_to_json + + +class ConnectionManager: + def __init__(self) -> None: + self.active_connections: list[WebSocket] = [] + + async def connect(self, websocket: WebSocket) -> None: + await websocket.accept() + self.active_connections.append(websocket) + + def disconnect(self, websocket: WebSocket) -> None: + self.active_connections.remove(websocket) + + async def broadcast(self, message: dict) -> None: + disconnected: list[WebSocket] = [] + for connection in self.active_connections: + try: + await connection.send_json(message) + except Exception: + disconnected.append(connection) + for conn in disconnected: + self.active_connections.remove(conn) + + +class DebouncedFileHandler(FileSystemEventHandler): + def __init__(self, target_path: str, callback, debounce_s: float = 0.3) -> None: + self.target_path = os.path.realpath(target_path) + self.callback = callback + self.debounce_s = debounce_s + self._timer: Optional[threading.Timer] = None + + def on_modified(self, event) -> None: + if event.is_directory: + return + if os.path.realpath(event.src_path) != self.target_path: + return + if self._timer is not None: + self._timer.cancel() + self._timer = threading.Timer(self.debounce_s, self.callback) + self._timer.daemon = True + self._timer.start() + + +def create_app(source_path: Path) -> FastAPI: + manager = ConnectionManager() + current_json: dict = {} + loop: Optional[asyncio.AbstractEventLoop] = None + + def _reassemble() -> dict: + source = source_path.read_text() + result = run_progressive(source) + return graph_to_json(result) + + def _on_file_change() -> None: + nonlocal current_json + current_json = _reassemble() + if loop is not None: + asyncio.run_coroutine_threadsafe( + manager.broadcast(current_json), loop + ) + + @asynccontextmanager + async def lifespan(app: FastAPI): + nonlocal current_json, loop + loop = asyncio.get_event_loop() + current_json = _reassemble() + + handler = DebouncedFileHandler( + str(source_path), _on_file_change, debounce_s=0.3 + ) + observer = Observer() + observer.schedule(handler, str(source_path.parent), recursive=False) + observer.daemon = True + observer.start() + + yield + + observer.stop() + observer.join(timeout=2) + + app = FastAPI(lifespan=lifespan) + + @app.websocket("/ws") + async def websocket_endpoint(websocket: WebSocket) -> None: + await manager.connect(websocket) + try: + await websocket.send_json(current_json) + while True: + await websocket.receive_text() + except WebSocketDisconnect: + manager.disconnect(websocket) + + frontend_dir = Path(__file__).parent / "frontend" + app.mount( + "/dist", + StaticFiles(directory=str(frontend_dir / "dist")), + name="dist", + ) + app.mount( + "/", + StaticFiles(directory=str(frontend_dir), html=True), + name="frontend", + ) + + return app +``` + +**Verification:** + +```bash +cd /home/orual/Projects/or1-design +python -c "from dfgraph.server import create_app; print('OK')" +``` + +Expected: Prints `OK`. + +**Commit:** `feat: add FastAPI server with WebSocket and file watcher` + + + +### Task 2: Expand dfgraph/__main__.py to start the server + +**Verifies:** dataflow-renderer.AC4.2 + +**Files:** +- Modify: `/home/orual/Projects/or1-design/dfgraph/__main__.py` (created in Phase 1, Task 2) + +**Implementation:** + +Replace the placeholder `print()` statement with actual server startup. After parsing arguments, create the app, start uvicorn, and open the browser. + +Key decisions: +- Use `uvicorn.run(app, ...)` with the app instance directly (no reload needed for this use case) +- Open browser in a daemon thread with a 1-second delay to give the server time to start +- Bind to `127.0.0.1` (localhost only, not exposed to network) + +Replace the body of `main()` after argument validation with: + +```python +import threading +import webbrowser +import uvicorn +from dfgraph.server import create_app + +app = create_app(args.file.resolve()) + +def open_browser(): + import time + time.sleep(1) + webbrowser.open(f"http://127.0.0.1:{args.port}", new=2) + +thread = threading.Thread(target=open_browser, daemon=True) +thread.start() + +uvicorn.run(app, host="127.0.0.1", port=args.port, log_level="info") +``` + +The full file should have the existing argparse setup from Phase 1, with the placeholder print replaced by this startup code. + +**Verification:** + +```bash +cd /home/orual/Projects/or1-design +python -m dfgraph --help +``` + +Expected: Still shows usage with `file` and `--port` arguments. + +**Commit:** `feat: wire up dfgraph CLI to start server` + + + +### Task 3: Tests for server WebSocket and file watcher + +**Verifies:** dataflow-renderer.AC4.1, dataflow-renderer.AC4.2 + +**Files:** +- Create: `/home/orual/Projects/or1-design/tests/test_dfgraph_server.py` + +**Testing:** + +Tests must verify each AC listed above: +- **dataflow-renderer.AC4.2 (initial load):** Use FastAPI's `TestClient` (from `starlette.testclient`). Create a temporary `.dfasm` file with valid source. Create the app via `create_app()`. Connect to `/ws` via the test client's WebSocket support. Assert the first message received is a valid graph JSON with `type: "graph_update"` and non-empty `nodes`. + +- **dataflow-renderer.AC4.1 (live reload):** Create a temporary `.dfasm` file. Start the app. Connect via WebSocket. Receive initial graph. Modify the temporary file on disk (write new content). Wait up to 2 seconds. Assert a second `graph_update` message is received with updated content. + +- **HTTP serving:** Use `TestClient` to `GET /`. Assert response status is 200 and content contains "dfgraph" (from the HTML page title). This requires the frontend `index.html` to exist — the test should create a minimal temporary frontend directory structure or skip if the frontend hasn't been built yet. + +- **Debounce:** Modify the file rapidly (3 times in 100ms). Assert only 1 update is received (within a 2-second window), not 3. + +Implementation notes: +- Use `tmp_path` pytest fixture for temporary dfasm files +- FastAPI's TestClient from starlette supports WebSocket testing via `with client.websocket_connect("/ws") as ws:` +- The file watcher tests may need `time.sleep()` for timing — keep assertions generous (within 2 seconds) to avoid flaky tests +- For the HTTP test, you may need to create a minimal temporary frontend directory with an `index.html` + +Reference: See existing tests in `tests/test_e2e.py` for valid dfasm source strings to use as test fixtures. + +**Verification:** + +```bash +cd /home/orual/Projects/or1-design +python -m pytest tests/test_dfgraph_server.py -v +``` + +Expected: All tests pass. + +**Commit:** `test: add tests for dfgraph server` + + + + +### Task 4: Run full test suite to verify no regressions + +**Files:** None (verification only) + +**Step 1: Run all tests** + +```bash +cd /home/orual/Projects/or1-design +python -m pytest tests/ -v +``` + +Expected: All tests pass. + diff --git a/docs/implementation-plans/2026-02-23-dataflow-renderer/phase_05.md b/docs/implementation-plans/2026-02-23-dataflow-renderer/phase_05.md new file mode 100644 index 0000000..a5ac443 --- /dev/null +++ b/docs/implementation-plans/2026-02-23-dataflow-renderer/phase_05.md @@ -0,0 +1,481 @@ +# Dataflow Graph Renderer Implementation Plan — Phase 5: Frontend — Logical View + +**Goal:** Cytoscape.js rendering of the logical dataflow graph with coloured circular nodes, edge annotations, and function region boxes. + +**Architecture:** Three TypeScript modules: `main.ts` (WebSocket client, cytoscape init, graph updates), `style.ts` (stylesheet definitions), `layout.ts` (dagre layout config). The frontend receives `graph_update` JSON from the backend WebSocket and renders it using cytoscape.js with the dagre hierarchical layout. + +**Tech Stack:** TypeScript, cytoscape.js 3.30+, cytoscape-dagre 2.5+, esbuild + +**Scope:** 8 phases from original design (this is phase 5 of 8) + +**Codebase verified:** 2026-02-23 + +--- + +## Acceptance Criteria Coverage + +This phase implements and tests: + +### dataflow-renderer.AC1: Logical view renders dataflow graph +- **dataflow-renderer.AC1.1 Success:** Valid dfasm source renders as a directed graph with nodes and edges +- **dataflow-renderer.AC1.2 Success:** Nodes are circular with opcode mnemonic labels centred inside +- **dataflow-renderer.AC1.3 Success:** Nodes are coloured by opcode category (arithmetic=blue, logic=green, comparison=amber, routing=purple, memory=orange, io=teal, config=grey) +- **dataflow-renderer.AC1.4 Success:** Edges show port annotations (L/R) at target end and branch labels (T/F) at source end for routing ops +- **dataflow-renderer.AC1.5 Success:** Function regions render as dashed bounding boxes around their child nodes + +--- + + +### Task 1: Create dfgraph/frontend/src/types.ts — shared type definitions + +**Verifies:** (supports all AC1.x — defines the JSON contract) + +**Files:** +- Create: `/home/orual/Projects/or1-design/dfgraph/frontend/src/types.ts` + +**Implementation:** + +Define TypeScript interfaces matching the JSON structure produced by `dfgraph/graph_json.py` (Phase 3). These types are used by all frontend modules. + +```typescript +export interface GraphNode { + id: string; + opcode: string; + category: string; + colour: string; + const: number | null; + pe: number | null; + iram_offset: number | null; + ctx: number | null; + has_error: boolean; + loc: SourceLoc; +} + +export interface SourceLoc { + line: number; + column: number; + end_line: number | null; + end_column: number | null; +} + +export interface AddrInfo { + offset: number; + port: string; + pe: number | null; +} + +export interface GraphEdge { + source: string; + target: string; + port: string; + source_port: string | null; + has_error: boolean; + addr?: AddrInfo; +} + +export interface GraphRegion { + tag: string; + kind: string; + node_ids: string[]; +} + +export interface GraphError { + line: number; + column: number; + category: string; + message: string; + suggestions: string[]; +} + +export interface GraphUpdate { + type: "graph_update"; + stage: string; + nodes: GraphNode[]; + edges: GraphEdge[]; + regions: GraphRegion[]; + errors: GraphError[]; + parse_error: string | null; + metadata: { + stage: string; + pe_count: number; + sm_count: number; + }; +} +``` + +**Verification:** + +```bash +cd /home/orual/Projects/or1-design/dfgraph/frontend +npx esbuild src/types.ts --bundle --outfile=/dev/null --format=esm --target=es2020 +``` + +Expected: No type errors. + +**Commit:** `feat: add TypeScript type definitions for graph JSON` + + + +### Task 2: Create dfgraph/frontend/src/style.ts — cytoscape stylesheet + +**Verifies:** dataflow-renderer.AC1.2, dataflow-renderer.AC1.3, dataflow-renderer.AC1.4, dataflow-renderer.AC1.5 + +**Files:** +- Create: `/home/orual/Projects/or1-design/dfgraph/frontend/src/style.ts` + +**Implementation:** + +Export a cytoscape stylesheet array. Key styling rules: + +1. **Nodes** (AC1.2): shape `ellipse`, `width`/`height` sized to label with padding, centered label via `text-valign: center` and `text-halign: center`, `label` from `data(label)`. + +2. **Category colours** (AC1.3): `background-color` from `data(colour)` — the backend sets the colour per node based on opcode category. The colours are: arithmetic=#4a90d9 (blue), logic=#4caf50 (green), comparison=#ff9800 (amber), routing=#9c27b0 (purple), memory=#ff5722 (orange), config=#9e9e9e (grey). + +3. **Edge annotations** (AC1.4): + - Port labels (L/R) at target end via `target-label: data(targetLabel)` with `target-text-offset` + - Branch labels (T/F) at source end for routing ops via `source-label: data(sourceLabel)` with `source-text-offset` + - Constant values shown as node labels (already part of the opcode label) + +4. **Function regions** (AC1.5): compound parent nodes with `$node > node` selector — dashed border, light background, label at top. + +5. **Error styling**: nodes with `.error` class get red dashed border. Edges with `.error` class get red dashed line. + +```typescript +import cytoscape from "cytoscape"; + +export const stylesheet: cytoscape.Stylesheet[] = [ + { + selector: "node", + style: { + shape: "ellipse", + width: "label", + height: "label", + padding: "12px", + "text-valign": "center", + "text-halign": "center", + label: "data(label)", + "font-size": 11, + "font-family": "monospace", + color: "#fff", + "text-outline-width": 0, + "background-color": "data(colour)", + "border-width": 2, + "border-color": "data(colour)", + }, + }, + { + selector: "$node > node", + style: { + shape: "roundrectangle", + "border-style": "dashed", + "border-width": 2, + "border-color": "#888", + "background-color": "rgba(200, 200, 200, 0.08)", + padding: "24px", + "text-valign": "top", + "text-halign": "center", + label: "data(label)", + "font-size": 12, + color: "#666", + }, + }, + { + selector: "edge", + style: { + "curve-style": "bezier", + "target-arrow-shape": "triangle", + "target-arrow-color": "#999", + "line-color": "#999", + width: 2, + "target-label": "data(targetLabel)", + "target-text-offset": 18, + "target-text-margin-y": -10, + "source-label": "data(sourceLabel)", + "source-text-offset": 18, + "source-text-margin-y": -10, + "font-size": 9, + "font-family": "monospace", + color: "#666", + "text-background-color": "#fff", + "text-background-opacity": 0.8, + "text-background-padding": "2px", + }, + }, + { + selector: "node.error", + style: { + "border-style": "dashed", + "border-width": 3, + "border-color": "#e53935", + }, + }, + { + selector: "edge.error", + style: { + "line-style": "dashed", + "line-color": "#e53935", + "target-arrow-color": "#e53935", + }, + }, +]; +``` + +**Verification:** + +```bash +cd /home/orual/Projects/or1-design/dfgraph/frontend +npx esbuild src/style.ts --bundle --outfile=/dev/null --format=esm --target=es2020 +``` + +Expected: No errors. + +**Commit:** `feat: add cytoscape stylesheet for logical view` + + + +### Task 3: Create dfgraph/frontend/src/layout.ts — dagre layout configuration + +**Verifies:** dataflow-renderer.AC1.1, dataflow-renderer.AC1.5 + +**Files:** +- Create: `/home/orual/Projects/or1-design/dfgraph/frontend/src/layout.ts` + +**Implementation:** + +Export a function that returns the dagre layout options. The layout should be top-to-bottom hierarchical with reasonable spacing for small-to-medium graphs (5–50 nodes). + +```typescript +export function logicalLayout(): object { + return { + name: "dagre", + rankDir: "TB", + nodeSep: 60, + rankSep: 80, + edgeSep: 20, + animate: false, + }; +} +``` + +**Verification:** + +```bash +cd /home/orual/Projects/or1-design/dfgraph/frontend +npx esbuild src/layout.ts --bundle --outfile=/dev/null --format=esm --target=es2020 +``` + +Expected: No errors. + +**Commit:** `feat: add dagre layout configuration` + + + +### Task 4: Rewrite dfgraph/frontend/src/main.ts — WebSocket client and graph rendering + +**Verifies:** dataflow-renderer.AC1.1, dataflow-renderer.AC1.2, dataflow-renderer.AC1.3, dataflow-renderer.AC1.4, dataflow-renderer.AC1.5 + +**Files:** +- Modify: `/home/orual/Projects/or1-design/dfgraph/frontend/src/main.ts` (replace Phase 1 placeholder) + +**Implementation:** + +Replace the minimal placeholder from Phase 1 with the full WebSocket client and graph rendering logic. + +Key responsibilities: +1. Initialize cytoscape with the stylesheet and an empty graph +2. Connect to WebSocket at `ws://${location.host}/ws` +3. On `graph_update` message, convert JSON to cytoscape elements and re-render +4. Handle reconnection on WebSocket close + +Converting backend JSON to cytoscape elements: +- Each `GraphNode` becomes a cytoscape node element with `data: { id, label, colour, category, ... }` +- The label should include the opcode mnemonic, plus constant value if present (e.g., "const\n42") +- Each `GraphEdge` becomes a cytoscape edge element with `data: { source, target, targetLabel, sourceLabel }` + - `targetLabel` = edge port ("L" or "R") + - `sourceLabel` = branch label for routing ops — check if source node is a routing op, and if `source_port` is set, use "T" for L and "F" for R +- Each `GraphRegion` with `kind: "function"` becomes a compound parent node, and its `node_ids` nodes get `parent` set to the region's `tag` +- Nodes with `has_error: true` get the `error` class +- Edges with `has_error: true` get the `error` class + +```typescript +import cytoscape from "cytoscape"; +import dagre from "cytoscape-dagre"; +import type { GraphUpdate, GraphNode, GraphEdge, GraphRegion } from "./types"; +import { stylesheet } from "./style"; +import { logicalLayout } from "./layout"; + +cytoscape.use(dagre); + +const ROUTING_CATEGORY = "routing"; + +const cy = cytoscape({ + container: document.getElementById("graph"), + style: stylesheet, + elements: [], +}); + +function buildLabel(node: GraphNode): string { + if (node.const !== null) { + return `${node.opcode}\n${node.const}`; + } + return node.opcode; +} + +function buildElements( + update: GraphUpdate +): cytoscape.ElementDefinition[] { + const elements: cytoscape.ElementDefinition[] = []; + const regionParents = new Map(); + + for (const region of update.regions) { + if (region.kind === "function") { + elements.push({ + data: { id: region.tag, label: region.tag }, + }); + for (const nodeId of region.node_ids) { + regionParents.set(nodeId, region.tag); + } + } + } + + for (const node of update.nodes) { + const el: cytoscape.ElementDefinition = { + data: { + id: node.id, + label: buildLabel(node), + colour: node.colour, + category: node.category, + pe: node.pe, + iram_offset: node.iram_offset, + ctx: node.ctx, + }, + classes: node.has_error ? "error" : undefined, + }; + const parent = regionParents.get(node.id); + if (parent) { + el.data.parent = parent; + } + elements.push(el); + } + + for (const edge of update.edges) { + const sourceNode = update.nodes.find((n) => n.id === edge.source); + let sourceLabel: string | undefined; + if (sourceNode && sourceNode.category === ROUTING_CATEGORY && edge.source_port) { + sourceLabel = edge.source_port === "L" ? "T" : "F"; + } + + elements.push({ + data: { + id: `${edge.source}->${edge.target}:${edge.port}`, + source: edge.source, + target: edge.target, + targetLabel: edge.port, + sourceLabel: sourceLabel ?? "", + }, + classes: edge.has_error ? "error" : undefined, + }); + } + + return elements; +} + +function renderGraph(update: GraphUpdate): void { + cy.batch(() => { + cy.elements().remove(); + cy.add(buildElements(update)); + }); + cy.layout(logicalLayout()).run(); + cy.fit(undefined, 40); +} + +function connect(): void { + const protocol = location.protocol === "https:" ? "wss:" : "ws:"; + const ws = new WebSocket(`${protocol}//${location.host}/ws`); + + ws.onmessage = (event: MessageEvent) => { + const update: GraphUpdate = JSON.parse(event.data); + if (update.type === "graph_update") { + renderGraph(update); + } + }; + + ws.onclose = () => { + setTimeout(connect, 2000); + }; +} + +connect(); +``` + +**Verification:** + +```bash +cd /home/orual/Projects/or1-design/dfgraph/frontend +npm run build +ls -la dist/bundle.js +``` + +Expected: `dist/bundle.js` is created without errors. + +**Commit:** `feat: implement WebSocket client and logical view rendering` + + + +### Task 5: Manual verification of the logical view + +**Files:** None (manual testing) + +**Step 1: Create a test dfasm file** + +Create a temporary test file (e.g., `/tmp/test_render.dfasm`) with a simple program. Use a valid program from the existing test suite. A minimal example: + +```dfasm +@system pe=2 sm=1 + +$main { + &a = add @const_1 @const_2 + &b = sub &a @const_3 + &out = pass &b + + &const_1 = const 10 + &const_2 = const 20 + &const_3 = const 5 +} +``` + +(Adapt this to match actual dfasm syntax — check `tests/test_e2e.py` for known-working examples.) + +**Step 2: Build the frontend** + +```bash +cd /home/orual/Projects/or1-design/dfgraph/frontend +npm run build +``` + +**Step 3: Start the server** + +```bash +cd /home/orual/Projects/or1-design +python -m dfgraph /tmp/test_render.dfasm +``` + +Expected: Browser opens showing an interactive graph. + +**Step 4: Verify visual elements** + +Check visually: +- Nodes are circular with opcode labels (AC1.2) +- Nodes have category-appropriate colours (AC1.3) +- Edges have arrowheads and port labels (AC1.4) +- Function region (`$main`) has a dashed bounding box (AC1.5) + +**Step 5: Run full test suite** + +```bash +cd /home/orual/Projects/or1-design +python -m pytest tests/ -v +``` + +Expected: All tests pass. + diff --git a/docs/implementation-plans/2026-02-23-dataflow-renderer/phase_06.md b/docs/implementation-plans/2026-02-23-dataflow-renderer/phase_06.md new file mode 100644 index 0000000..8dca459 --- /dev/null +++ b/docs/implementation-plans/2026-02-23-dataflow-renderer/phase_06.md @@ -0,0 +1,346 @@ +# Dataflow Graph Renderer Implementation Plan — Phase 6: Frontend — Physical View and View Toggle + +**Goal:** Add the PE-clustered physical view and a toggle to switch between logical and physical views. + +**Architecture:** The physical view reuses the same graph data from the backend but restructures the cytoscape elements: nodes are grouped into PE compound parent nodes instead of function regions, node labels include IRAM offset and context slot annotations, and cross-PE edges are visually distinguished from intra-PE edges. A toolbar button toggles between views by re-building the element structure and re-running layout. + +**Tech Stack:** TypeScript, cytoscape.js 3.30+, cytoscape-dagre 2.5+ + +**Scope:** 8 phases from original design (this is phase 6 of 8) + +**Codebase verified:** 2026-02-23 + +--- + +## Acceptance Criteria Coverage + +This phase implements and tests: + +### dataflow-renderer.AC3: Physical view shows PE-grouped layout +- **dataflow-renderer.AC3.1 Success:** Nodes grouped into PE cluster boxes labelled by PE ID +- **dataflow-renderer.AC3.2 Success:** Nodes within clusters annotated with IRAM offset and context slot +- **dataflow-renderer.AC3.3 Success:** Cross-PE edges visually distinct from intra-PE edges +- **dataflow-renderer.AC3.4 Failure:** Physical view unavailable when pipeline hasn't reached allocate stage + +--- + + +### Task 1: Add physical view layout to layout.ts + +**Verifies:** dataflow-renderer.AC3.1 + +**Files:** +- Modify: `/home/orual/Projects/or1-design/dfgraph/frontend/src/layout.ts` (created in Phase 5, Task 3) + +**Implementation:** + +Add a `physicalLayout()` function alongside the existing `logicalLayout()`. The physical layout also uses dagre top-to-bottom but with different spacing to accommodate PE cluster boxes. + +```typescript +export function physicalLayout(): object { + return { + name: "dagre", + rankDir: "TB", + nodeSep: 40, + rankSep: 60, + edgeSep: 15, + animate: false, + }; +} +``` + +**Verification:** + +```bash +cd /home/orual/Projects/or1-design/dfgraph/frontend +npx esbuild src/layout.ts --bundle --outfile=/dev/null --format=esm --target=es2020 +``` + +Expected: No errors. + +**Commit:** `feat: add physical layout configuration` + + + +### Task 2: Add physical view styles to style.ts + +**Verifies:** dataflow-renderer.AC3.1, dataflow-renderer.AC3.2, dataflow-renderer.AC3.3 + +**Files:** +- Modify: `/home/orual/Projects/or1-design/dfgraph/frontend/src/style.ts` (created in Phase 5, Task 2) + +**Implementation:** + +Add new stylesheet entries for the physical view. These styles coexist with the logical view styles — the view switch changes which elements exist (PE clusters vs function regions) and which classes are applied. + +Add to the stylesheet array: + +1. **PE cluster parent nodes** (AC3.1): rounded rectangle with solid border, PE label at top +2. **IRAM/ctx annotation** (AC3.2): handled via a different label format in the node data (set in main.ts during element building) +3. **Cross-PE edges** (AC3.3): thicker, darker edges with `.cross-pe` class; intra-PE edges lighter + +```typescript +// Add these entries to the stylesheet array: + +// PE cluster parent node style +{ + selector: "node.pe-cluster", + style: { + shape: "roundrectangle", + "border-width": 2, + "border-color": "#5c6bc0", + "background-color": "rgba(92, 107, 192, 0.06)", + padding: "20px", + "text-valign": "top", + "text-halign": "center", + label: "data(label)", + "font-size": 13, + "font-weight": "bold", + color: "#5c6bc0", + }, +}, + +// Cross-PE edge (thicker, darker) +{ + selector: "edge.cross-pe", + style: { + width: 3, + "line-color": "#5c6bc0", + "target-arrow-color": "#5c6bc0", + }, +}, + +// Intra-PE edge (lighter) +{ + selector: "edge.intra-pe", + style: { + width: 1.5, + "line-color": "#bbb", + "target-arrow-color": "#bbb", + }, +}, +``` + +**Verification:** + +```bash +cd /home/orual/Projects/or1-design/dfgraph/frontend +npx esbuild src/style.ts --bundle --outfile=/dev/null --format=esm --target=es2020 +``` + +Expected: No errors. + +**Commit:** `feat: add physical view styles` + + + +### Task 3: Add view toggle and physical element building to main.ts + +**Verifies:** dataflow-renderer.AC3.1, dataflow-renderer.AC3.2, dataflow-renderer.AC3.3, dataflow-renderer.AC3.4 + +**Files:** +- Modify: `/home/orual/Projects/or1-design/dfgraph/frontend/src/main.ts` (modified in Phase 5, Task 4) +- Modify: `/home/orual/Projects/or1-design/dfgraph/frontend/index.html` (add toggle button) + +**Implementation:** + +Add two things: (1) a `buildPhysicalElements()` function that creates PE cluster parent nodes and assigns nodes to them, and (2) a view toggle button in the toolbar. + +**Physical element building** (AC3.1, AC3.2, AC3.3): + +```typescript +function buildPhysicalLabel(node: GraphNode): string { + const parts = [node.opcode]; + if (node.const !== null) parts.push(`=${node.const}`); + if (node.iram_offset !== null) parts.push(`\n[iram:${node.iram_offset}, ctx:${node.ctx}]`); + return parts.join(""); +} + +function buildPhysicalElements(update: GraphUpdate): cytoscape.ElementDefinition[] { + const elements: cytoscape.ElementDefinition[] = []; + const peNodes = new Map(); + + // Group nodes by PE + for (const node of update.nodes) { + if (node.pe !== null) { + if (!peNodes.has(node.pe)) peNodes.set(node.pe, []); + peNodes.get(node.pe)!.push(node.id); + } + } + + // Create PE cluster parent nodes + for (const peId of peNodes.keys()) { + elements.push({ + data: { id: `pe-${peId}`, label: `PE ${peId}` }, + classes: "pe-cluster", + }); + } + + // Create operation nodes parented to PE clusters + for (const node of update.nodes) { + const el: cytoscape.ElementDefinition = { + data: { + id: node.id, + label: buildPhysicalLabel(node), + colour: node.colour, + category: node.category, + pe: node.pe, + iram_offset: node.iram_offset, + ctx: node.ctx, + }, + classes: node.has_error ? "error" : undefined, + }; + if (node.pe !== null) { + el.data.parent = `pe-${node.pe}`; + } + elements.push(el); + } + + // Create edges with cross-PE / intra-PE classification + const nodePeMap = new Map(); + for (const node of update.nodes) { + nodePeMap.set(node.id, node.pe); + } + + for (const edge of update.edges) { + const sourcePe = nodePeMap.get(edge.source); + const targetPe = nodePeMap.get(edge.target); + const isCrossPe = sourcePe !== null && targetPe !== null && sourcePe !== targetPe; + + elements.push({ + data: { + id: `${edge.source}->${edge.target}:${edge.port}`, + source: edge.source, + target: edge.target, + targetLabel: edge.port, + sourceLabel: "", + }, + classes: isCrossPe ? "cross-pe" : "intra-pe", + }); + } + + return elements; +} +``` + +**View toggle** (AC3.4): + +Track the current view state and the latest graph update. When toggling: +- If switching to physical and stage is not "allocate", show a message (physical view unavailable) +- Otherwise, re-build elements for the new view and re-layout + +Add a toggle button to `index.html` toolbar: +```html + +``` + +In main.ts, add toggle logic: +```typescript +type ViewMode = "logical" | "physical"; +let currentView: ViewMode = "logical"; +let latestUpdate: GraphUpdate | null = null; + +function renderUpdate(update: GraphUpdate): void { + latestUpdate = update; + if (currentView === "physical" && update.stage !== "allocate") { + currentView = "logical"; + // Update button text + } + if (currentView === "logical") { + renderLogical(update); + } else { + renderPhysical(update); + } +} + +function renderLogical(update: GraphUpdate): void { + cy.batch(() => { + cy.elements().remove(); + cy.add(buildElements(update)); // existing from Phase 5 + }); + cy.layout(logicalLayout()).run(); + cy.fit(undefined, 40); +} + +function renderPhysical(update: GraphUpdate): void { + cy.batch(() => { + cy.elements().remove(); + cy.add(buildPhysicalElements(update)); + }); + cy.layout(physicalLayout()).run(); + cy.fit(undefined, 40); +} + +const toggleBtn = document.getElementById("view-toggle"); +if (toggleBtn) { + toggleBtn.addEventListener("click", () => { + if (!latestUpdate) return; + if (currentView === "logical") { + if (latestUpdate.stage !== "allocate") { + // Physical view not available + return; + } + currentView = "physical"; + toggleBtn.textContent = "Logical View"; + renderPhysical(latestUpdate); + } else { + currentView = "logical"; + toggleBtn.textContent = "Physical View"; + renderLogical(latestUpdate); + } + }); +} +``` + +**Import `physicalLayout` from layout.ts** at the top of main.ts. + +**Verification:** + +```bash +cd /home/orual/Projects/or1-design/dfgraph/frontend +npm run build +``` + +Expected: Bundle builds without errors. + +**Commit:** `feat: add physical view and view toggle` + + + +### Task 4: Manual verification of the physical view + +**Files:** None (manual testing) + +**Step 1: Build and start** + +```bash +cd /home/orual/Projects/or1-design/dfgraph/frontend && npm run build +cd /home/orual/Projects/or1-design && python -m dfgraph /tmp/test_render.dfasm +``` + +**Step 2: Verify logical view loads (default)** + +Expected: Same logical view as Phase 5. + +**Step 3: Click "Physical View" toggle** + +Expected: +- Nodes grouped into PE cluster boxes labelled "PE 0", "PE 1", etc. (AC3.1) +- Each node shows `[iram:N, ctx:M]` annotation (AC3.2) +- Cross-PE edges are thicker/darker than intra-PE edges (AC3.3) + +**Step 4: Test AC3.4 — physical view unavailable for partial graphs** + +Edit the test dfasm file to introduce an error (undefined name). Save. The view should revert to logical if currently showing physical, and the toggle button should be disabled or ineffective. + +**Step 5: Run full test suite** + +```bash +cd /home/orual/Projects/or1-design +python -m pytest tests/ -v +``` + +Expected: All tests pass. + +**Note:** The design mentions SM connections as a separate cluster in the physical view. SM nodes are not part of the IRGraph node set (they are referenced by `sm_id` on MemOp nodes), so SM clusters are not rendered in this phase. If SM cluster visualisation is needed later, it would require synthesising virtual SM nodes from `sm_id` references. + diff --git a/docs/implementation-plans/2026-02-23-dataflow-renderer/phase_07.md b/docs/implementation-plans/2026-02-23-dataflow-renderer/phase_07.md new file mode 100644 index 0000000..588a88a --- /dev/null +++ b/docs/implementation-plans/2026-02-23-dataflow-renderer/phase_07.md @@ -0,0 +1,252 @@ +# Dataflow Graph Renderer Implementation Plan — Phase 7: Error Display + +**Goal:** Render partial graphs with error highlighting and a collapsible error panel. + +**Architecture:** The error data is already in the graph JSON (from Phase 3). This phase adds: (1) the error panel UI at the bottom of the page, populated from the `errors` array, (2) click-to-highlight behaviour linking errors to graph nodes, and (3) a parse-error-only view when no graph is available. Error node/edge styling was already defined in Phase 5's style.ts (`.error` class with red dashed borders). + +**Tech Stack:** TypeScript, HTML/CSS (no new dependencies) + +**Scope:** 8 phases from original design (this is phase 7 of 8) + +**Codebase verified:** 2026-02-23 + +--- + +## Acceptance Criteria Coverage + +This phase implements and tests: + +### dataflow-renderer.AC5: Error display +- **dataflow-renderer.AC5.1 Success:** Nodes/edges with errors render with red highlight +- **dataflow-renderer.AC5.4 Success:** Fixing errors and saving clears error highlights +- **dataflow-renderer.AC5.5 Failure:** Lark parse error (no graph possible) shows error-only view + +--- + + +### Task 1: Add error panel HTML and CSS to index.html + +**Verifies:** dataflow-renderer.AC5.1, dataflow-renderer.AC5.5 + +**Files:** +- Modify: `/home/orual/Projects/or1-design/dfgraph/frontend/index.html` (add error panel markup and styles) + +**Implementation:** + +Add a collapsible error panel below the graph area. When errors exist, the panel appears with a list of errors. When no errors, the panel is hidden. + +Add to the `