diff --git a/DESIGN_SHOW.md b/DESIGN_SHOW.md new file mode 100644 index 0000000..92cedb9 --- /dev/null +++ b/DESIGN_SHOW.md @@ -0,0 +1,186 @@ +# odoc-switchdocs — `show`: print the docs for a reference + +A CLI subcommand that takes an odoc reference and prints the docstring of the +item it names, as Markdown: + +``` +$ switchdocs show Odoc_model.Paths.Identifier +``` + +The job is essentially the reference-resolution half of `odoc link`, run +standalone against every `.odoc` file in the switch, followed by rendering the +resolved item's doc comment to Markdown. odoc already does almost all of this +in `src/odoc/url.ml` (the `odoc html-url` command), which resolves a reference +to an identifier; we extend that path to fetch and render the full docstring +rather than compute a URL. + +## How `odoc link` resolves references (the part we reuse) + +`odoc link` (`src/odoc/odoc_link.ml`) loads an `.odoc`, builds an +`Odoc_xref2.Env.t` from a `Resolver.t`, and calls into `Odoc_xref2.Link`. Link +in turn uses `Odoc_xref2.Ref_tools.resolve_reference` for every `{!...}` it +encounters. The environment is what gives the resolver access to other units: +`Resolver.create` scans a list of `-I` directories into an +`Accessible_paths` table (capitalised module name → `.odoc` file), and +`Resolver.build_env_for_reference` wraps that into an `Env.t` whose `lookup_unit` +loads units by name on demand. This is the *most permissive* env builder +(`important_digests = false`, no current unit), which is exactly right for +resolving an arbitrary reference from the command line — see `url.ml`, which +already uses it. + +So our resolution code is, almost verbatim, `Url.resolve`: + +```ocaml +let resolver = + Resolver.create ~important_digests:false ~directories ~open_modules:[] ~roots:None +in +let reference = + Odoc_model.Semantics.parse_reference s (* string -> Reference.t *) + |> Odoc_model.Error.handle_errors_and_warnings ~warnings_options +in +let env = Resolver.build_env_for_reference resolver in +Odoc_xref2.Ref_tools.resolve_reference env reference + |> Odoc_model.Error.raise_warnings +(* -> (Reference.Resolved.t * Comment.paragraph option, _) result *) +``` + +`Reference.Resolved.identifier resolved` then yields the canonical +`Identifier.t` of the target (the same step `url.ml` feeds to +`Document.Url.from_identifier`). + +Note the second element of the result is only a *synopsis* paragraph (the +first paragraph, used when expanding `{!ref}` inline). We want the whole doc +comment, so we ignore it and fetch the docs separately. + +## Building `directories` — always the whole switch + +Unlike `odoc html-url`, which is handed an explicit `-I` set, `show` always +searches across *every package installed in the switch*: the include set is +derived, not configured. + +`Resolver.create`'s `Accessible_paths` looks up units by capitalised base name, +scanning the directories it is given for `*.odoc` files. In a switch the driver +writes these under `$OPAM_SWITCH_PREFIX/doc///.odoc` (verified +on the dev switch: e.g. `doc/astring/astring/astring.odoc`). So: + +- discover the switch prefix exactly as the existing commands do + (`Switchdocs.Switch.detect`, `--prefix`); +- recursively collect every directory under `/doc` that contains a + `.odoc` file, and pass them all as `directories` (`Show.scan`). + +Only *top-level unit* files need to be reachable by name; sub-modules +(`.Paths`, `.Identifier`) are resolved by walking into the parent unit's +signature in memory, not from files. Cross-package module-name clashes are +handled by odoc already (it warns and picks the first match). + +We resolve against `.odoc` (compiled) files, not `.odocl` — `Accessible_paths` +only loads the `.odoc` extension, and compiled units already carry their +docstrings. (References *inside* a docstring are unresolved in `.odoc`; that +only matters if we want to turn doc-comment cross-references into links — see +Open questions.) + +## From resolved identifier to docstring + +This is the one piece `odoc link`/`url.ml` doesn't already hand us — and the +chosen approach is to **load the target unit's linked `.odocl` and walk its +`Lang` tree to the item** (an env `lookup_by_id`-based shortcut was considered +and rejected: `lookup_by_id` only finds ids already registered in `env.ids`, +which is not guaranteed for an arbitrary deep reference after resolution). + +The resolved identifier names a root compilation unit (`Odoc_model`) and a path +of sub-modules down to the item. So: + +1. `Identifier.fullname id` gives the dotted path; its head is the root unit. +2. Load that unit's `.odocl` and walk `cu.content`'s `Lang.Signature`, + recursing through module / module-type expansions and `include` expansions, + comparing each item's identifier to the target with `Identifier.equal`. + `search_*` in `lib/show.ml` is that walk; expansions are reached via the + `simple_expansion`/`p_expansion`/`w_expansion`/… fields, always populated in + a linked file. +3. Return the matched item's `doc : Comment.docs`. Because it comes from the + **linked** file, cross-references inside the comment are already resolved, so + they render as proper links rather than bare text. + +**Disambiguating the unit (the subtle bit).** A switch routinely contains the +same module name in several packages — e.g. both `ocaml-compiler` and +`ocamlfind` ship a `Stdlib.odocl`. Looking the unit up by *name* alone can load +a different `Stdlib` than the one the reference resolved into, and then +`Identifier.equal` never matches. odoc identifiers are keyed by a dotted string +(`ikey`) that runs from the leaf up through the root module to its container +*page* (the package): e.g. `t_result.r_Stdlib.p_stdlib.p_ocamlfind`. The owning +unit is therefore the candidate whose own `ikey` is a **suffix** of the +target's. `Show.owns` uses exactly this test to pick the correct `.odocl` among +same-named candidates, keeping the loaded unit consistent with the resolved +identifier. + +**Module top-comments.** A module/module-type declaration is often itself +undocumented, the prose living in the top-comment of its signature (the +`(** … *)` just inside `sig`). When the target *is* a module, `show` prefers the +attached doc and falls back to the expansion signature's `doc`. (A bare section +heading such as `{2 …}` is *not* a top-comment — it is a floating comment item — +so a module whose signature opens with only a heading correctly reports "no +documentation found".) + +## Rendering to Markdown + +odoc's rendering pipeline is Lang → `Odoc_document` IR → backend. The doc +comment's `elements` go into the IR via `Odoc_document.Comment.standalone`, and +the Markdown backend (`src/markdown2`, library `odoc.markdown`) serialises it. +`Renderer.to_string` takes a single block, so the item list is wrapped in +`Renderer.Block.Blocks`: + +```ocaml +let blocks = + Odoc_document.Comment.standalone docs.elements + |> Odoc_markdown.Generator.items ~config ~resolve:(Odoc_markdown.Link.Base "") +in +Odoc_markdown.Renderer.to_string (Odoc_markdown.Renderer.Block.Blocks blocks) +``` + +`Base ""` is fine: we are not producing a browsable tree, so cross-reference +links render as relative-ish paths/text, acceptable for terminal output. + +## Tool shape (as built) + +A user-facing subcommand alongside `search`, `rebuild`, etc. in `bin/main.ml`, +backed by `lib/show.ml` (mirroring how `lib/search.ml` backs `search`): + +``` +switchdocs show [--prefix DIR] REFERENCE +``` + +- `REFERENCE` is an odoc reference string (`Odoc_model.Paths.Identifier`, + `Stdlib.List.map`, optionally tagged like `val:`/`type:`), parsed with + `Odoc_model.Semantics.parse_reference`. +- Exit codes: `0` on success; `1` (the shared `exit_doc_failure`) on a parse, + resolution, load, or no-docs failure, with the message on stderr; cmdliner's + CLI-error code only for switch-detection problems. +- Resolution failures surface odoc's own + `Errors.Tools_error.pp_reference_lookup_error`, as `url.ml` does. +- The resolver emits unavoidable "ambiguous lookup" notices straight to stderr + when a name occurs in several packages; `show` follows the resolved + identifier (so the choice is correct) and suppresses that stderr around the + resolve call (`with_suppressed_stderr`) to keep output clean. Errors are + returned as values, so none are lost. + +### Dependencies + +`lib/dune` gains the odoc internal libraries `odoc.model`, `odoc.xref2`, +`odoc.odoc` (for `Resolver`/`Semantics`), `odoc.document`, `odoc.markdown`. +These must be the *same* odoc that produced the switch's `.odoc`/`.odocl` (the +formats are version-coupled). In this repo's dev switch odoc is pinned to the +driver's source, so they match; in general `switchdocs` should be built against +the switch's odoc. + +## Open questions / future work + +- **Choosing among multiple matches.** A bare module name ambiguous across + packages currently resolves to whichever odoc lists first. A `--package` + filter (as `search` has) would let the user disambiguate deliberately. +- **Whole-item rendering.** `show` prints only the doc *comment*. Rendering the + item's signature (the `type`/`val` declaration) above it is a different + `Odoc_document` entry point (`Generator.Make`) and a larger job; out of scope + for the first cut. +- **Cross-reference link targets.** With `Base ""` the in-comment links are not + meaningfully clickable from a terminal; a future mode could rewrite them to + `switchdocs show` invocations or to the switch's HTML. diff --git a/bin/main.ml b/bin/main.ml index 9ad7eb5..10d2945 100644 --- a/bin/main.ml +++ b/bin/main.ml @@ -302,6 +302,50 @@ let search_cmd = (Cmd.info "search" ~doc ~man) Term.(ret (const run $ switch_t $ query $ limit $ packages)) +(* show — user-facing: real exit codes. *) +let show_cmd = + let reference = + Arg.( + required + & pos 0 (some string) None + & info [] ~docv:"REFERENCE" + ~doc: + "An odoc reference to an item, e.g. \ + $(b,Odoc_model.Paths.Identifier) or $(b,Stdlib.List.map). The \ + same syntax accepted inside $(b,{!...}) in doc comments, including \ + kind tags such as $(b,val:) or $(b,type:).") + in + let run switch reference = + match switch with + | Error (`Msg m) -> `Error (false, m) + | Ok sw -> ( + match Switchdocs.Show.show sw reference with + | Ok markdown -> + print_string markdown; + `Ok 0 + | Error (`Msg m) -> + (* A failure to resolve or find docs is a runtime outcome, not a + command-line misuse, so report it on stderr and exit non-zero + rather than with cmdliner's CLI-error code. *) + Printf.eprintf "switchdocs show: %s\n" m; + `Ok exit_doc_failure) + in + let doc = "Print the documentation for a reference, as Markdown" in + let man = + [ + `S Manpage.s_description; + `P + "Resolves $(i,REFERENCE) against every package documented in the \ + switch, using the same reference-resolution machinery as \ + $(b,odoc link), then prints the resolved item's doc comment rendered \ + as Markdown. Unlike a single-package tool, the whole switch is always \ + searched."; + ] + in + Cmd.v + (Cmd.info "show" ~doc ~man) + Term.(ret (const run $ switch_t $ reference)) + (* setup *) let setup_cmd = let apply = @@ -360,6 +404,6 @@ let main_cmd = in Cmd.group (Cmd.info "switchdocs" ~version:"%%VERSION%%" ~doc ~man) - [ record_cmd; sync_cmd; rebuild_cmd; order_cmd; search_cmd; setup_cmd ] + [ record_cmd; sync_cmd; rebuild_cmd; order_cmd; search_cmd; show_cmd; setup_cmd ] let () = exit (Cmd.eval' main_cmd) diff --git a/lib/dune b/lib/dune index b6aac43..fd306ce 100644 --- a/lib/dune +++ b/lib/dune @@ -5,6 +5,11 @@ fpath opam-format unix + odoc.model + odoc.xref2 + odoc.odoc + odoc.document + odoc.markdown sherlodoc.db sherlodoc.db_store sherlodoc.query)) diff --git a/lib/show.ml b/lib/show.ml new file mode 100644 index 0000000..1beca49 --- /dev/null +++ b/lib/show.ml @@ -0,0 +1,302 @@ +(* Print the documentation of an item named by an odoc reference. + + This is, in essence, the reference-resolution half of [odoc link] (see + [src/odoc/url.ml] in odoc, the [odoc html-url] command) run standalone over + the whole switch, followed by fetching and rendering the resolved item's doc + comment. Three stages: + + 1. Scan the switch's [doc/] tree once for the include directories (dirs + holding [.odoc] files) and an index of linked units ([.odocl] by name). + 2. Resolve the reference string to a canonical identifier with + [Odoc_xref2.Ref_tools.resolve_reference], exactly as odoc does. + 3. Load the identifier's owning unit's [.odocl] and walk its [Lang] tree to + the item, returning its doc comment, then render it to Markdown. *) + +module Id = Odoc_model.Paths.Identifier +module Lang = Odoc_model.Lang + +(* {1 Stage 1: scan the switch} *) + +(* All files (not directories) beneath [root], recursively. *) +let all_files root = + let rec loop acc dir = + match Bos.OS.Dir.contents ~rel:false dir with + | Error _ -> acc + | Ok entries -> + List.fold_left + (fun acc p -> + match Bos.OS.Dir.exists p with + | Ok true -> loop acc p + | _ -> p :: acc) + acc entries + in + loop [] root + +(* The include directories (every dir containing a [.odoc] file) and an index + from capitalised unit name to its [.odocl] files. odoc names a compilation + unit's file after the lowercased module name, e.g. [Astring] -> + [astring.odocl], and [Accessible_paths] looks units up by capitalised + basename, so we key the index the same way. *) +let scan sw = + let files = all_files (Switch.doc_dir sw) in + let dirs = Hashtbl.create 256 in + let odocl = Hashtbl.create 256 in + List.iter + (fun p -> + match Fpath.get_ext p with + | ".odoc" -> + let d = Fpath.parent p in + Hashtbl.replace dirs (Fpath.to_string d) d + | ".odocl" -> + let name = + Fpath.(rem_ext p |> basename) |> String.capitalize_ascii + in + Hashtbl.add odocl name p + | _ -> ()) + files; + let dirs = Hashtbl.fold (fun _ d acc -> d :: acc) dirs [] in + (dirs, odocl) + +(* {1 Stage 2: resolve the reference} *) + +let resolve_to_id ~directories ref_str = + let directories = + List.map + (fun d -> Odoc_odoc.Fs.Directory.of_string (Fpath.to_string d)) + directories + in + let resolver = + Odoc_odoc.Resolver.create ~important_digests:false ~directories + ~open_modules:[] ~roots:None + in + let warnings_options = + { + Odoc_model.Error.warn_error = false; + print_warnings = false; + warnings_tag = None; + } + in + match + Odoc_model.Semantics.parse_reference ref_str + |> Odoc_model.Error.handle_errors_and_warnings ~warnings_options + with + | Error (`Msg m) -> + Error (`Msg (Printf.sprintf "cannot parse reference %S: %s" ref_str m)) + | Ok reference -> ( + let env = Odoc_odoc.Resolver.build_env_for_reference resolver in + match + Odoc_xref2.Ref_tools.resolve_reference env reference + |> Odoc_model.Error.raise_warnings + with + | Error e -> + Error + (`Msg + (Format.asprintf "cannot resolve reference %S: %a" ref_str + Odoc_xref2.Errors.Tools_error.pp_reference_lookup_error e)) + | Ok (resolved, _synopsis) -> ( + match Odoc_model.Paths.Reference.Resolved.identifier resolved with + | Some id -> Ok id + | None -> + Error + (`Msg + (Printf.sprintf + "reference %S resolves to a hidden item with no \ + identifier" + ref_str)))) + +(* {1 Stage 3: fetch the doc comment from the linked unit} *) + +(* The signature exposed by a module/module-type expression, when an expansion + is present (it always is in a linked [.odocl]). *) +let rec simple_expansion_sig : Lang.ModuleType.simple_expansion -> _ = function + | Signature s -> Some s + | Functor (_, e) -> simple_expansion_sig e + +let rec modtype_expr_sig : Lang.ModuleType.expr -> Lang.Signature.t option = + function + | Signature s -> Some s + | Path { p_expansion = Some se; _ } -> simple_expansion_sig se + | With { w_expansion = Some se; _ } -> simple_expansion_sig se + | TypeOf { t_expansion = Some se; _ } -> simple_expansion_sig se + | Strengthen { s_expansion = Some se; _ } -> simple_expansion_sig se + | Functor (_, e) -> modtype_expr_sig e + | Path { p_expansion = None; _ } + | With { w_expansion = None; _ } + | TypeOf { t_expansion = None; _ } + | Strengthen { s_expansion = None; _ } -> + None + +let module_decl_sig : Lang.Module.decl -> Lang.Signature.t option = function + | Alias (_, Some se) -> simple_expansion_sig se + | Alias (_, None) -> None + | ModuleType e -> modtype_expr_sig e + +(* A module's own declaration is frequently undocumented, the prose living in + the top-comment of its signature instead (the [(** ... *)] just inside + [sig]). Prefer the attached doc, fall back to that top-comment. *) +let prefer (doc : Odoc_model.Comment.docs) fallback = + if doc.elements <> [] then doc + else match fallback with Some sg -> sg.Lang.Signature.doc | None -> doc + +let rec search_sig matches (sg : Lang.Signature.t) = + List.find_map (search_item matches) sg.items + +and search_item matches (item : Lang.Signature.item) : + Odoc_model.Comment.docs option = + match item with + | Module (_, m) -> + let sg = module_decl_sig m.type_ in + if matches (m.id :> Id.t) then Some (prefer m.doc sg) + else Option.bind sg (search_sig matches) + | ModuleType mt -> + let sg = Option.bind mt.expr modtype_expr_sig in + if matches (mt.id :> Id.t) then Some (prefer mt.doc sg) + else Option.bind sg (search_sig matches) + | ModuleSubstitution ms -> + if matches (ms.id :> Id.t) then Some ms.doc else None + | ModuleTypeSubstitution mts -> + if matches (mts.id :> Id.t) then Some mts.doc else None + | Type (_, t) -> if matches (t.id :> Id.t) then Some t.doc else search_type matches t + | TypeSubstitution t -> + if matches (t.id :> Id.t) then Some t.doc else search_type matches t + | TypExt te -> + List.find_map + (fun (c : Lang.Extension.Constructor.t) -> + if matches (c.id :> Id.t) then Some c.doc else None) + te.constructors + | Exception e -> + if matches (e.id :> Id.t) then Some e.doc else search_args matches e.args + | Value v -> if matches (v.id :> Id.t) then Some v.doc else None + | Class (_, c) -> + if matches (c.id :> Id.t) then Some c.doc + else Option.bind c.expansion (search_class_sig matches) + | ClassType (_, ct) -> + if matches (ct.id :> Id.t) then Some ct.doc + else Option.bind ct.expansion (search_class_sig matches) + | Include i -> search_sig matches i.expansion.content + | Open _ | Comment _ -> None + +and search_type matches (t : Lang.TypeDecl.t) = + match t.representation with + | None -> None + | Some (Variant cs) -> List.find_map (search_constructor matches) cs + | Some (Record fs) -> List.find_map (search_field matches) fs + | Some (Record_unboxed_product fs) -> + List.find_map + (fun (f : Lang.TypeDecl.UnboxedField.t) -> + if matches (f.id :> Id.t) then Some f.doc else None) + fs + | Some Extensible -> None + +and search_constructor matches (c : Lang.TypeDecl.Constructor.t) = + if matches (c.id :> Id.t) then Some c.doc else search_args matches c.args + +and search_args matches : Lang.TypeDecl.Constructor.argument -> _ = function + | Record fs -> List.find_map (search_field matches) fs + | Tuple _ -> None + +and search_field matches (f : Lang.TypeDecl.Field.t) = + if matches (f.id :> Id.t) then Some f.doc else None + +and search_class_sig matches (cs : Lang.ClassSignature.t) = + List.find_map + (fun (item : Lang.ClassSignature.item) -> + match item with + | Method m -> if matches (m.id :> Id.t) then Some m.doc else None + | InstanceVariable v -> + if matches (v.id :> Id.t) then Some v.doc else None + | Constraint _ | Inherit _ | Comment _ -> None) + cs.items + +let docs_of_unit target (cu : Lang.Compilation_unit.t) = + let matches id = Id.equal id target in + match cu.content with + | Pack _ -> None + | Module sg -> + if matches (cu.id :> Id.t) then Some sg.doc else search_sig matches sg + +(* A unit owns [target] when its root identifier is a suffix of [target]'s key. + Identifier keys are dotted from the leaf up to the root container page (the + package), e.g. [t_result.r_Stdlib.p_stdlib.p_ocamlfind], so this both + confirms the right unit and disambiguates a module name that occurs in more + than one package — we must load the very unit the reference resolved into, + not just any file of the same name. *) +let owns target (cu : Lang.Compilation_unit.t) = + let tk = target.Id.ikey and uk = (cu.id :> Id.t).Id.ikey in + let lt = String.length tk and lu = String.length uk in + lt >= lu && String.sub tk (lt - lu) lu = uk + +let load_cu path = + match Odoc_odoc.Odoc_file.load path with + | Ok { content = Unit_content cu; _ } -> Some cu + | Ok _ | Error (`Msg _) -> None + +let load_unit odocl ref_str id = + match Id.fullname id with + | [] -> Error (`Msg "could not determine the root module of the reference") + | root :: _ -> ( + match Hashtbl.find_all odocl (String.capitalize_ascii root) with + | [] -> + Error + (`Msg + (Printf.sprintf + "reference %S resolves into %s, but no linked documentation \ + for it was found in the switch" + ref_str root)) + | candidates -> ( + (* Prefer the unit that actually owns the identifier; fall back to + the first loadable candidate if the keying scheme ever changes. *) + let rec pick fallback = function + | [] -> fallback + | path :: rest -> ( + match load_cu path with + | Some cu when owns id cu -> Some cu + | Some cu -> pick (match fallback with None -> Some cu | f -> f) rest + | None -> pick fallback rest) + in + match pick None candidates with + | Some cu -> Ok cu + | None -> + Error + (`Msg + (Printf.sprintf + "could not load any linked documentation for %s" root)))) + +(* {1 Rendering} *) + +let render_markdown (docs : Odoc_model.Comment.docs) = + let config = Odoc_markdown.Config.make ~root_url:None ~allow_html:false () in + let blocks = + Odoc_document.Comment.standalone docs.elements + |> Odoc_markdown.Generator.items ~config + ~resolve:(Odoc_markdown.Link.Base "") + in + Odoc_markdown.Renderer.to_string (Odoc_markdown.Renderer.Block.Blocks blocks) + +(* {1 Entry point} *) + +(* The resolver prints ambiguity notices straight to stderr (it cannot help it: + the same module name legitimately occurs in several packages when the whole + switch is on the search path). We follow the resolved identifier, so the + "first match" it reports is the right one — silence the noise around the + call. Errors are returned as values, not printed, so nothing useful is lost. *) +let with_suppressed_stderr f = + flush stderr; + let saved = Unix.dup Unix.stderr in + let devnull = Unix.openfile "/dev/null" [ Unix.O_WRONLY ] 0o600 in + Unix.dup2 devnull Unix.stderr; + Unix.close devnull; + Fun.protect f ~finally:(fun () -> + flush stderr; + Unix.dup2 saved Unix.stderr; + Unix.close saved) + +let show sw ref_str = + let directories, odocl = scan sw in + Result.bind (with_suppressed_stderr (fun () -> resolve_to_id ~directories ref_str)) + @@ fun id -> + Result.bind (load_unit odocl ref_str id) @@ fun cu -> + match docs_of_unit id cu with + | None | Some { elements = []; _ } -> + Error (`Msg (Printf.sprintf "no documentation found for %s" ref_str)) + | Some docs -> Ok (render_markdown docs) diff --git a/lib/show.mli b/lib/show.mli new file mode 100644 index 0000000..e2f8320 --- /dev/null +++ b/lib/show.mli @@ -0,0 +1,12 @@ +(** Resolve an odoc reference against every package installed in the switch and + render the referenced item's documentation as Markdown. *) + +val show : Switch.t -> string -> (string, [> `Msg of string ]) result +(** [show sw reference] parses [reference] (an odoc reference such as + [Odoc_model.Paths.Identifier] or [Stdlib.List.map]), resolves it against the + [.odoc] files of all packages documented in [sw] using the same machinery + [odoc link] uses, fetches the resolved item's doc comment from its (linked) + [.odocl] file, and returns it rendered as Markdown. + + Errors if the reference cannot be parsed or resolved, if the owning unit + cannot be loaded, or if the item carries no documentation. *)