From 79d12d697341a963d79d3d735ed00a2469f4b1a9 Mon Sep 17 00:00:00 2001 From: oppiliappan Date: Fri, 08 Aug 2025 08:36:25 +0000 Subject: [PATCH] add bits about workflow debugging Signed-off-by: oppiliappan --- pages/blog/ci.md | 116 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++-------------------------------------------- 1 file(s) changed, 72 insertion(s)(+), 44 deletion(s)(-) diff --git a/pages/blog/ci.md b/pages/blog/ci.md --- a/pages/blog/ci.md +++ b/pages/blog/ci.md @@ -10,13 +10,14 @@ - name: Anirudh email: anirudh@tangled.sh handle: icyphox.sh + - name: Akshay + email: akshay@tangled.sh + handle: oppi.li --- -Since launching Tangled, continuous integration has consistently topped our -feature request list. And rightfully so -- modern software development is -unthinkable without automated testing, building, and deployment pipelines. -Today, we're excited to announce that CI is no longer a wishlist item, but a -fully-featured reality. +Since launching Tangled, continuous integration has +consistently topped our feature request list. Today, CI is +no longer a wishlist item, but a fully-featured reality. Meet **spindle**: Tangled's new CI runner that brings powerful automation directly to your repositories. In typical Tangled fashion we've been dogfooding @@ -36,7 +37,7 @@ Once triggered, spindle reads your pipeline manifest, spins up the necessary execution environment (covered below), and runs your defined workflow steps. Throughout execution, it streams real-time logs and status updates -(`sh.tangled.pipeline.status`) back through websocktes, which the Tangled +(`sh.tangled.pipeline.status`) back through websockets, which the Tangled appview subscribes to for live updates. This architecture keeps everything responsive and real-time while maintaining @@ -45,7 +46,7 @@ ## spindle pipelines The pipeline manifest is defined in YAML, and should be relatively familiar to -those that have used other CI products. Here's a minimal example: +those that have used other CI solutions. Here's a minimal example: ```yaml # test.yaml @@ -59,10 +60,6 @@ - go steps: - - name: patch static dir - command: | - mkdir -p appview/pages/static; touch appview/pages/static/x - - name: run all tests environment: CGO_ENABLED: 1 @@ -70,26 +67,23 @@ go test -v ./... ``` -Manifests are stored under your repo's `.tangled/workflows` directory. There may -be multiple manifests here describing different workflows -- for example, a -`build.yaml`, `test.yaml` and a `lint.yaml`. You can read the [full manifest spec -here](https://tangled.sh/@tangled.sh/core/blob/master/docs/spindle/pipeline.md). - -The `when` block defines the set of events that can trigger the workflow run. -The above example will run tests on any push or pull request targeting the -`master` branch. - -Now the `dependencies` block is the real interesting bit. Dependencies for your -workflow, like Go, Node.js, Python etc. can be pulled in from nixpkgs. nixpkgs --- for the uninitiated -- is a vast collection of packages for the Nix package -manager. Fortunately, you needn't know nor care about Nix to use it! Just head -to https://search.nixos.org to find your package of choice (I'll bet 1€ that -it's there[^1]), toss it in the list and run your build. The Nix-savvy of you -lot will be happy to know that you can toss in custom registries there. +You can read the [full manifest spec +here](https://tangled.sh/@tangled.sh/core/blob/master/docs/spindle/pipeline.md), +but the `dependencies` block is the real interesting bit. +Dependencies for your workflow, like Go, Node.js, Python +etc. can be pulled in from nixpkgs. +[Nixpkgs](https://github.com/nixos/nixpkgs/) -- for the +uninitiated -- is a vast collection of packages for the Nix +package manager. Fortunately, you needn't know nor care +about Nix to use it! Just head to https://search.nixos.org +to find your package of choice (I'll bet 1€ that it's +there[^1]), toss it in the list and run your build. The +Nix-savvy of you lot will be happy to know that you can use +custom registries too. [^1]: I mean, if it isn't there, it's nowhere. -Finally, define your steps, neccesary environment variables and commands. +Finally, define your steps, necessary environment variables and commands. Commands can be multi-lined. Let's take a look at how spindle executes workflow steps. @@ -116,21 +110,53 @@ [hello-go](https://search.nixos.org/packages?channel=25.05&show=hello-go) package from nixpkgs. -Nixery is super handy since we can construct these images for CI environments on -the fly, with all dependencies baked in, and the best part: caching for commonly -used packages is free thanks to Docker (pre-existing layers get reused). We run -a Nixery instance of our own at https://nixery.tangled.sh but you may override -that if you choose not to trust us. +Nixery is super handy since we can construct these images +for CI environments on the fly, with all dependencies baked +in, and the best part: caching for commonly used packages is +free thanks to Docker (pre-existing layers get reused). We +run a Nixery instance of our own at +https://nixery.tangled.sh but you may override that if you +choose to. -## pipeline statuses and log streaming +## debugging CI -Now that your workflow is running, watching it run to completion is half the -fun! Or watching it fail inexplicably for the hundredth time... which is -decidedly unfun. In any case, logs and pipeline statuses are streamed websockets -exposed by spindle, and show up in the brand new "pipelines" tab in your -repository. +We understand that debugging CI can be the worst. There are +two parts to this problem: -![pipeline logs](/static/img/pipeline-logs.png) +- CI services often bring their own workflow definition + formats and it can sometimes be difficult to know why the + workflow won't run or why the workflow definition is + incorrect +- The CI job itself fails, but this has more to do with the + build system of choice + +To mend the first problem: we are making use of git +[push-options](https://git-scm.com/docs/git-push#Documentation/git-push.txt--ooption). +When you push to a repository with an option like so: + +``` +git push origin master -o verbose-ci +``` + +The server runs a basic set of analysis rules on your +workflow file, and reports any errors: + +``` +λ git push origin main -o verbose-ci + . + . + . + . +remote: error: failed to parse workflow(s): +remote: - at .tangled/workflows/fmt.yml: yaml: line 14: did not find expected key +remote: +remote: warning(s) on pipeline: +remote: - at build.yml: workflow skipped: did not match trigger push +``` + +The analysis performed at the moment is quite basic (expect +it to get better over time), but it is already quite useful +to help debug workflows that don't trigger! ## pipeline secrets @@ -144,10 +170,12 @@ ![pipeline secrets](/static/img/pipeline-secrets.png) -The secrets themselves are stored in a configured secret manager. By default, -this is the same sqlite database that spindle uses. This is *fine* for -self-hosters. The hosted, flagship instance at https://spindle.tangled.sh -however uses [OpenBao](https://openbao.org), an OSS fork of HashiCorp Vault. +The secrets themselves are stored in a secret manager. By +default, this is the same sqlite database that spindle uses. +This is *fine* for self-hosters. The hosted, flagship +instance at https://spindle.tangled.sh however uses +[OpenBao](https://openbao.org), an OSS fork of HashiCorp +Vault. ## get started now -- tangled.sh