diff --git a/NOTES.md b/NOTES.md index b41b414..c8d061a 100644 --- a/NOTES.md +++ b/NOTES.md @@ -2,7 +2,7 @@ Loose ends and follow-ups for imgiter. -## Pod SSH: sshd is NOT running by default (2026-09-12) +## Pod SSH: sshd not running by default (2026-09-12; root cause found & solved 2026-09-12) **Symptom:** every non-interactive access path fails while the pod itself is healthy — `runpodctl ssh info` ip:port gives *connection refused*, `exec` @@ -13,26 +13,51 @@ Only the console **web terminal** and the `ssh -@ssh.runpod.io` gateway (PTY required — plain `ssh host cmd` is rejected with "Your SSH client doesn't support PTY"; scripted use needs a pty wrapper + fed commands) work. -**Cause:** the pod image does not start sshd, and ships without host keys. +**Cause (corrected — the original "the pod image does not start sshd" was +wrong):** the base image's `/start.sh` *does* set up sshd — host keys, +`authorized_keys`, `service ssh start` — but its `setup_ssh` block is gated +on `$PUBLIC_KEY`, which Runpod injects from the **account's registered SSH +keys at pod start**. This pod had booted with none registered, so `PUBLIC_KEY` +was empty and the whole block no-opped. Not an image defect; baking keys +into the image was considered and rejected (README_DEFERRED.md). -**Fix (on the pod, via web terminal or the gateway):** +**Fix (permanent):** register a key once, then restart the pod. Every +subsequent start brings sshd up automatically — nothing to redo manually +(procedure: README_RUNPOD.md §3, "SSH access"): + +```bash +runpodctl ssh add-key --key-file ~/.ssh/id_ed25519.pub +runpodctl ssh list-keys # confirm it's on the account +# then pod stop && pod start — keys added after boot are ignored until restart +``` + +**Manual fallback** (still valid, for a pod you cannot restart mid-run; +ephemeral — redo after every pod start): ```bash ssh-keygen -A # generates /etc/ssh/ssh_host_* keys service ssh start # "no hostkeys available -- exiting" without the line above ss -tlnp | grep :22 +echo '' >> /root/.ssh/authorized_keys # else publickey auth fails ``` -Then the `runpodctl ssh info` ip:port mapping answers. Second gotcha: the -gateway authenticates against the *Runpod account*, the in-container sshd -against `/root/.ssh/authorized_keys` — append the pubkey you connect with -(`echo '' >> /root/.ssh/authorized_keys`) or publickey auth -still fails. With that, plain `rsync -P -e "ssh -i -p "` works -and is the preferred bulk-transfer path (measured ~14 MB/s). - -**All of it is ephemeral** — host keys, authorized_keys, and the running sshd -live on the container disk and vanish on stop/restart/recreate. Redo the -two-liner after every pod start. +**Troubleshooting notes worth keeping:** + +- The **gateway** authenticates against the *Runpod account*; the in-container + sshd against `/root/.ssh/authorized_keys` — two separate trust paths, each + can break independently. A pod reachable via gateway but refused on the + ip:port mapping means in-container sshd is down (this incident); the + reverse (ip:port OK, gateway rejected) would point at the account side. +- `ssh -@ssh.runpod.io` needs a PTY — scripted use requires a + pty wrapper with fed commands; prefer the ip:port mapping for automation. +- Host keys, authorized_keys and the running sshd live on the ephemeral + container disk and vanish on stop/restart/recreate. With the account key + registered, `/start.sh` regenerates all of it at every boot. +- After a stop→start the external port is **reassigned**, and the first + `runpodctl ssh info` can report a stale port for ~90 s — retry until a + connection succeeds before assuming sshd is down. +- With sshd reachable, plain `rsync -P -e "ssh -i -p "` works and + is the preferred bulk-transfer path (measured ~14 MB/s). ## macOS `._*` AppleDouble files in pod bundles @@ -190,7 +215,7 @@ cross-model comparison. No conclusion yet on whether the model is suitable at all, or whether a different conditioning/topology is needed (`flux2-klein-9b` is a reference-image editor: no `--strength`, full 4-step regeneration). -2026-09-21: that topology redesign is now in the tree as `dltb-klein` + +2026-09-12: that topology redesign is now in the tree as `dltb-klein` + `scripts/sweep-klein.sh` (prompt-as-strength ladder, guidance probes). Later the same day the guidance-probe leg turned out to be inert for klein — see the next section. diff --git a/README_DEFERRED.md b/README_DEFERRED.md new file mode 100644 index 0000000..c819e75 --- /dev/null +++ b/README_DEFERRED.md @@ -0,0 +1,68 @@ +# Deferred / rejected image decisions + +Parked changes to the pod image (`image/Dockerfile`, published as +`refinementsystems/imgiter` on Docker Hub). Nothing here is scheduled; items +under *Rejected* are decided unless new information arrives. + +Ground rules for any future rebuild: + +- The pod template pins the image **by digest** and published tags are never + mutated, so existing pods and other users cannot break. A changed image + ships as a **new tag** (`0.2.0` for dependency/behavior changes, `0.1.1` for + metadata-only), plus a `runpodctl template update` of the image reference. +- Prefer bundling cosmetic changes into the next rebuild that is forced anyway + (i.e. whenever `uv.lock` changes). + +## Parked + +### Add `torchvision` (flips the preprocessing backend) + +NOTES.md, "Runtime log noise" item 3: without torchvision, transformers falls +back to `CLIPImageProcessorPil` — slightly slower preprocessing, same results. +Adding it to `pyproject.toml`/`uv.lock` switches the backend back, so runs +from before/after must not be mixed (reproducibility); that makes it an image +`0.2.0`. Do it only if preprocessing ever becomes a bottleneck or a +correctness question. + +### OCI labels + `EXPOSE 22` (cosmetic) + +`org.opencontainers.image.{source,description,licenses}` (ISC) for the public +Docker Hub page; `EXPOSE 22` documents the SSH port the template maps. +Trivial — ride along with the next rebuild. + +### Re-pin the base image off the rc tag + +`runpod/base:1.3.0-rc.164-ubuntu2404` is pinned by digest (immutable), so +this is not urgent; when a stable `1.3.0`+ tag exists, re-pin during the next +rebuild and re-check the uv version constraint (`>=0.12.7,<0.13.0`) at the +same time. + +### Exercise the Jupyter option + +Documented in README_RUNPOD.md §3 as untested (`JUPYTER_PASSWORD` + port +`8888/http`). Try it once on a live pod, then either bless it in the runbook +or drop the section. + +## Rejected + +### Bake SSH host keys / authorized_keys into the image + +A public image would give every user's pods identical host keys +(impersonation risk), and baked `authorized_keys` pin one person's key into a +public artifact. Unnecessary anyway: the base image's `/start.sh` generates +host keys at container start and builds `authorized_keys` from `$PUBLIC_KEY` +(account keys injected by Runpod at pod start). The 2026-09-12 "no sshd" +incident (NOTES.md) was a missing registered key at first boot, not an image +defect — see README_RUNPOD.md §3, "SSH access". + +### Bake `runpodctl` into the image + +rsync over direct SSH (automatic once account keys are registered, ~14 MB/s +measured) is the preferred transfer path; a baked CLI would duplicate it and +drift out of date. + +### Bake model weights or the HF token into the image + +~87.5 GB of weights, one model is gated (needs a per-pod `HF_TOKEN` anyway), +and the keep-all cache policy on the 150 GB disk already works +(README_RUNPOD.md §5). diff --git a/README_RUNPOD.md b/README_RUNPOD.md index b01b979..d834e5a 100644 --- a/README_RUNPOD.md +++ b/README_RUNPOD.md @@ -149,6 +149,29 @@ aws s3 cp --recursive \ ## 3. Pod setup +### The template is private — bring your own + +The `imgiter` template is **private and stays that way on purpose**: it wires +account-internal secrets by name (`{{ RUNPOD_SECRET_HF_TOKEN }}`), and other +users have no reason to use the same secret names (no leak risk either way — +secrets resolve per account — but a public template would simply not work for +others). The **image** is public (`refinementsystems/imgiter` on Docker Hub), +so anyone can run the stack without the template: + +```bash +runpodctl pod create \ + --image refinementsystems/imgiter:0.1.0 \ + --gpu-id "NVIDIA RTX 6000 Ada" \ + --container-disk-in-gb 150 \ + --ports "22/tcp" \ + --env '{"HF_TOKEN":""}' \ + --name imgiter --wait +``` + +…or recreate that as a private template of your own in the console. The rest +of this section documents the author's template as the reference +configuration. + Template `imgiter` (`04u1mmp8nf`): container disk 150 GB, no volume, env `HF_TOKEN={{ RUNPOD_SECRET_HF_TOKEN }}`, port `22/tcp`, image pinned by digest `sha256:68d934…` (tag `0.1.0`). @@ -194,6 +217,53 @@ echo "HF_TOKEN prefix=${HF_TOKEN:0:3} len=${#HF_TOKEN}" # expect hf_ and ~37 If it prints `{{...}}`, the secret did not resolve — `export HF_TOKEN=hf_...` for the session. +### SSH access (direct ssh / rsync) + +The image needs **no manual sshd setup**: the base image's `/start.sh` +generates host keys, writes `$PUBLIC_KEY` into `/root/.ssh/authorized_keys` +and starts sshd — but only when `PUBLIC_KEY` is non-empty, and Runpod injects +that variable from the **account's registered SSH keys at pod start**. A pod +that boots with none registered comes up without sshd (the console web +terminal and the `ssh -@ssh.runpod.io` gateway still work; the +gateway requires a PTY). That, not an image defect, was the cause of the +2026-09-12 "connection refused" session in NOTES.md. + +One-time setup — keys added after boot are ignored until a restart: + +```bash +runpodctl ssh add-key --key-file ~/.ssh/id_ed25519.pub +runpodctl ssh list-keys # confirm it's on the account +``` + +Then `pod stop` / `pod start` (or create a new pod). Every subsequent start +brings sshd up automatically — nothing to redo manually: + +```bash +runpodctl ssh info # ip:port + paste-ready ssh_command +rsync -P -e "ssh -i -p " ./dir/ root@:/workspace/dir/ # ~14 MB/s measured +``` + +Caveat: after a stop→start the external port is reassigned, and the first +`ssh info` can report a stale port for ~90 s — retry until a connection +succeeds. + +### Jupyter Lab (optional, untested) + +`/start.sh` also auto-starts Jupyter Lab on port 8888 (preferred dir +`/workspace`) whenever `JUPYTER_PASSWORD` is set in the pod env — zero image +change. Potentially handy for browsing `output/` frames and timelapses on a +running pod. **Not yet exercised on this pod**; to try it, set + +```text +JUPYTER_PASSWORD= +``` + +in the pod env and expose `8888/http` (template edit, or +`--ports "22/tcp,8888/http"` on create). Runpod then serves it at +`https://-8888.proxy.runpod.net`; mark the port **secure** in the +console so it requires your Runpod login instead of being open to anyone who +has the URL. + ## 4. On-pod workflow ```bash @@ -298,7 +368,9 @@ disk. gitignored `input/` directory into every bundle (tracked sample inputs live in `input_example/`). Alternatively `runpodctl send`/`scp` files into `input/` on the pod after extracting. -- `scp` works with the connection from `runpodctl pod get ` / `ssh info`. +- `scp`/`rsync` work over the direct-SSH mapping from `runpodctl ssh info` + (see [SSH access](#3-pod-setup) — register an account key once; rsync + measured ~14 MB/s). - Or `runpodctl send ` locally and `runpodctl receive ` on the pod (install runpodctl there first). - Outputs live under `output//_/`.