Monorepo for Tangled tangled.org

Fork 234

migrator service, github migration #3mvs7zorimcyt

Merged opened by

ptr.pet · 2w ago

migrator service, github migration #3mvs7zorimcyt

Merged opened by and assigned to

ptr.pet · 2w ago · targeting sv-fe

this adds:

prereqs for migrations; object format negotiation and background imports for knot2
create repo rpc gets a source kind; so it can either be a fork or an import
describeRepo gets a content field for expressing what state a repo is in (fetching, partial (lfs missing), etc)
the migrator service; which does the actual repo create rpc + repo declaration record
web client server apis for connecting to github
the web UI for github onboarding + single URL anonymous repo migration

how it works:

user authenticates to github, the session is stored on web client server side
their profile gets filled in, emails get added to deliberi and ssh keys are imported
onboarding finishes; user can choose "migrate repos" in the finish step
they get redirected to the import / migrate repo page
user authenticates the migrator service (this creates an oauth token scoped to create repo rpc + create repo record, the oauth token is stored on migrator)
user chooses repos, blabla, they can start import
each repo gets made into a task and is sent to the migrator via createTask rpc
currently since every repo is public, migrator just passes on the url to the source field of create repo rpc
- if private, migrator would first clone down the repo using the github token that the user authenticated for, and then expose a secret url, which gets passed onto the knot via source field again. once the repo is polled to be fetched by the knot, the repo is wiped from disk
- this is also the step where we would do issue / pr migration by writing COBs. i was thinking we can just do this migrator side without having to add more oauth scopes and make calls to the knot, not exactly sure if this is possible but it seemed so, and knot should just be able to import the cobs alongside the repo?
repo gets imported yadayada, with listTasks to migrator you get the state of the task and such.
...and thats it. the repos are migrated now!

the choices:

why migrator service? because embedding application logic inside knot is bad, and the xrpc surface would suck. plus knot should not be holding secrets.
why have the user oauth against the migrator? because even if we did some token grant surface between knot and migrator, we still need to be able to write to the PDS. plus its the cleaner way anyway, why add a separate auth thing...
- we could potentially hook up our tngl.sh pds here and skip the oauth screen mayhaps...
why the secret url? because then we reuse knot rpc. is it a security risk, no, at least i havent thought of anyway it could be. knot will be able to see it but the knot has to clone it and the knot owner is your host so that part is unavoidable. it has to be public so other knots can clone it. but this part should get more checking.
thats all i think? can bikeshed on the migrator lexicons...
its missing progress bar thing rn. we could add a streaming xrpc for this to the migrator maybe... i havent bothered for this pr though

i didnt test it against real github, still have to do that since it needs app and deploy of migrator and knot2, localinfra has a stub server that mocks github apis. unfortunately knot upgrade is mandatory here because of the new lexicon surface in repo.create and describeRepo

Labels

No labels

mtzsrxqz

web/github-import: styling nits

+18 -19

vyzvumky

localinfra: a fake github and the migrator for local e2e

+874 -76

tvmskyls

web: add onboarding step and new-repo entry point for importing

+4412 -259

smotpuxx

web/src/lib/api: probe repo names, import github repos, poll the knot

+2532 -201

umokwpov

web: broker the github app through the server and list installations

+1666 -16

usnoqmss

migrator: add the batch importer service and nix module

+9862

mskoqmlq

netutil, idresolver: absorb the safe connect proxy and guard did resolution, bump the indigo fork for its upstream ssrf fixes

+661 -135

psprqluk

knot2/crates/knot-xrpc: run background imports, place refs idempotently

+1335 -172

xxorwvkk

knot2/crates/knot-pack: negotiate object format when fetching a source

+495 -82

rolnvqxy

lexicons/repo: add source kind to create and content to describeRepo

+47 -7

Base

Comments

v0 2w ago

boltless.me

2w ago

migrator would first clone down the repo using the github token that the user authenticated for, and then expose a secret url, which gets passed onto the knot via source field again

This sounds like a knot.
Is migrator service a knot?
Maybe migration == "special knot designed for long-running clone jobs and auto migrate repo to different knot once done"?

Migrator can be literally a knot heavily integrated to svelte app, just like our primary PDS.

This way, we don't need extra oauth scopes for issue/PR imports because migrator service itself has all admin access against the repository. "secret url" hack becomes more generic problem of: "how do we migrate private repos between knots".

Just dumping my thoughts. We can do this later as it's not decentralized yet and fully controlled by us.

we still need to be able to write to the PDS

Needing both atproto oauth flow sounds bit tedious. That means 3 separate oauth flows: atproto (initial signin) + github + atproto again (migrator).

The only record we should write on the PDS is org.tangled.repo.declaration (and maybe org.tangled.feed.subscription too.) Then, won't repo initialization can be done from svelte? We can create a repository without git capability. This state can be represented by repo not having org.tangled.git.refs (no default branch) but only has org.tangled.repo.manifest.

...and this won't really work since we need gh token for both repo listing and cloning. hmmm yeah. maybe you are right on that.

can bikeshed on the migrator lexicons...

as long as it's only used internally I think it's fine.

ptr.pet

1w ago

Migrator can be literally a knot heavily integrated to svelte app, just like our primary PDS.

i mean its not really a knot though? its not supposed to expose any of the knot surface, thats not its function. it just does a git clone (for private repos only, public repos shortcircuit to just passing the url directly to the target knot).

the knot repo create part is easily "integratable", all we have to do is just pass a service did to knot2 for the migrator and we can have migrator do whatever call it wants via service auth against knot1.tangled.sh, but it doesnt mean we can get rid of the oauth screen for everything, other knots will still exist, so we need that handling still.

but the problem there is then we would have to do the repo declaration in the web client if we want to get rid of the oauth screen... i guess its fine though no? describeRepo already has fetching status. so the only place where a status doesnt exist is before repo create is ran on the knot, so the time between when we make the declaration to when the migrator makes the call against knot1

boltless.me

1w ago

i mean its not really a knot though? its not supposed to expose any of the knot surface, thats not its function.

But in same time, we already have knot->knot repo migration path. We don't have to make a new service type involved in repository creation. If migrator service is literally a knot that temporarily holds the repository, data management becomes way simpler. Instead of Knot serving non-git data while migrator preparing git data in background, there is always one service holding any data of the repository, and it has full admin control about repository's identity. Migrator service don't need cooperation of knot while that "not ready" state since target knot won't be involved in that point of time.

For example if repository state can be controlled by knot, it should wait until migrator's job to finish which is extra point of failure between two services. In contrast, if it's fully done by migrator service, it can do whatever from its own failure path. It can control the identity, repository state, whatever because it is literally the knot of the repository when it fails.

Again, not urgent, but one thing to think about. Maybe worth refactor considering we need private repository in 1.0.

ptr.pet

1w ago

i mean for the purposes of migration i guess it can implement the subset required for knot <-> knot migrations? but i dont think the code would change that much? migrator xrpc surface would still stay as is, etc. so it should be fine. also this works fine for private repos anyway since i imagine we'll have create private repo in the repo create xrpc so it would just be a matter of a flag?

v1 1w ago

oppi.li

1w ago

bunch of stuff added to static dir, are they necessary?

v2 1w ago

v3 1w ago

ptr.pet

1w ago

bunch of stuff added to static dir, are they necessary?

nope it was accidental, fixed it

v4 1w ago

migrator service, github migration #3mvs7zorimcyt

Labels

Versions