Search Engine Integrations: Google Search Console & IndexNow #

This document outlines Tangled's search engine discovery mechanisms, sitemap architecture, verification methods, and instant re-indexing integrations.

1. Dynamic XML Sitemap (TAN-737 & TAN-745) #

Tangled serves an automated, dynamically generated XML sitemap adhering to the standard Sitemap Protocol (0.9).

URL: https://tangled.org/sitemap.xml
Implementation: web/src/routes/sitemap.xml/+server.ts
Cache-Control: public, max-age=3600, s-maxage=3600 (1-hour CDN/browser caching)
Specification Limit: Hard cap at 50,000 URLs per sitemap specification.

URL Structure & Priority #

Core Static Pages:
- https://tangled.org/ — priority 1.0, changefreq daily
- https://tangled.org/about — priority 0.8, changefreq weekly
Public Profiles:
- https://tangled.org/{handle} — priority 0.7, changefreq daily
- Dynamically gathered from trending actors (sh.tangled.actor.getTrending) and public timeline events (sh.tangled.feed.getTimeline).
Public Repositories:
- https://tangled.org/{handle}/{repo} — priority 0.8, changefreq daily
- Includes <lastmod> derived from the repository's creation timestamp.
- Paginates repository lists for discovered users via sh.tangled.repo.listRepos with cursor-based pagination.

Resilience & Graceful Degradation #

If upstream backend services (Bobbin / XRPC) are temporarily unreachable, degraded, or timing out, the endpoint catches errors silently and serves the static core URLs cleanly with HTTP 200, ensuring search engine crawlers are never served a 5xx error.

2. Google Search Console Verification (TAN-746) #

Google Search Console (GSC) is used to monitor search performance, coverage, indexing status, and submit sitemaps.

Verification Methods #

Tangled supports two primary verification methods:

HTML Meta Tag Verification: Add the verification tag provided by Google Search Console into the <head> of web/src/app.html:
```
<meta name="google-site-verification" content="YOUR_GSC_VERIFICATION_TOKEN" />
```
DNS TXT Record (Recommended for domain-level property): Add a TXT record to the DNS configuration for tangled.org:
```
Record Type: TXT
Host: @
Value: google-site-verification=YOUR_GSC_VERIFICATION_TOKEN
```

Sitemap Submission #

Log into Google Search Console.
Select the tangled.org property.
In the left navigation, click Indexing > Sitemaps.
Under "Add a new sitemap", enter sitemap.xml and click Submit.
Google will periodically poll and re-index URLs from https://tangled.org/sitemap.xml.

3. IndexNow Instant Indexing Protocol (TAN-747) #

IndexNow allows websites to immediately notify participating search engines (including Microsoft Bing, Yandex, Seznam, and Naver) whenever content is created, updated, or deleted. Submitted URLs are crawled and reflected in search results within minutes.

Verification Key #

Key: d76fae270607bfc761b35c14f3182c9e
Key Location: https://tangled.org/d76fae270607bfc761b35c14f3182c9e.txt
Static File: web/static/d76fae270607bfc761b35c14f3182c9e.txt

Search engines verify domain ownership by requesting https://tangled.org/{key}.txt and verifying that the body matches the key.

IndexNow Submission Utility #

Implementation located at web/src/lib/server/indexnow.ts.

API Usage #

import {
  submitToIndexNow,
  notifyProfileCreated,
  notifyRepoCreated
} from "$lib/server/indexnow";

// Submit an arbitrary list of changed URLs:
await submitToIndexNow([
  "https://tangled.org/alice.tangled.sh",
  "https://tangled.org/alice.tangled.sh/my-repo"
]);

// Helper when a new public profile is created:
await notifyProfileCreated("alice.tangled.sh");

// Helper when a new public repository is created:
await notifyRepoCreated("alice.tangled.sh", "my-repo");

Protocol Payload #

Requests are sent via HTTP POST to https://api.indexnow.org/indexnow:

{
  "host": "tangled.org",
  "key": "d76fae270607bfc761b35c14f3182c9e",
  "keyLocation": "https://tangled.org/d76fae270607bfc761b35c14f3182c9e.txt",
  "urlList": [
    "https://tangled.org/alice.tangled.sh/my-repo"
  ]
}

Fault-Tolerance & Non-Blocking Design #

All IndexNow helper calls catch network and HTTP status errors internally and log warnings to the console without throwing exceptions. User-facing mutation flows (such as repository creation or profile setup) will never fail or be delayed if the external IndexNow API is slow or unavailable.