# Search Engine Integrations: Google Search Console & IndexNow This document outlines Tangled's search engine discovery mechanisms, sitemap architecture, verification methods, and instant re-indexing integrations. --- ## 1. Dynamic XML Sitemap (TAN-737 & TAN-745) Tangled serves an automated, dynamically generated XML sitemap adhering to the standard Sitemap Protocol (0.9). - **URL**: `https://tangled.org/sitemap.xml` - **Implementation**: `web/src/routes/sitemap.xml/+server.ts` - **Cache-Control**: `public, max-age=3600, s-maxage=3600` (1-hour CDN/browser caching) - **Specification Limit**: Hard cap at 50,000 URLs per sitemap specification. ### URL Structure & Priority 1. **Core Static Pages**: - `https://tangled.org/` — priority `1.0`, changefreq `daily` - `https://tangled.org/about` — priority `0.8`, changefreq `weekly` 2. **Public Profiles**: - `https://tangled.org/{handle}` — priority `0.7`, changefreq `daily` - Dynamically gathered from trending actors (`sh.tangled.actor.getTrending`) and public timeline events (`sh.tangled.feed.getTimeline`). 3. **Public Repositories**: - `https://tangled.org/{handle}/{repo}` — priority `0.8`, changefreq `daily` - Includes `` derived from the repository's creation timestamp. - Paginates repository lists for discovered users via `sh.tangled.repo.listRepos` with cursor-based pagination. ### Resilience & Graceful Degradation If upstream backend services (Bobbin / XRPC) are temporarily unreachable, degraded, or timing out, the endpoint catches errors silently and serves the static core URLs cleanly with HTTP 200, ensuring search engine crawlers are never served a 5xx error. --- ## 2. Google Search Console Verification (TAN-746) Google Search Console (GSC) is used to monitor search performance, coverage, indexing status, and submit sitemaps. ### Verification Methods Tangled supports two primary verification methods: 1. **HTML Meta Tag Verification**: Add the verification tag provided by Google Search Console into the `` of `web/src/app.html`: ```html ``` 2. **DNS TXT Record (Recommended for domain-level property)**: Add a TXT record to the DNS configuration for `tangled.org`: ```text Record Type: TXT Host: @ Value: google-site-verification=YOUR_GSC_VERIFICATION_TOKEN ``` ### Sitemap Submission 1. Log into [Google Search Console](https://search.google.com/search-console). 2. Select the `tangled.org` property. 3. In the left navigation, click **Indexing** > **Sitemaps**. 4. Under "Add a new sitemap", enter `sitemap.xml` and click **Submit**. 5. Google will periodically poll and re-index URLs from `https://tangled.org/sitemap.xml`. --- ## 3. IndexNow Instant Indexing Protocol (TAN-747) IndexNow allows websites to immediately notify participating search engines (including Microsoft Bing, Yandex, Seznam, and Naver) whenever content is created, updated, or deleted. Submitted URLs are crawled and reflected in search results within minutes. ### Verification Key - **Key**: `d76fae270607bfc761b35c14f3182c9e` - **Key Location**: `https://tangled.org/d76fae270607bfc761b35c14f3182c9e.txt` - **Static File**: `web/static/d76fae270607bfc761b35c14f3182c9e.txt` Search engines verify domain ownership by requesting `https://tangled.org/{key}.txt` and verifying that the body matches the key. ### IndexNow Submission Utility Implementation located at `web/src/lib/server/indexnow.ts`. #### API Usage ```typescript import { submitToIndexNow, notifyProfileCreated, notifyRepoCreated } from "$lib/server/indexnow"; // Submit an arbitrary list of changed URLs: await submitToIndexNow([ "https://tangled.org/alice.tangled.sh", "https://tangled.org/alice.tangled.sh/my-repo" ]); // Helper when a new public profile is created: await notifyProfileCreated("alice.tangled.sh"); // Helper when a new public repository is created: await notifyRepoCreated("alice.tangled.sh", "my-repo"); ``` #### Protocol Payload Requests are sent via HTTP POST to `https://api.indexnow.org/indexnow`: ```json { "host": "tangled.org", "key": "d76fae270607bfc761b35c14f3182c9e", "keyLocation": "https://tangled.org/d76fae270607bfc761b35c14f3182c9e.txt", "urlList": [ "https://tangled.org/alice.tangled.sh/my-repo" ] } ``` #### Fault-Tolerance & Non-Blocking Design All IndexNow helper calls catch network and HTTP status errors internally and log warnings to the console without throwing exceptions. User-facing mutation flows (such as repository creation or profile setup) will never fail or be delayed if the external IndexNow API is slow or unavailable.