ContentAPI Crawler
qcscrawl docs contentapi-contract.md
19 kB
Markdown
at main

ContentAPI contract evidence #

This is the Phase 0 wire-contract record for qcscrawl. The command name is qcscrawl. It describes evidence captured from the configured server, not assumptions about an arbitrary ContentAPI deployment.

Capture identity #

  • Target: https://qcs.shsbs.xyz (base URL has no /api suffix).
  • Capture: 2026-09-01T19:08:30Z, anonymous, read-only.
  • /api/status returned HTTP 200, Content-Type: application/json; charset=utf-8, 307 bytes. Raw-body SHA-256: 790185eae96f13f5ec6b7d909d8c588c034ed6d5e618aa67eae25006d97dbdb1.
  • /api/request/about returned HTTP 200, Content-Type: application/json; charset=utf-8, 24,523 bytes. Raw-body SHA-256: 3e3e875f8c92eb71ca8a31ea08a2f536499e2d31518412e5a215b30f111c2ce4.
  • The Phase 0 status/about captures were deliberately removed from the public repository because they were evidence only, not runtime fixtures or test inputs. Their capture metadata and SHA-256 values remain recorded in this document.
  • No bearer token was configured or used. Authenticated visibility is therefore not observed. No writes, deletes, uploads, or credentials were attempted.

The status body identified version 3.3.2.744, app name contentapi, environment Production, repository https://github.com/randomouscrap98/contentapi, and the public bug-report/contact links. The process start/runtime values are deployment observations and are not protocol guarantees.

Pinned evidence #

Live probe requests and results #

All POST probes used Content-Type: application/json, an Accept: application/json header, and no Authorization header. They were read-only requests to /api/request. Response hashes below are SHA-256 of the complete raw HTTP response body; timing fields and generated permission query text make each response capture-specific.

Probe Request intent HTTP/result Raw response SHA-256
message_min message, id > @after_id, order: id, limit 1, ordinary fields 200; 1 row fbb5d220ae718e6d8b9b4967958ec9c2242c66def667b4083b55697b47aaed39
message_extras same, plus values, uidsInText 200; 1 row; both extras returned 36e1da5a5a7a228445d1037c4fc6cbf11086eb425b044a428ec1c98fa43f7f73
content_extras explicit core fields plus permissions, values, keywords, lastRevisionId 200; 1 row; all requested fields returned c831b98c9f02c5304f66af84701ba6de656289e5207ff76b6aa3892e5c303a64
user_public public UserView fields, keyset query 200; 1 row 4f7377a4014fd4acd3f52297682fea0138569090b4c4f2a8aa82dcf2c7a0718d
activity_public ActivityView fields, keyset query 200; 1 row f02b6ac38fb570573386dbe365c17f6469369dbf796336105f3b127187c5bb98
message_max message fields id,contentId, limit 1001 200; 1000 rows c4fe78569b04b9b73b5b1fda01d714fd7cb95839260994abad983a71a8dd1634
message_after_7388 same message keyset query after cursor 7388, limit 10 200; IDs 7389..7398 in ascending order ba288267a7565bc46f589f1fd7376c089e7ef17a18ad30bdff674ab03441ba5f
deleted_message message query deleted = @deleted_flag, deleted_flag: 1, no !notdeleted() 200; 1 deleted row returned 230e6f2c90fd7e6af7cd7c6f533df5e768889833d4b25d646596fdd075ff22b5
message_max with only id deliberately invalid: provider-injected permission query needs selected contentId 400; no cursor use fd582fcb045900201faf9df8f49779ba1a17f17775f9cfabfcbe4628eedd6c19
deleted query with literal deleted = 1 deliberately invalid live query syntax; parameter form works 400; use @parameter instead 082b0f86bff297d0ccaccd73cd8a18039080b87f75d9c76f81cd58c51e96fa3d

The first message-extra row had values and uidsInText; the first content-extra row had all four requested expensive/derived fields. The anonymous deleted-row probe returned a row with deleted: 1, empty text, and redacted/zero creator in that response. These are observations from this capture, not completeness claims about all deleted or private rows.

The request response envelope observed was lower camel case with totalTime, nonDbTime, requestUser: null, an echoed search, databaseTimes, and objects. The server added permission and recipient restrictions to the echoed query. objects is keyed by each request's name (messages, content, etc.). The source DTOs are RequestResponse and GenericSearchResult.

Contract recorded from live /api/request/about #

The live response advertised these request types, exactly matching the pinned source and SBS2 snapshot:

user, content, message, activity, watch, adminlog, uservariable, message_aggregate, activity_aggregate, content_engagement, ban, keyword_aggregate, message_engagement, userrelation.

Public view field names also matched source/SBS2 for the four archived entities:

  • user: id, username, avatar, special, type, createDate, createUserId, super, registered, deleted, groups, usersInGroup.
  • content: id, deleted, createUserId, createDate, contentType, name, parentId, text, literalType, meta, description, hash, permissions, values, keywords, engagement, lastCommentId, commentCount, popScore1, lastActionDate, watchCount, keywordCount, lastRevisionId.
  • message: id, contentId, createUserId, createDate, text, values, engagement, editDate, editUserId, edited, deleted, module, receiveUserId, uidsInText.
  • activity: id, contentId, userId, date, message, action.

Live expensive metadata is important for field selection. message.values and message.engagement are level 2; message.uidsInText is level 1. content.permissions, values, keywords, and engagement are level 2; lastRevisionId is level 1; commentCount and popScore1 are level 3. User groups are level 2. Explicit field lists above worked without asking for engagement. Do not use fields: * for archive requests.

Live code dictionaries matched the pinned Enums.cs:

  • InternalContentType: 0 none, 1 page, 2 module, 3 file, 4 userpage, 5 system.
  • UserAction: 1 create, 2 read, 4 update, 8 delete (bit values, not ordinal labels).
  • UserType: 1 user, 2 group.
  • Live metadata additionally advertised BanType 0/1/2/4 and AdminLogType 0..20; EventType is from the live-event source, not a request type.

Raw content/file endpoints #

The pinned server distinguishes textual content from uploaded files. ContentController.cs serves GET /api/Content/raw/{hash}, looks up a ContentView by hash, and emits an ETag combining hash and lastRevisionId. FileController.cs serves GET /api/File/raw/{hash} with an ETag based on the file hash. SBS2 calls its route parameter an ID in some application code, but the pinned server route is named hash; qcscrawl does not download either endpoint in the initial sync and must not conflate them when media support is added.

Wire observations and implementation decisions #

  1. Casing and defaults. Send lower-camel JSON (values, requests, name, type, fields, query, order, limit, skip, expensive). The server response echoed skip: 0 and expensive: false even when omitted.
  2. Keyset pagination. Use a named request, a selected id and any provider-injected fields needed by the entity (contentId for messages/activities), query: "id > @after_id", order: "id", and a bounded limit. Verify strictly increasing IDs before committing. The live 1000-row request returned IDs 128..7388 in increasing order; the next request returned 7389..7398.
  3. Result-set limit. The pinned source defaults GenericSearcherConfig.MaxIndividualResultSet to 1000. The live limit: 1001 probe returned exactly 1000 rows. Treat 1000 as the observed ceiling and keep the limit configurable below it; re-probe after deployment changes.
  4. Parameters, not bare numeric literals. The live parser rejected deleted = 1 with a lexical error but accepted deleted = @deleted_flag with value 1. Always bind cursor and filter values through values; do not write a local query parser.
  5. Nulls and numbers. Live rows contained JSON null for nullable values (module, description, message) and numeric 0/1 for fields advertised as bool (deleted, edited, super, registered). Date strings varied in lexical precision (...Z and ...000Z). Preserve raw bytes and decode numeric booleans with a compatibility decoder for derived columns.
  6. Arbitrary extras. Explicit values/uidsInText and content permissions/values/keywords/lastRevisionId worked anonymously. Keep arbitrary values as raw JSON; do not round-trip through map[string]any and re-marshal.
  7. Permission filtering. Anonymous responses had requestUser: null; the server injected permissionlimit into content-related searches and receiveuserlimit into message searches. Absence from a response is not proof of provider deletion. No authenticated comparison was possible.
  8. Deletion. A query without !notdeleted() returned an explicitly deleted row. Keep deletion markers. Do not infer deletion from local/provider omission.
  9. Response timing. databaseTimes is keyed by request name and also included a <name>_extras measurement for derived fields. It is diagnostic only.
  10. Raw fidelity. Read the complete body, extract each json.RawMessage object, and store the exact object bytes as a BLOB plus SHA-256. Typed/index fields are derived. Preserve unknown keys, null-vs-missing distinctions, numeric spelling, escaping, and timestamp spelling.

Comparison matrix #

Links in the column headings are pinned revisions: ContentAPI source, ContentAPI tests, SBS2, and ABOUT.js.

Concept ContentAPI source ContentAPI tests SBS2 ABOUT.js Live server Decision
Request envelope SearchRequests: values, requests Chained request fixtures RequestParameter same lower camel shape Same Accepted; response echoed envelope Emit lower camel values/requests
Response envelope RequestResponse adds timing/user to GenericSearchResult Generic search assertions Consumes objects/errors Schema snapshot 200 responses had totalTime, nonDbTime, requestUser, search, databaseTimes, objects Decode envelope flexibly; preserve raw objects
Named chaining RequestController + SearchBase appends named result sets GenericSearchDbTests Comments view chains replies, content, user about text advertises chaining Not needed for single-page probes; endpoint accepted standard requests Use only where useful; no local parser
Request types 14 enum names Enum/query tests Same list Same list Same list Validate against live about; archive four public types
Message fields MessageView has 14 fields View/query behavior Message object has same fields Same Same; extras returned Explicit field list; preserve extras/raw
Content fields ContentView has core + extras QueryBuilder expensive-field tests Same Same Same; selected extras returned Explicit non-engagement list
User fields UserView public fields Query/view tests Same Same Same Never request private DB model fields
Activity fields ActivityView six fields History/search tests Activity handling Same Same Scan by id, retain action integer
RequestType enum C# enum Covered by query tests Snapshot Snapshot Exact match Keep provider strings
InternalContentType Enums.cs 0,1..5 Enum behavior Uses numeric contentType Codes 0,1..5 Exact match Store integer; label at display
UserAction Enums.cs bit values 1/2/4/8 History tests Uses action values Codes 1/2/4/8 Exact match Preserve integer, including unknown future values
UserType Enums.cs 1/2 User query tests Uses numeric type Codes 1/2 Exact match Preserve integer
EventType Live/EventType.cs 0..5 Live queue tests Socket event handling Codes 0..5 Not probed (not needed for polling) Future tail only; not a durable cursor
deleted visibility !notdeleted() is an optional macro Macro tests UI normally filters Advertises macro Explicit deleted = @deleted_flag returned one row without macro Do not add !notdeleted() to archive scans
values / uidsInText Extra fields added by GenericSearcher Extra-field behavior Reads values.replyingTo, uidsInText Both fields Both returned in explicit message probe Retain arbitrary raw values; derive reply ID only
permissions / lastRevisionId Expensive ContentView extras Expensive selection tests Requests permissions/content values Same All requested content extras returned Request permissions/values/keywords/lastRevisionId, not aggregates
replyingTo Generic values has no fixed keys No fixed SBS key entity.js reads message.values.replyingTo Not a field Not present in first sample; values returned Parse defensively from raw values
receiveUserId MessageView field + receive limit Permission/search tests Preserved by frontend Same Anonymous sample was 0 Preserve and classify nonzero as private candidate
Maximum limit GenericSearcher cap 1000 Search limit tests request.js says always 1000 No separate guarantee limit 1001 yielded 1000 Cap requests at verified 1000; re-probe
Order syntax QueryBuilder supports _desc Query/order tests Sets order Same order: id ascending worked Require monotonic ascending IDs
Chaining syntax @previousRequest.field GenericSearchDbTests Used extensively Advertised Not required in evidence page Generate only validated provider syntax
Retry-After Not server contract in source files above Not a search semantic request.js handles 429 header, 408/502/524 Not in ABOUT.js No error probe sent Implement bounded retries; honor header
Live checkpoints LiveEventQueue cache/checkpoint comments EventQueueTest socket.js handles ExpiredCheckpoint/restart Not relevant WebSocket not probed Polling is durable; tail is future optimization
Raw content/file route ContentController uses /api/Content/raw/{hash}; FileController uses /api/File/raw/{hash} Not a search semantic SBS2 uses /api/File/raw/{id} Not in ABOUT.js Not probed Keep routes distinct; defer downloads and treat route parameter as provider hash until live evidence

Deliberate limitations #

  • There is no authenticated probe. A token was not available and was not sought. The contract therefore says nothing about additional private/permission-restricted visibility.
  • No websocket probe was made. The pinned sources document checkpoint expiry/server restart behavior; live tailing is out of Phase 0.
  • POST entity response bodies were inspected in memory for evidence but not committed because they contain user-generated/public entity data. Only public status/about fixtures are committed.
  • One successful deleted query proves endpoint behavior, not complete historical deleted-row coverage. Permission filtering and redaction remain possible.
  • Live observations are tied to version 3.3.2.744 and the capture date. /api/request/about hash changes require compatibility validation and a new evidence capture.