From b2d3164e27dae44efdaa7f4c25c7878dbf7713fe Mon Sep 17 00:00:00 2001 From: Michael Stahnke Date: Wed, 11 Feb 2026 15:34:23 -0600 Subject: [PATCH] chore: Update README, add license --- LICENSE | 21 +++ README.md | 435 +++++++++++++++--------------------------------------- 2 files changed, 140 insertions(+), 316 deletions(-) create mode 100644 LICENSE diff --git a/LICENSE b/LICENSE new file mode 100644 index 0000000..674e848 --- /dev/null +++ b/LICENSE @@ -0,0 +1,21 @@ +MIT License + +Copyright (c) 2026 Michael Stahnke, et al + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. diff --git a/README.md b/README.md index 5013c74..cca236d 100644 --- a/README.md +++ b/README.md @@ -4,87 +4,40 @@ The canonical installation of tumble is found at [http://tumble.wcyd.org](http:/ ## History -Tumble was a "Wouldn't it be cool?" project handed to [Scott Schnedier](https://github.com/sschneid) back in 2004. The idea was to create a website similar to a tumbleblog. Obviously, eventually tumblr ccame along and the rest was history. +Tumble was a "Wouldn't it be cool?" project handed to [Scott Schnedier](https://github.com/sschneid) back in 2004. The idea was to create a website similar to a tumbleblog. Eventually tumblr came along and the rest was history. -## Deployment - -Run - -`make build` - -Have a configuration file. - -See the systemd unit files in the `contrib` directory for more information. - - -## Database Support - -Tumble now supports both **MySQL** and **SQLite** databases. Choose the one that fits your needs: - -- **SQLite**: Recommended for development, testing, and small deployments. No separate database server required. -- **MySQL**: Recommended for production deployments with higher traffic. - -## Development Workflows - -### Database Migrations - -Tumble uses **GORM AutoMigrate** to manage database schemas. Migrations are automatically applied on application startup. - -- **Mechanism**: The application checks the database schema on startup and creates/updates tables to match the Go structs in `internal/data/store.go`. - -### Testing Infrastructure - -A robust test infrastructure is available for rapid development: - -- **Run Tests**: `make test-api` runs the integration tests. -- **Test Database**: `make test-db` creates a fresh, disposable SQLite database (`tumble-test.sqlite`), runs migrations, and loads sample fixtures. -- **Run with Test DB**: `make run-test` starts the application using the test database. -- **Backup/Restore**: `make backup` and `make restore` allow you to snapshot your current development database. - -To customize the test environment, you can edit `conf/config-test.yaml`. - -### Manual MySQL Verification - -To validate MySQL migration support without Docker, you can run against a local MySQL server: +The project was originally written in Perl and has since been rewritten in Go. -1. **Create Local Database/User**: +# Developer Guide - ```bash - mysql -u root -p -e "CREATE DATABASE tumble_test;" - mysql -u root -p -e "CREATE USER 'tumble'@'localhost' IDENTIFIED BY 'password';" - mysql -u root -p -e "GRANT ALL PRIVILEGES ON tumble_test.* TO 'tumble'@'localhost';" - ``` +## Quick Start -2. **Configure**: Check `conf/config-test-mysql.yaml` matches your local credentials. +This project uses [Flox](https://flox.dev) to manage its development environment. The Flox environment provides Go, make, jq, SQLite, ImageMagick, and libxml2. -3. **Run Application**: - - ```bash - bin/tumble conf/config-test-mysql.yaml - ``` +```bash +# Activate the development environment +flox activate -4. **Load Fixtures**: - ```bash - export DRIVER=mysql - export MYSQL_PASSWORD=password - ./tests/load_fixtures.sh - ``` +# Build +make build -## Quick Setup +# Configure (edit to match your environment) +cp conf/config.yaml.sqlite conf/config.yaml +vi conf/config.yaml -### 1. Configure Database and Application +# Run +make restart +``` -Tumble uses **Viper** for configuration, allowing you to configure the application using a file (`config.yaml`), environment variables, or default values. +Tumble will automatically create database tables on first startup. -**Configuration Search Paths:** +## Configuration -- `conf/` -- `htdocs/` -- Current directory (`.`) +Tumble uses [Viper](https://github.com/spf13/viper) for configuration via YAML files, environment variables, or defaults. -**Configuration File (config.yaml):** +**Configuration search paths:** `conf/`, current directory (`.`) -**For SQLite (default):** +### SQLite (recommended for development) ```yaml driver: sqlite @@ -94,7 +47,7 @@ port: 8080 request_timeout: 2s ``` -**For MySQL:** +### MySQL ```yaml driver: mysql @@ -106,317 +59,167 @@ port: 8080 baseurl: your.domain.com ``` -**Environment Variables:** +### Environment Variables -You can override any configuration value using environment variables prefixed with `TUMBLE_`. Use underscores (`_`) to access nested keys. +Override any config value with the `TUMBLE_` prefix. Use underscores for nested keys. -- `TUMBLE_PORT=9090` -- `TUMBLE_DRIVER=mysql` -- `TUMBLE_DATABASE=production_db` -- `TUMBLE_MODE=development` (Options: `development`, `production`. Default: `production`) -- `TUMBLE_EMBED_ASSETS=true` (Options: `true`, `false`. Default: `true`) -- `TUMBLE_LOGGING_LEVEL=debug` -- `TUMBLE_REQUEST_TIMEOUT=2s` (Default: `2s`) -- `TUMBLE_CLICK_SIGNING_KEY=your-secret` (Optional, enables signed click tracking) +| Variable | Description | Default | +|----------|-------------|---------| +| `TUMBLE_PORT` | Listen port | `8080` | +| `TUMBLE_DRIVER` | Database driver (`sqlite`, `mysql`) | `mysql` | +| `TUMBLE_DATABASE` | Database name or file path | | +| `TUMBLE_MODE` | `development` or `production` | `production` | +| `TUMBLE_EMBED_ASSETS` | Load assets from binary or filesystem | `true` | +| `TUMBLE_LOGGING_LEVEL` | Log level (`debug`, `info`, `warn`, `error`) | `info` | +| `TUMBLE_REQUEST_TIMEOUT` | HTTP request timeout | `2s` | +| `TUMBLE_ADMIN_SECRET` | Secret for admin API operations | | +| `TUMBLE_CLICK_SIGNING_KEY` | Secret for signed click tracking | | ### Environment Modes (`TUMBLE_MODE`) -- **development**: - - **Logging**: Text format, Debug level, Full SQL query logging. - - **Errors**: Displays detailed error messages in the browser. -- **production** (default): - - **Logging**: JSON format, Info level, Error-only SQL logging. - - **Errors**: Displays generic "Internal Server Error" message to users. +- **development**: Text-format logging, debug level, full SQL query logging, detailed error messages in the browser. +- **production** (default): JSON-format logging, info level, error-only SQL logging, generic error messages to users. ### Asset Embedding (`TUMBLE_EMBED_ASSETS`) -Controls whether templates and static assets are loaded from the embedded binary or from the filesystem. +Controls whether templates and static assets are loaded from the compiled binary or the filesystem. -- **true** (default): Assets are loaded from the compiled binary. This is the recommended setting for deployment - you only need the binary and a config file. -- **false**: Assets are loaded from the filesystem (`internal/templates/views/` and `internal/assets/`). Templates are re-parsed on every request, enabling hot-reload during development. +- **true** (default): Assets are loaded from the binary. Only the binary and a config file are needed for deployment. +- **false**: Assets are loaded from `internal/templates/views/` and `internal/assets/`. Templates are re-parsed on every request, enabling hot-reload during development. -This setting is independent of `TUMBLE_MODE`, allowing you to run in development mode (for verbose logging and detailed errors) while still using embedded assets for deployment. +This setting is independent of `TUMBLE_MODE`. -**Deployment Example:** - -```yaml -mode: development # Get detailed error messages and text logging -embed_assets: true # But still use embedded assets (no source checkout needed) -``` +## Database Support -### 2. Initialize Database +Tumble supports both **MySQL** and **SQLite** databases. -Simply start the application! Tumble will automatically detect your database type (configured in `config.yaml` or via env vars) and create the necessary tables if they don't exist. +- **SQLite**: Recommended for development, testing, and small deployments. No separate database server required. +- **MySQL**: Recommended for production deployments with higher traffic. -### 3. Configure Web Server +### Migrations -1. Update `/etc/httpd/conf.d/tumble.conf` with your server configuration -2. Disable SELinux or set proper context -3. Start httpd: +Tumble uses **GORM AutoMigrate** to manage database schemas. Migrations are automatically applied on application startup based on the Go structs in `internal/data/store.go`. -```bash -chkconfig httpd on -service httpd start -``` +### MySQL Setup -## Detailed Setup (MySQL) - -If you prefer manual MySQL setup: +To set up a MySQL database manually: ```bash -# Install MySQL -yum install mysql-server -service mysqld start -chkconfig mysqld on - -# Create database and user -mysql -u root -p < sql/sql_setup - -# Run schema -mysql -u tumble -p tumble < sql/schema.mysql +mysql -u root -p < sql/user_and_database_setup.mysql ``` -## Migration from MySQL to SQLite - -1. Export your MySQL data: - - ```bash - mysqldump -u tumble -p tumble > tumble_backup.sql - ``` - -2. Update `config.yaml` to use SQLite - -3. Run setup script: - - ```bash - perl scripts/setup_database.pl - ``` - -4. Import data (requires conversion from MySQL to SQLite format) - -See [docs/database_setup.md](docs/database_setup.md) for detailed instructions. +Then configure `conf/config.yaml` with your MySQL credentials. -## Debugging and Logging +## Development -Tumble provides comprehensive database logging to help troubleshoot connection issues and diagnose problems. +### Makefile Targets -### Automatic Database Diagnostics +| Command | Description | +|---------|-------------| +| `make build` | Build the binary | +| `make restart` | Build and restart the application | +| `make kill` | Stop the running application | +| `make test` | Run unit tests | +| `make test-api` | Run API integration tests | +| `make test-db` | Create a fresh test database with fixtures | +| `make run-test` | Run the application with the test database | +| `make backup` | Backup the current SQLite database | +| `make restore` | Restore the SQLite database from backup | +| `make fmt` | Run `go fmt` and clean up whitespace | +| `make build-linux` | Cross-compile for Linux amd64 | +| `make help` | Show all available targets | -When the application starts, it automatically: +### Testing -- **Logs connection attempts** with database details (host, database name, file paths) -- **Verifies database health** by checking for expected tables (`ircLink`, `image`, `quote`) -- **Reports table statistics** including row counts for each table -- **Warns about issues** such as: - - Missing database files (SQLite) - - Empty databases (no tables) - - Missing expected tables - - Empty tables (no data) - - File permission problems (SQLite) - - Connection failures with troubleshooting suggestions +- `make test-api` runs integration tests against a test instance. +- `make test-db` creates a fresh SQLite test database with sample fixtures. +- `make run-test` starts the application using `conf/config-test.yaml`. -### Debug Mode - -For verbose logging of all database operations, enable debug mode: +To test against MySQL, edit `conf/config-test-mysql.yaml` and run: ```bash -export TUMBLE_DEBUG=1 +bin/tumble conf/config-test-mysql.yaml ``` -With debug mode enabled, you'll see: - -- All SQL queries being executed -- Row counts for each query result -- Detailed query execution information - -**Example:** +### Logging -```bash -# Enable debug mode -export TUMBLE_DEBUG=1 - -# Start your web server or run the application -perl -I htdocs/lib htdocs/index.cgi -``` +Set `TUMBLE_MODE=development` or `TUMBLE_LOGGING_LEVEL=debug` for verbose output including SQL queries. Logs are written to `tumble.log` by default. -### Log Output Examples +## Deployment -**Successful MySQL connection:** +See the systemd unit files in the `contrib/` directory for production deployment examples. -``` -[MySQL] Attempting to connect to database 'tumble' on host 'localhost' as user 'tumble' -[MySQL] Connection SUCCESSFUL -[MySQL] Database contains 3 table(s) -[MySQL] Tables: ircLink, image, quote -``` +The recommended approach: -**SQLite with missing database file:** +1. Build the binary with `make build` (or `make build-linux` for Linux targets). +2. Create a `config.yaml` with your production settings. +3. Run the binary: `bin/tumble config.yaml` -``` -[SQLite] Attempting to connect to database file: tumble.db -[SQLite] Database file DOES NOT EXIST -[SQLite] - SQLite will create a new empty database file -[SQLite] - You will need to run the database setup script to create tables -[SQLite] Connection SUCCESSFUL -[SQLite] WARNING: Database appears to be empty (no tables found) -[SQLite] - You need to run the database setup script -``` +With `embed_assets: true` (the default), only the binary and config file are needed on the server. -**Connection failure with diagnostics:** +The production systemd units in `contrib/` use Flox to manage the MySQL service (`tumble-db.service` runs `flox activate --start-services`). See `contrib/README` for details. -``` -[MySQL] Connection FAILED: Access denied for user 'tumble'@'localhost' -[MySQL] Diagnostics: -[MySQL] - DSN: dbi:mysql:tumble;host=localhost -[MySQL] - Username: tumble -[MySQL] Troubleshooting suggestions: -[MySQL] 1. Verify MySQL server is running: systemctl status mysql -[MySQL] 2. Check credentials in config.yaml are correct -[MySQL] 3. Verify user has permissions: GRANT ALL ON tumble.* TO 'tumble'@'localhost' -``` +## API -### API Features +Full API documentation is available at `/api/docs` on a running instance, generated from `internal/assets/openapi.json`. -#### Link Submission with Duplicate Detection +### Link Submission -When submitting links via `/link/`, the API automatically detects if a URL has been previously posted and provides contextual information: +Submit links via `POST /link/`. Duplicate detection is automatic: -- **Behavior**: Links are always added to the database, even if duplicates exist - **JSON API** (`Accept: application/json` or `source=api`): - - **201 Created**: New link (first time posted) - - **208 Already Reported**: Duplicate detected, includes details of all previous submissions + - **201 Created**: New link + - **208 Already Reported**: Duplicate, includes previous submission details - **IRC Source** (`source=irc`): Returns ID with duplicate marker if applicable - - New: `"123"` - - Duplicate: `"123 (duplicate, previously posted by alice)"` -- **HTML Response**: Shows duplicate notification with original poster and timestamp - -**Example JSON Response (Duplicate):** - -```json -{ - "link_id": 456, - "is_duplicate": true, - "previous_submissions": [ - { - "link_id": 123, - "user": "alice", - "timestamp": "2026-01-15T10:30:00Z", - "title": "Example Page" - } - ] -} -``` - -For complete API documentation, visit `/docs` on your running instance or see `internal/assets/openapi.json`. - -> **Note:** The legacy `/irclink/` endpoint is still supported for backwards compatibility but `/link/` is preferred. - -#### Link Deletion +- **HTML**: Shows duplicate notification with original poster and timestamp -Links can be deleted via the API using the `DELETE` method on `/link/123` (where `123` is the link ID). This requires authentication using an admin secret. - -**Configuration:** - -Add an `admin_secret` to your `config.yaml`: - -```yaml -admin_secret: "your-random-secret-string" -``` +The legacy `/irclink/` endpoint is still supported but `/link/` is preferred. -You can also set it via environment variable: `TUMBLE_ADMIN_SECRET=your-secret` +### Link and Quote Deletion -**Usage:** +Delete links or quotes via `DELETE /link/{id}` or `DELETE /quote/{id}`. Requires authentication: ```bash -# Using X-Admin-Secret header (recommended) +# Using header (recommended) curl -X DELETE -H "X-Admin-Secret: your-secret" https://your-server/link/123 # Using query parameter curl -X DELETE "https://your-server/link/123?secret=your-secret" ``` -**Responses:** +Configure `admin_secret` in your config file or via `TUMBLE_ADMIN_SECRET`. Without it, deletion falls back to localhost-only access. -- **200 OK**: Link deleted successfully -- **400 Bad Request**: Missing or invalid ID -- **403 Forbidden**: Missing or invalid admin secret -- **404 Not Found**: Link does not exist +### Click Signature Tracking -If no `admin_secret` is configured, deletion falls back to localhost-only access for backwards compatibility. +Optional HMAC-signed URLs for verified click tracking. When `click_signing_key` is configured, links render as `/link/123?sig=abc123...` and the server validates signatures to distinguish real clicks from bots. -#### Quote Deletion +### Dead Link Detection -Quotes can be deleted via the API using the `DELETE` method on `/quote/123` (where `123` is the quote ID). This uses the same authentication mechanism as link deletion. +Tumble automatically detects dead links (4xx/5xx responses) and checks the [Wayback Machine](https://archive.org) for archived snapshots. A background job runs on startup and daily. Archive.org requests are rate-limited to 5 req/s. -**Usage:** +### Caching -```bash -# Using X-Admin-Secret header (recommended) -curl -X DELETE -H "X-Admin-Secret: your-secret" https://your-server/quote/123 - -# Using query parameter -curl -X DELETE "https://your-server/quote/123?secret=your-secret" -``` - -**Responses:** - -- **200 OK**: Quote deleted successfully -- **400 Bad Request**: Missing or invalid ID -- **403 Forbidden**: Missing or invalid admin secret -- **404 Not Found**: Quote does not exist +Link previews are cached in the database. Disable with `caching.enabled: false` in config. Invalidate manually via `GET /api/caching/invalidate?url=`. -#### Click Signature Tracking +## Project Structure -Tumble supports signed URLs for verified click tracking. When enabled, links include an HMAC signature that validates clicks came from the rendered page rather than bots or direct URL access. - -**Configuration:** - -Add a `click_signing_key` to your `config.yaml`: - -```yaml -click_signing_key: "your-random-secret-string" +``` +cmd/tumble/ Entry point +internal/ + assets/ Static assets and OpenAPI spec + config/ Configuration loading + data/ Database models and queries + handler/ HTTP handlers + templates/ HTML templates + service/ Business logic + scheduler/ Background jobs + archive/ Archive.org integration +conf/ Configuration files +contrib/ Systemd unit files +sql/ SQL setup scripts +tests/ Integration tests and fixtures ``` -Or set via environment variable: `TUMBLE_CLICK_SIGNING_KEY=your-secret` - -**How it works:** - -- When configured, links render as `/link/123?sig=abc123...` instead of `/link/123` -- The signature is an HMAC-SHA256 hash of the link ID using your secret key -- On redirect, the server validates the signature to distinguish verified clicks from unverified access -- This helps track genuine user engagement vs. crawler/bot traffic - -**Note:** If no `click_signing_key` is configured, links work normally without signatures. This feature is optional and doesn't affect basic functionality. - -#### Dead Link Detection and Archive.org Integration - -With ~20 years of links, many URLs have gone dead over time. Tumble -automatically detects dead links and checks the [Wayback Machine](https://archive.org) -for archived snapshots. - -- **Detection**: Links returning 4xx/5xx errors are flagged with an HTTP - status badge and excluded from search results. -- **Archive.org lookups**: A background job checks dead links against the - Wayback Machine Availability API on startup and daily. Newly detected - dead links are also checked on-demand. -- **UI**: When an archived snapshot exists, a "View on Archive.org" link - appears next to the error badge with the snapshot date, giving users - access to the original content. Works in both the default theme and - Scott Mode. -- **Rate limiting**: Archive.org requests are limited to 5 req/s to stay - well within API limits. - -#### Caching - -Link previews are cached in the database to reduce external requests. - -- `caching.enabled`: Set to `false` to disable server-side caching. -- To invalidate a cache entry manually: - `GET /api/caching/invalidate?url=` - -## Bugs +## License - * fix user-agent being hardy for link verification - * abstract quantity of items to be in 'hot shit' category - * Fix odd encoding bugs for web site titles - * Probably lots of others, but it has been in production for 10 years. +This project is licensed under the MIT License. See the [LICENSE](LICENSE) file for details. -- 2.51.2