diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml index 35cf477..b35f9c1 100644 --- a/.pre-commit-config.yaml +++ b/.pre-commit-config.yaml @@ -71,11 +71,12 @@ repos: pass_filenames: false - id: lexicon-generate - name: Generate Lexicons + name: Generate Lexicons (files ignored by .gitignore) entry: pnpm lex:gen-server language: system files: lexicons/.*\.json$ pass_filenames: false + always_run: false # Optional: Additional checks - repo: local diff --git a/docs/generated-files-strategy.md b/docs/generated-files-strategy.md new file mode 100644 index 0000000..f4e4ec6 --- /dev/null +++ b/docs/generated-files-strategy.md @@ -0,0 +1,265 @@ +# Generated Files Strategy + +This document explains our approach to handling generated files in the Teal project, specifically for lexicon-generated TypeScript files. + +## TL;DR + +**Generated files are NOT tracked in git** - they are ignored by `.gitignore` and regenerated automatically when needed. + +## The Problem + +Generated files (like TypeScript types from lexicon schemas) present a common dilemma: + +### Option A: Track Generated Files in Git +**Pros:** +- Immediate availability after clone +- Clear diff of what changed +- No build step required for basic usage + +**Cons:** +- Merge conflicts in generated code +- Bloated git history with auto-generated changes +- Risk of generated files becoming out of sync with source +- Larger repository size +- Confusing diffs mixing human and generated changes + +### Option B: Ignore Generated Files (Our Choice) +**Pros:** +- Clean git history with only human changes +- No merge conflicts in generated code +- Smaller repository size +- Generated files always match current source +- Clear separation between source and generated code + +**Cons:** +- Requires build step after clone +- Not immediately usable without generation + +## Our Implementation + +We chose **Option B** for these reasons: + +### 1. Automatic Generation Pipeline + +Generated files are created automatically in multiple scenarios: + +```bash +# After fresh install +pnpm install # → triggers postinstall → lex:gen-server + +# Before builds +pnpm turbo build --filter=@teal/amethyst # → generates lexicons first + +# During development +pnpm lex:watch # → regenerates on source changes + +# In Docker builds +docker build ... # → includes generation step + +# Via git hooks +git commit # → validates and regenerates if lexicon files changed +``` + +### 2. Zero Developer Friction + +Developers don't need to think about generated files: + +```bash +# This just works - lexicons generated automatically +git clone +pnpm install +pnpm build +``` + +### 3. Build System Integration + +Turbo ensures lexicons are always fresh: + +```json +{ + "@teal/amethyst#build": { + "dependsOn": ["@teal/lexicons#lex:gen-server"] + } +} +``` + +### 4. Git Hook Validation + +When lexicon source files change, hooks: +1. Validate lexicon syntax +2. Regenerate TypeScript files +3. Ensure generation succeeds +4. **Don't stage generated files** (they remain ignored) + +## File Patterns + +### Tracked (Source Files) +``` +lexicons/fm.teal.alpha/*.json ✅ Tracked +packages/lexicons/package.json ✅ Tracked +packages/lexicons/lex-gen.sh ✅ Tracked +``` + +### Ignored (Generated Files) +``` +packages/lexicons/src/ ❌ Ignored (.gitignore) +services/types/src/ ❌ Ignored (.gitignore) +``` + +### .gitignore Entry +```gitignore +# generated lexicons +# js lexicons +packages/lexicons/src +# rust lexicons (types :))) +services/types/src +``` + +## Benefits in Practice + +### Clean Git History +```bash +# Only meaningful changes show up in git log +commit abc123: feat: add new actor profile fields +commit def456: fix: update feed lexicon validation +``` + +Instead of: +```bash +commit abc123: feat: add new actor profile fields +commit abc124: [auto] regenerate lexicons +commit abc125: fix: regenerated lexicon formatting +commit abc126: merge conflict in generated files +``` + +### No Merge Conflicts +When multiple developers change lexicons, git only needs to merge the source JSON files, not generated TypeScript. + +### Always Fresh +Generated files always match the current lexicon sources - no risk of drift. + +### Faster CI/CD +CI systems generate files once and use them, rather than pulling large generated file diffs. + +## Developer Workflow + +### First Time Setup +```bash +git clone +pnpm install # Generates lexicons automatically +pnpm dev # Ready to develop +``` + +### Making Lexicon Changes +```bash +# Edit lexicon files +vim lexicons/fm.teal.alpha/actor/profile.json + +# Validate and regenerate (automatic via git hooks) +git add . +git commit -m "feat: add profile status field" + +# Or manually +pnpm lex:validate +pnpm lex:gen-server +``` + +### Checking Generated Output +```bash +# View generated files (not tracked) +ls packages/lexicons/src/types/ + +# Regenerate if needed +pnpm lex:gen-server +``` + +## CI/CD Considerations + +### GitHub Actions +Our workflows automatically handle generation: + +```yaml +- name: Install dependencies + run: pnpm install # Triggers postinstall generation + +- name: Build applications + run: pnpm build # Triggers lexicon generation via Turbo +``` + +### Docker Builds +```dockerfile +# Generate lexicons during build +RUN pnpm install +RUN pnpm lex:gen-server +RUN pnpm run build:web +``` + +## Troubleshooting + +### "Module not found" errors +```bash +# Regenerate lexicons +pnpm lex:gen-server + +# Check if files were created +ls packages/lexicons/src/ +``` + +### After switching branches +```bash +# Regenerate for new lexicon state +pnpm lex:gen-server +``` + +### Fresh environment setup +```bash +# This should be all you need +pnpm install +``` + +## Comparison with Other Projects + +### Projects That Track Generated Files +- **Protocol Buffers in some repos** - Often track `.pb.go` files +- **OpenAPI generators** - Sometimes track generated client code +- **GraphQL codegen** - Mixed approaches + +### Projects That Ignore Generated Files +- **Create React App** - Ignores build output +- **Next.js** - Ignores `.next/` directory +- **Rust projects** - Ignore `target/` directory +- **Our approach** - Ignore `packages/lexicons/src/` + +## Alternative Approaches Considered + +### 1. Separate Generated Files Repo +- **Pro**: Clean main repo +- **Con**: Complex CI/CD, dependency management nightmare + +### 2. Git Submodules for Generated Code +- **Pro**: Separation of concerns +- **Con**: Submodule complexity, versioning issues + +### 3. Package Registry for Generated Code +- **Pro**: Versioned, distributed +- **Con**: Build complexity, circular dependencies + +### 4. Build-time Generation Only +- **Pro**: Always fresh +- **Con**: Slower builds, requires build for development + +## Conclusion + +Our strategy of **ignoring generated files** with **automatic regeneration** provides: + +1. **Clean git history** - Only human changes tracked +2. **Zero friction** - Developers don't manage generated files +3. **Always consistent** - Generated files match current source +4. **Robust pipeline** - Multiple generation triggers ensure availability +5. **CI/CD friendly** - Clean, predictable builds + +This approach scales well with team size and project complexity while maintaining developer productivity and code quality. + +--- + +**Key Principle**: *Source of truth is the lexicon JSON files. Everything else is derived and regenerated automatically.* \ No newline at end of file diff --git a/docs/git-hooks-setup.md b/docs/git-hooks-setup.md index a15f90d..b6040cf 100644 --- a/docs/git-hooks-setup.md +++ b/docs/git-hooks-setup.md @@ -18,6 +18,10 @@ We provide two approaches for setting up git hooks: - ⚠️ **Console.log detection** (warning only) - ⚠️ **TODO/FIXME comments** (warning only) +### Lexicon Files +- ✅ **Lexicon validation** - Schema validation for lexicon JSON files +- ✅ **Lexicon generation** - Regenerates TypeScript types (files remain ignored by .gitignore) + ### Rust Files - ✅ **cargo fmt** - Code formatting - ✅ **cargo clippy** - Linting with warnings as errors @@ -148,6 +152,8 @@ The hooks use existing npm scripts from `package.json`: - `pnpm rust:clippy` - Rust linting - `pnpm prettier --write` - JavaScript/TypeScript formatting - `pnpm biome check --apply` - Biome linting and formatting +- `pnpm lex:validate` - Lexicon schema validation +- `pnpm lex:gen-server` - TypeScript type generation from lexicons ## Troubleshooting @@ -165,7 +171,12 @@ The hooks use existing npm scripts from `package.json`: - Run `pnpm rust:fmt` and `pnpm rust:clippy` manually - Fix clippy warnings or adjust clippy configuration -4. **Hook is too slow:** +4. **Generated files ignored warning:** + - This is expected behavior - generated lexicon files are ignored by .gitignore + - Only source lexicon JSON files should be committed + - Generated TypeScript files are recreated automatically + +5. **Hook is too slow:** - Use pre-commit framework for better performance - Consider running lighter checks in pre-commit and full checks in CI @@ -259,6 +270,11 @@ Configure your IDE to: 5. **CI/CD integration:** Run the same checks in your CI pipeline +6. **Generated files approach:** Remember that generated files are ignored by .gitignore + - Only commit source lexicon JSON files + - Generated TypeScript files are automatically recreated + - This keeps git history clean and avoids merge conflicts + ## Monitoring and Maintenance ### Regular Tasks diff --git a/docs/lexicon-build-setup.md b/docs/lexicon-build-setup.md index 110bdc2..02810b3 100644 --- a/docs/lexicon-build-setup.md +++ b/docs/lexicon-build-setup.md @@ -98,9 +98,9 @@ This document summarizes the lexicon build integration setup that ensures lexico ↓ 5. Hook runs: pnpm lex:gen-server ↓ -6. Hook stages generated TypeScript files +6. Hook validates lexicons are properly generated ↓ -7. Commit proceeds with both source and generated files +7. Commit proceeds with only source files (generated files are ignored by .gitignore) ``` ## 🛠️ Available Commands @@ -151,8 +151,9 @@ pnpm turbo build --filter=@teal/amethyst 2. **Build Reliability**: Amethyst builds can't proceed without fresh lexicons 3. **Developer Experience**: No need to remember to run lexicon commands 4. **CI/CD Safety**: Docker builds include lexicon generation -5. **Git Safety**: Commits with lexicon changes include generated files +5. **Git Safety**: Commits with lexicon changes trigger validation and regeneration 6. **Caching**: Turbo caches lexicon generation for performance +7. **Clean Repository**: Generated files are ignored, only source lexicons are tracked ## 🔍 Verification @@ -176,7 +177,7 @@ docker build -f apps/amethyst/Dockerfile . # Make a lexicon change echo '{}' > lexicons/test.json -# Commit should validate and regenerate +# Commit should validate and regenerate (generated files won't be staged) git add . && git commit -m "test lexicon change" ``` diff --git a/docs/lexicon-development.md b/docs/lexicon-development.md index 0c14c20..2919d7c 100644 --- a/docs/lexicon-development.md +++ b/docs/lexicon-development.md @@ -17,7 +17,7 @@ teal/ ├── lexicons/ # Source lexicon JSON files │ └── fm.teal.alpha/ # Lexicon namespace ├── packages/lexicons/ # Generated TypeScript package -│ ├── src/ # Generated TypeScript files +│ ├── src/ # Generated TypeScript files (ignored by .gitignore) │ │ ├── types/ # Generated type definitions │ │ ├── index.ts # Main exports │ │ └── lexicons.ts # Lexicon registry diff --git a/scripts/pre-commit-hook.sh b/scripts/pre-commit-hook.sh index 416e62a..6dd399c 100755 --- a/scripts/pre-commit-hook.sh +++ b/scripts/pre-commit-hook.sh @@ -118,14 +118,8 @@ if [ -n "$LEXICON_FILES" ]; then exit 1 fi - # Add generated lexicon files to staging - if [ -d "packages/lexicons/src" ]; then - find packages/lexicons/src -name "*.ts" -type f | while read -r file; do - if [ -f "$file" ]; then - git add "$file" - fi - done - fi + # Note: Generated lexicon files are ignored by .gitignore and not added to staging + print_status "Generated lexicon files are ignored by .gitignore (as intended)" else print_warning "pnpm not found, skipping lexicon checks" fi diff --git a/test-git-hooks.md b/test-git-hooks.md deleted file mode 100644 index 7f4ecae..0000000 --- a/test-git-hooks.md +++ /dev/null @@ -1 +0,0 @@ -# Test git hooks with lexicons