| Age | Commit message (Collapse) | Author |
|---|
|
|
|
This merge includes critical bug fixes and comprehensive migration tooling
developed during the relay.ngit.dev migration effort.
Bug Fixes:
- Fix git protocol error handling to return HTTP 200 with ERR pkt-line
- Fix naughty list false positives and DNS failure identification
- Fix database query filters in load_existing_events (remove .since())
- Fix OID fetch tracking to distinguish 0 OIDs from successful fetches
- Fix purgatory event source tracking for filtered expiry logging
- Implement OID retry logic for 'not our ref' errors
Migration Tools & Documentation:
- Complete 5-phase migration analysis pipeline with orchestration script
- Phase 1: Event fetching from source relay
- Phase 2: Git sync verification
- Phase 3: Categorization and relay comparison
- Phase 4: Log extraction (parse failures, purgatory expiry)
- Phase 5: Action classification for migration decisions
- Comprehensive migration guide with lessons learned
- Troubleshooting guide for permission and corruption issues
Configuration:
- Add NGIT_LOG_LEVEL configuration option
- Update git throttle limits to 60/minute
- Improve logging throughout for better observability
|
|
Move migration guide and scripts to docs/archive/2026-01-relay-ngit-dev-migration/
with clear warnings that these are reference-only materials from a specific
migration context, not general-purpose tools.
These materials document the relay.ngit.dev migration from ngit-relay to
ngit-grasp in January 2026. The scripts were developed iteratively during
the migration and are specific to that context. They are preserved for:
- Historical reference
- Context for production fixes in this branch
- Inspiration for future migrations (not direct reuse)
The migration uncovered critical bugs now fixed in this branch:
- Git protocol error handling
- Naughty list false positives
- Purgatory event tracking
- Sync startup issues
- Configuration management
|
|
|
|
Add git ancestry comparison (22-compare-git-data.sh) to determine
commit relationships between prod and archive repos. Repos where
archive is ahead are now correctly classified as ready-for-migration
since ngit-grasp only accepts git data authorized by state events.
Previously, repos with different git data were flagged as needs-resync
even when archive had newer/better data than prod.
|
|
|
|
|
|
|
|
Remove --analysis-root flag and external data file dependencies. The script
now extracts repo/npub information directly from 'Added rejected announcement'
log entries (which include pubkey and identifier fields) and uses
`nak encode npub <hex-pubkey>` to convert hex pubkeys to npub format.
This simplification was enabled by the recent logging improvement that added
pubkey to the 'Added rejected announcement' log entries.
|
|
|
|
The new script implements the redesigned classification system with:
- Tier 1: No Action Required (complete in both, deleted, empty, archive-only)
- Tier 2: Action Required (complete in prod but missing/incomplete in archive)
- Tier 3: Manual Investigation (partial/no-match in prod, archive-only anomalies)
Produces cleaner output format with actionable categories and reasons.
|
|
Phase 4 (30-extract-parse-failures.sh) now enriches parse failures with
repo name and npub by looking up event_id in announcements.json. This is
critical because 'Invalid announcement' rejections only log event_id and
kind, not the repo name or npub.
Phase 5 (40-classify-actions.sh) was also fixed to extract columns 4 and 5
(repo|npub) instead of columns 1 and 2 (event_id|kind) from parse-failures.txt.
Without this fix, action-required.txt showed unusable output like:
000014b2... | 30617 | parse failure logged | fix event format...
Now it correctly shows:
scripts | npub1hs5244... | parse failure logged | fix event format...
The enrichment uses jq to build a lookup table from announcements.json and
optionally uses 'nak' to convert hex pubkeys to npub format.
|
|
Filter parse failures to only those for announcements that are in
production but missing from the archive. This eliminates noise from
rejections of events from other relays that don't affect migration.
Before: 223 parse failures (all rejections from all relays)
After: 18 parse failures (only for missing announcements)
The filter works by:
1. Reading missing announcements from comparison data
2. Extracting event IDs from production announcements JSON
3. Filtering parse failures to only matching event IDs
|
|
The script was counting the same invalid announcement twice because:
- Write policy logs use hex event IDs
- Builder logs use note1 (bech32) event IDs
- Deduplication only worked within each format
Fix: Only extract from write policy logs (hex IDs) to avoid the
format mismatch. Builder logs contain the same events, so we don't
lose any data.
Result: 446 entries → 223 unique invalid announcements (correct count)
|
|
Update parse failures script to also extract 'Invalid announcement'
rejections from logs. These are announcement events that failed
validation (e.g., multiple clone tags instead of single tag with
multiple values).
Changes:
- Search for 'Event rejected by write policy' pattern with 'Invalid announcement'
- Search for 'Rejected repository announcement' pattern from builder
- Extract event_id, kind, and reason from rejection logs
- Combine with [PARSE_FAIL] entries in output
- Deduplicate entries by event_id
- Update header to clarify both patterns are captured
- Update migration guide to document this
- Fix SIGPIPE handling in purgatory script (minor)
This captures the ~446 unique announcements rejected for NIP-34 format
violations (multiple clone tags), which were previously unexplained
in the migration analysis.
|
|
Make scripts fully automatic with no manual intervention needed.
Changes:
- Add --no-pager to journalctl commands in validate-service.sh
- Add service existence validation with helpful error messages
- Capture and report journalctl stderr for better error visibility
- Improve error handling without failing on empty logs
The main issue was missing --no-pager in validate-service.sh which
could cause scripts to hang when run non-interactively (e.g., via SSH).
Tested locally - scripts run without hanging and produce correct output.
|
|
Add validation to ensure Phase 4 scripts use ngit-grasp service
(with structured logging) instead of ngit-relay service.
Changes:
- Add validate-service.sh helper for reusable service validation
- Add validation to run-migration-analysis.sh before Phase 4
- Add validation to 30-extract-parse-failures.sh
- Add validation to 31-extract-purgatory-expiry.sh
- Update migration guide with clear warnings about service selection
- Expand troubleshooting for 'Phase 4 finds no logs' issue
- Emphasize lesson learned in relay.ngit.dev notes
This prevents the issue where Phase 4 was run against ngit-relay.service
and found no parse failures because structured logging only exists in
ngit-grasp services.
|
|
- Add Gotchas section with common issues: git installation, localhost-only
archive relays, non-standard git paths, service name variations, and
permission requirements
- Add relay.ngit.dev-specific migration notes with actual paths, service
names, and analysis results (315 repos need re-sync, 382 purgatory expired)
- Enhance Running the Analysis section with path discovery guidance
- Expand Troubleshooting section with solutions for git not found, archive
connection failures, and wrong git paths
- Add git --version to prerequisite checks
- Update examples to use realistic localhost archive URLs
|
|
- 10-check-git-sync.sh: Check for git before running
- run-migration-analysis.sh: Include git in prerequisite checks
- Fixes script failures when git is not installed
|
|
- Rename guide: migrate-ngit-relay-to-ngit-grasp.md → migrate-to-ngit-grasp.md
- Remove ngit-relay and relay.ngit.dev specific references
- Use generic terminology: source/target relay, current implementation
- Add Compatibility section explaining requirements
- Update examples to be implementation-agnostic
- Update script comments to reference GRASP relay (not ngit-relay)
- Update README.md to link to the new guide
Scripts already work with any GRASP implementation via parameters.
|
|
Transforms the guide from a technical reference into a practical
step-by-step guide with:
- Quick Start section at the top with copy-paste commands
- Prerequisites section with verification steps
- Migration Overview explaining the 3-stage process
- Running the Analysis section with all options documented
- Understanding Results section explaining output files
- Troubleshooting section for common issues
- Architecture section (moved from top) for those wanting details
- Next Steps section for post-analysis workflow
The guide now follows a practical flow: get started fast, understand
results, then dive into architecture details if needed.
|
|
Adds run-migration-analysis.sh that orchestrates all 5 phases of the
migration analysis with:
- Parameterized inputs for relay URLs, git paths, and service name
- Phase control (skip, only, from-phase options)
- Dry-run mode to preview execution
- Progress indicators and timing information
- Error handling with continue-on-error option
- Auto-detection of available features (git paths, journalctl)
- Summary display with results overview
|
|
- Combines all data sources from Phases 1-4
- Produces three actionable outputs: no-action, action-required, manual-investigation
- Generates comprehensive summary with recommendations
- Handles missing Phase 4 logs gracefully
- Classification logic for migration decision-making
|
|
- 30-extract-parse-failures.sh: Extracts parse failure events from logs
- 31-extract-purgatory-expiry.sh: Extracts purgatory expiry events from logs
- Both support time range filtering (--since, --until)
- Includes dry-run mode for testing
- Gracefully handles missing logs with dependency notes
- TSV output format for Phase 5 consumption
- Ready for when structured logging is implemented in ngit-grasp
|
|
- Compares state event refs to actual git data on disk
- Uses git show-ref to handle both loose and packed refs
- Outputs TSV format compatible with Phase 3 categorization
- Optional --categorize flag for inline categorization
- Includes progress indicators and ETA (~20 min runtime on VPS)
- Improved error handling and validation over original script
|
|
- 20-categorize.sh: Categorizes git sync status into 4 categories
- 21-compare-relays.sh: Compares prod vs archive to find gaps
- Updated how-to doc with detailed Phase 3 outputs and directory structure
- Tested with Jan 22 data: 231 complete in both, 276 complete in prod but missing from archive
|
|
- Fetches kind 30618 (state), 30617 (announcement), 5 (deletion) events
- Uses nak req --paginate for complete event retrieval
- Outputs JSONL format for downstream processing
- Includes error handling and timing information
|
|
|
|
|
|
- Update default bind address in src/config.rs to 127.0.0.1:7334
- Update all four critical config sources per AGENTS.md:
- src/config.rs (code default and tests)
- .env.example (development template)
- docs/reference/configuration.md (user documentation)
- nix/module.nix (NixOS deployment)
- Update all documentation examples and references:
- README.md (with note about phone keypad mnemonic)
- docs/how-to/*.md (deploy, prometheus-setup, test-compliance)
- docs/explanation/*.md (architecture, comparison)
- docs/learnings/grasp-audit.md
Port 7334 spells NGIT on a phone keypad, making it memorable and
project-specific.
All tests pass (336 lib tests + 51 integration tests).
|
|
- Add new how-to guide covering hash updates for git dependencies
- Applies to any git dependency (e.g., nostr-sdk fork)
- Add critical note in AGENTS.md linking to this guide
- Emphasize that hash updates in both flake.nix and nix/module.nix are MANDATORY
|
|
- Complete guide for deploying ngit-grasp to NixOS servers
- Step-by-step deployment instructions
- Configuration options reference
- Troubleshooting section
- Security hardening recommendations
- Multiple instance examples
- References nix/example-configuration.nix which has clear examples
|
|
The sync system uses a 5-second batch window for discovered repos.
Repos discovered late in a 30-second test don't have enough time for
the full Layer 2→3→4 cascade:
- Layer 1: Discover repo announcements (0-5s)
- Layer 2: Send #a, #A, #q filters for repos (5-30s)
- Layer 3: Receive issues, patches, PRs (30-60s)
- Layer 4: Receive comments on root events (40-60s)
Testing confirmed that 60 seconds allows late-discovered repos
(gitworkshop, ngit) to complete all layers, while 30 seconds only
allows 1 second after Layer 2 filters are sent.
Updated all references from 30s to 60s throughout the guide and added
explanation of why this duration is necessary.
|
|
- Update production sync testing workflow to save both sync-raw.log and sync.log
- Raw log contains complete untruncated messages (rejection reasons, event data, etc.)
- Sanitized log remains for quick scanning and pattern recognition
- Add guidance on when to use each log and how to retrieve full details from raw log
- Resolves truncated rejection warning issue by making full details accessible
|
|
Mode 1 (Fix Existing Issues) now requires reviewing the proposed fix
and asking for user permission before implementing changes. This ensures
users have visibility and control over what code changes are made.
Changes:
- Added Step 3: Review Proposed Fix and Get Permission
- Renumbered subsequent steps (4-7)
- Updated both overview and detailed workflow sections
- Updated workflow diagram to show review/permission steps
|
|
- Mode 1: Fix one existing issue, test, commit, report
- Mode 2: Discover new issues with minimal documentation
- Emphasize stopping after each cycle
- Remove detailed investigation requirements
- Simplify issue documentation format
|
|
- Update production-sync-testing.md to document issues as individual markdown files in work/active-issues/ instead of polluting the tracked how-to guide
- Add issue template and workflow for creating, viewing, and resolving issues
- Document active-issues/ purpose in work/README.md
- Prevents accidental commits of transient testing issues
- Makes issue management cleaner and more focused
|
|
- Fix shebang in sanitize-logs.sh from #!/bin/bash to #!/usr/bin/env bash for NixOS compatibility
- Update sanitizer defaults from 100/20 to 200/100 chars for better log readability
- Fix CLI argument names in guide: --sync-bootstrap-relay -> --sync-bootstrap-relay-url
- Fix CLI argument names in guide: --git-path -> --git-data-path
These issues were discovered during first-time testing of the production sync testing guide.
|
|
Add infrastructure for iterative debugging of sync against production data:
- scripts/sanitize-logs.sh: Truncates verbose log lines for LLM analysis
- docs/how-to/production-sync-testing.md: Step-by-step guide for testing
sync against real relays, identifying issues, and improving logging
|
|
|
|
- Remove unnecessary 'nix' dev dependency (Unix syscalls crate, not needed)
- Migrate announcement tests to new TestRelay fixture pattern
- Delete legacy test files (announcement_tests.rs, test_relay.sh)
- Add comprehensive test documentation (docs/how-to/test-compliance.md)
- Update README.md with new test commands
- All 18 integration tests passing (NIP-01 + NIP-34)
Benefits:
- Automatic relay lifecycle management
- No manual setup required
- Pure Rust integration tests
- Better developer experience
- CI/CD ready
|
|
|