| Age | Commit message (Collapse) | Author |
|---|
|
|
|
The new script implements the redesigned classification system with:
- Tier 1: No Action Required (complete in both, deleted, empty, archive-only)
- Tier 2: Action Required (complete in prod but missing/incomplete in archive)
- Tier 3: Manual Investigation (partial/no-match in prod, archive-only anomalies)
Produces cleaner output format with actionable categories and reasons.
|
|
Phase 4 (30-extract-parse-failures.sh) now enriches parse failures with
repo name and npub by looking up event_id in announcements.json. This is
critical because 'Invalid announcement' rejections only log event_id and
kind, not the repo name or npub.
Phase 5 (40-classify-actions.sh) was also fixed to extract columns 4 and 5
(repo|npub) instead of columns 1 and 2 (event_id|kind) from parse-failures.txt.
Without this fix, action-required.txt showed unusable output like:
000014b2... | 30617 | parse failure logged | fix event format...
Now it correctly shows:
scripts | npub1hs5244... | parse failure logged | fix event format...
The enrichment uses jq to build a lookup table from announcements.json and
optionally uses 'nak' to convert hex pubkeys to npub format.
|
|
Filter parse failures to only those for announcements that are in
production but missing from the archive. This eliminates noise from
rejections of events from other relays that don't affect migration.
Before: 223 parse failures (all rejections from all relays)
After: 18 parse failures (only for missing announcements)
The filter works by:
1. Reading missing announcements from comparison data
2. Extracting event IDs from production announcements JSON
3. Filtering parse failures to only matching event IDs
|
|
The script was counting the same invalid announcement twice because:
- Write policy logs use hex event IDs
- Builder logs use note1 (bech32) event IDs
- Deduplication only worked within each format
Fix: Only extract from write policy logs (hex IDs) to avoid the
format mismatch. Builder logs contain the same events, so we don't
lose any data.
Result: 446 entries → 223 unique invalid announcements (correct count)
|
|
Update parse failures script to also extract 'Invalid announcement'
rejections from logs. These are announcement events that failed
validation (e.g., multiple clone tags instead of single tag with
multiple values).
Changes:
- Search for 'Event rejected by write policy' pattern with 'Invalid announcement'
- Search for 'Rejected repository announcement' pattern from builder
- Extract event_id, kind, and reason from rejection logs
- Combine with [PARSE_FAIL] entries in output
- Deduplicate entries by event_id
- Update header to clarify both patterns are captured
- Update migration guide to document this
- Fix SIGPIPE handling in purgatory script (minor)
This captures the ~446 unique announcements rejected for NIP-34 format
violations (multiple clone tags), which were previously unexplained
in the migration analysis.
|
|
Make scripts fully automatic with no manual intervention needed.
Changes:
- Add --no-pager to journalctl commands in validate-service.sh
- Add service existence validation with helpful error messages
- Capture and report journalctl stderr for better error visibility
- Improve error handling without failing on empty logs
The main issue was missing --no-pager in validate-service.sh which
could cause scripts to hang when run non-interactively (e.g., via SSH).
Tested locally - scripts run without hanging and produce correct output.
|
|
Add validation to ensure Phase 4 scripts use ngit-grasp service
(with structured logging) instead of ngit-relay service.
Changes:
- Add validate-service.sh helper for reusable service validation
- Add validation to run-migration-analysis.sh before Phase 4
- Add validation to 30-extract-parse-failures.sh
- Add validation to 31-extract-purgatory-expiry.sh
- Update migration guide with clear warnings about service selection
- Expand troubleshooting for 'Phase 4 finds no logs' issue
- Emphasize lesson learned in relay.ngit.dev notes
This prevents the issue where Phase 4 was run against ngit-relay.service
and found no parse failures because structured logging only exists in
ngit-grasp services.
|
|
- 10-check-git-sync.sh: Check for git before running
- run-migration-analysis.sh: Include git in prerequisite checks
- Fixes script failures when git is not installed
|
|
- Rename guide: migrate-ngit-relay-to-ngit-grasp.md → migrate-to-ngit-grasp.md
- Remove ngit-relay and relay.ngit.dev specific references
- Use generic terminology: source/target relay, current implementation
- Add Compatibility section explaining requirements
- Update examples to be implementation-agnostic
- Update script comments to reference GRASP relay (not ngit-relay)
- Update README.md to link to the new guide
Scripts already work with any GRASP implementation via parameters.
|
|
Adds run-migration-analysis.sh that orchestrates all 5 phases of the
migration analysis with:
- Parameterized inputs for relay URLs, git paths, and service name
- Phase control (skip, only, from-phase options)
- Dry-run mode to preview execution
- Progress indicators and timing information
- Error handling with continue-on-error option
- Auto-detection of available features (git paths, journalctl)
- Summary display with results overview
|
|
- Combines all data sources from Phases 1-4
- Produces three actionable outputs: no-action, action-required, manual-investigation
- Generates comprehensive summary with recommendations
- Handles missing Phase 4 logs gracefully
- Classification logic for migration decision-making
|
|
- 30-extract-parse-failures.sh: Extracts parse failure events from logs
- 31-extract-purgatory-expiry.sh: Extracts purgatory expiry events from logs
- Both support time range filtering (--since, --until)
- Includes dry-run mode for testing
- Gracefully handles missing logs with dependency notes
- TSV output format for Phase 5 consumption
- Ready for when structured logging is implemented in ngit-grasp
|
|
- Compares state event refs to actual git data on disk
- Uses git show-ref to handle both loose and packed refs
- Outputs TSV format compatible with Phase 3 categorization
- Optional --categorize flag for inline categorization
- Includes progress indicators and ETA (~20 min runtime on VPS)
- Improved error handling and validation over original script
|
|
- 20-categorize.sh: Categorizes git sync status into 4 categories
- 21-compare-relays.sh: Compares prod vs archive to find gaps
- Updated how-to doc with detailed Phase 3 outputs and directory structure
- Tested with Jan 22 data: 231 complete in both, 276 complete in prod but missing from archive
|
|
- Fetches kind 30618 (state), 30617 (announcement), 5 (deletion) events
- Uses nak req --paginate for complete event retrieval
- Outputs JSONL format for downstream processing
- Includes error handling and timing information
|