upleb.uk

Public git repos — served from a NIP-34 GRASP relay at git.upleb.uk

summaryrefslogtreecommitdiff
path: root/docs/how-to/migrate-to-ngit-grasp.md
diff options
context:
space:
mode:
authorDanConwayDev <DanConwayDev@protonmail.com>2026-01-23 16:59:24 +0000
committerDanConwayDev <DanConwayDev@protonmail.com>2026-01-27 20:38:06 +0000
commitcbb0e768641a6ca0cbd7e7013437cc71b920004d (patch)
tree3d350eec31ea2fcd2c533f949bb136498ea45767 /docs/how-to/migrate-to-ngit-grasp.md
parent0b9527ede03521a40f1174a5a6e40a943bf27e2d (diff)
Capture invalid announcement rejections in Phase 4
Update parse failures script to also extract 'Invalid announcement' rejections from logs. These are announcement events that failed validation (e.g., multiple clone tags instead of single tag with multiple values). Changes: - Search for 'Event rejected by write policy' pattern with 'Invalid announcement' - Search for 'Rejected repository announcement' pattern from builder - Extract event_id, kind, and reason from rejection logs - Combine with [PARSE_FAIL] entries in output - Deduplicate entries by event_id - Update header to clarify both patterns are captured - Update migration guide to document this - Fix SIGPIPE handling in purgatory script (minor) This captures the ~446 unique announcements rejected for NIP-34 format violations (multiple clone tags), which were previously unexplained in the migration analysis.
Diffstat (limited to 'docs/how-to/migrate-to-ngit-grasp.md')
-rw-r--r--docs/how-to/migrate-to-ngit-grasp.md20
1 files changed, 13 insertions, 7 deletions
diff --git a/docs/how-to/migrate-to-ngit-grasp.md b/docs/how-to/migrate-to-ngit-grasp.md
index 00af6c8..62cad87 100644
--- a/docs/how-to/migrate-to-ngit-grasp.md
+++ b/docs/how-to/migrate-to-ngit-grasp.md
@@ -122,7 +122,7 @@ ls /path/to/git/npub1*/ # Should show *.git directories
122 122
123### Phase 4 Needs the Correct Service Name 123### Phase 4 Needs the Correct Service Name
124 124
125> **CRITICAL:** Phase 4 extracts structured logs (`[PARSE_FAIL]`, `[PURGATORY_EXPIRED]`) from journald. These logs **ONLY exist in ngit-grasp services**, NOT in ngit-relay services. 125> **CRITICAL:** Phase 4 extracts structured logs (`[PARSE_FAIL]`, `[PURGATORY_EXPIRED]`, `Invalid announcement` rejections) from journald. These logs **ONLY exist in ngit-grasp services**, NOT in ngit-relay services.
126 126
127If you specify an ngit-relay service (like `ngit-relay.service`), Phase 4 will find **zero logs** and produce empty results. This is a common mistake that wastes time and produces misleading analysis. 127If you specify an ngit-relay service (like `ngit-relay.service`), Phase 4 will find **zero logs** and produce empty results. This is a common mistake that wastes time and produces misleading analysis.
128 128
@@ -140,10 +140,10 @@ If you specify an ngit-relay service (like `ngit-relay.service`), Phase 4 will f
140systemctl list-units 'ngit-*' --all 140systemctl list-units 'ngit-*' --all
141 141
142# Check which service has structured logging (should be ngit-grasp) 142# Check which service has structured logging (should be ngit-grasp)
143journalctl -u ngit-grasp-*.service | grep -E '\[PARSE_FAIL\]|\[PURGATORY_EXPIRED\]' | head -5 143journalctl -u ngit-grasp-*.service | grep -E '\[PARSE_FAIL\]|\[PURGATORY_EXPIRED\]|Invalid announcement' | head -5
144 144
145# Verify ngit-relay does NOT have structured logging 145# Verify ngit-relay does NOT have structured logging
146journalctl -u ngit-relay.service | grep -E '\[PARSE_FAIL\]|\[PURGATORY_EXPIRED\]' | head -5 146journalctl -u ngit-relay.service | grep -E '\[PARSE_FAIL\]|\[PURGATORY_EXPIRED\]|Invalid announcement' | head -5
147# ^ This should return nothing 147# ^ This should return nothing
148 148
149# Use the archive service name for Phase 4 149# Use the archive service name for Phase 4
@@ -448,7 +448,7 @@ The analysis will continue without log data.
448 448
449**Most common cause:** You're querying the wrong service (ngit-relay instead of ngit-grasp). 449**Most common cause:** You're querying the wrong service (ngit-relay instead of ngit-grasp).
450 450
451Structured logging (`[PARSE_FAIL]`, `[PURGATORY_EXPIRED]`) **only exists in ngit-grasp services**. If you specify an ngit-relay service, Phase 4 will find zero logs. 451Structured logging (`[PARSE_FAIL]`, `[PURGATORY_EXPIRED]`, `Invalid announcement` rejections) **only exists in ngit-grasp services**. If you specify an ngit-relay service, Phase 4 will find zero logs.
452 452
453**How to diagnose:** 453**How to diagnose:**
454 454
@@ -464,7 +464,7 @@ systemctl list-units 'ngit-grasp*' --all
464 464
465# 4. Verify the ngit-grasp service has structured logs 465# 4. Verify the ngit-grasp service has structured logs
466journalctl -u ngit-grasp-relay-ngit-dev.service --since "7 days ago" | \ 466journalctl -u ngit-grasp-relay-ngit-dev.service --since "7 days ago" | \
467 grep -E '\[PARSE_FAIL\]|\[PURGATORY_EXPIRED\]' | head -5 467 grep -E '\[PARSE_FAIL\]|\[PURGATORY_EXPIRED\]|Invalid announcement' | head -5
468``` 468```
469 469
470**How to fix:** 470**How to fix:**
@@ -482,12 +482,14 @@ journalctl -u ngit-grasp-relay-ngit-dev.service --since "7 days ago" | \
482 482
4831. **Structured logging not deployed:** If the ngit-grasp instance doesn't have the logging improvements deployed, no structured logs will exist. Check the ngit-grasp version. 4831. **Structured logging not deployed:** If the ngit-grasp instance doesn't have the logging improvements deployed, no structured logs will exist. Check the ngit-grasp version.
484 484
4852. **No events in time window:** If there genuinely were no parse failures or purgatory expiry events, the files will be empty. This is valid - it means everything parsed successfully. 4852. **No events in time window:** If there genuinely were no parse failures, purgatory expiry events, or invalid announcement rejections, the files will be empty. This is valid - it means everything parsed successfully.
486 486
4873. **Wrong time range:** The default is 30 days. If your archive has been running longer, you may need `--since` to extend the range. 4873. **Wrong time range:** The default is 30 days. If your archive has been running longer, you may need `--since` to extend the range.
488 488
489**Prevention:** The migration scripts now validate the service name and will error if you specify an ngit-relay service. 489**Prevention:** The migration scripts now validate the service name and will error if you specify an ngit-relay service.
490 490
491**Note on "Invalid announcement" rejections:** These are announcements (kind 30617) that were rejected by the write policy due to format violations. The most common reason is "multiple clone tags found" - the NIP-34 spec requires a single clone tag with multiple values, not multiple clone tags. These rejections are logged as `Event rejected by write policy ... reason=Invalid announcement: ...`.
492
491### Event counts are multiples of 250 493### Event counts are multiples of 250
492 494
493This suggests pagination may have failed. The scripts use `--paginate` by default, but if you see exactly 250, 500, 750 events, verify the relay is responding correctly. 495This suggests pagination may have failed. The scripts use `--paginate` by default, but if you see exactly 250, 500, 750 events, verify the relay is responding correctly.
@@ -529,7 +531,11 @@ The analysis is split into 5 modular phases:
529 531
530┌─────────────────────────────────────────────────────────────────┐ 532┌─────────────────────────────────────────────────────────────────┐
531│ PHASE 4: Log-Based Categories (VPS required) │ 533│ PHASE 4: Log-Based Categories (VPS required) │
532│ Extracts [PARSE_FAIL] and [PURGATORY_EXPIRED] from logs │ 534│ Extracts structured logs from the archive service: │
535│ - [PARSE_FAIL] - Events that failed to parse │
536│ - [PURGATORY_EXPIRED] - Repos where git data never arrived │
537│ - "Invalid announcement" - Announcements rejected for format │
538│ violations (e.g., multiple clone tags) │
533│ Provides context for why repos failed to sync │ 539│ Provides context for why repos failed to sync │
534└─────────────────────────────────────────────────────────────────┘ 540└─────────────────────────────────────────────────────────────────┘
535 541