diff options
| author | DanConwayDev <DanConwayDev@protonmail.com> | 2026-01-23 16:59:24 +0000 |
|---|---|---|
| committer | DanConwayDev <DanConwayDev@protonmail.com> | 2026-01-27 20:38:06 +0000 |
| commit | cbb0e768641a6ca0cbd7e7013437cc71b920004d (patch) | |
| tree | 3d350eec31ea2fcd2c533f949bb136498ea45767 /docs/how-to/migrate-to-ngit-grasp.md | |
| parent | 0b9527ede03521a40f1174a5a6e40a943bf27e2d (diff) | |
Capture invalid announcement rejections in Phase 4
Update parse failures script to also extract 'Invalid announcement'
rejections from logs. These are announcement events that failed
validation (e.g., multiple clone tags instead of single tag with
multiple values).
Changes:
- Search for 'Event rejected by write policy' pattern with 'Invalid announcement'
- Search for 'Rejected repository announcement' pattern from builder
- Extract event_id, kind, and reason from rejection logs
- Combine with [PARSE_FAIL] entries in output
- Deduplicate entries by event_id
- Update header to clarify both patterns are captured
- Update migration guide to document this
- Fix SIGPIPE handling in purgatory script (minor)
This captures the ~446 unique announcements rejected for NIP-34 format
violations (multiple clone tags), which were previously unexplained
in the migration analysis.
Diffstat (limited to 'docs/how-to/migrate-to-ngit-grasp.md')
| -rw-r--r-- | docs/how-to/migrate-to-ngit-grasp.md | 20 |
1 files changed, 13 insertions, 7 deletions
diff --git a/docs/how-to/migrate-to-ngit-grasp.md b/docs/how-to/migrate-to-ngit-grasp.md index 00af6c8..62cad87 100644 --- a/docs/how-to/migrate-to-ngit-grasp.md +++ b/docs/how-to/migrate-to-ngit-grasp.md | |||
| @@ -122,7 +122,7 @@ ls /path/to/git/npub1*/ # Should show *.git directories | |||
| 122 | 122 | ||
| 123 | ### Phase 4 Needs the Correct Service Name | 123 | ### Phase 4 Needs the Correct Service Name |
| 124 | 124 | ||
| 125 | > **CRITICAL:** Phase 4 extracts structured logs (`[PARSE_FAIL]`, `[PURGATORY_EXPIRED]`) from journald. These logs **ONLY exist in ngit-grasp services**, NOT in ngit-relay services. | 125 | > **CRITICAL:** Phase 4 extracts structured logs (`[PARSE_FAIL]`, `[PURGATORY_EXPIRED]`, `Invalid announcement` rejections) from journald. These logs **ONLY exist in ngit-grasp services**, NOT in ngit-relay services. |
| 126 | 126 | ||
| 127 | If you specify an ngit-relay service (like `ngit-relay.service`), Phase 4 will find **zero logs** and produce empty results. This is a common mistake that wastes time and produces misleading analysis. | 127 | If you specify an ngit-relay service (like `ngit-relay.service`), Phase 4 will find **zero logs** and produce empty results. This is a common mistake that wastes time and produces misleading analysis. |
| 128 | 128 | ||
| @@ -140,10 +140,10 @@ If you specify an ngit-relay service (like `ngit-relay.service`), Phase 4 will f | |||
| 140 | systemctl list-units 'ngit-*' --all | 140 | systemctl list-units 'ngit-*' --all |
| 141 | 141 | ||
| 142 | # Check which service has structured logging (should be ngit-grasp) | 142 | # Check which service has structured logging (should be ngit-grasp) |
| 143 | journalctl -u ngit-grasp-*.service | grep -E '\[PARSE_FAIL\]|\[PURGATORY_EXPIRED\]' | head -5 | 143 | journalctl -u ngit-grasp-*.service | grep -E '\[PARSE_FAIL\]|\[PURGATORY_EXPIRED\]|Invalid announcement' | head -5 |
| 144 | 144 | ||
| 145 | # Verify ngit-relay does NOT have structured logging | 145 | # Verify ngit-relay does NOT have structured logging |
| 146 | journalctl -u ngit-relay.service | grep -E '\[PARSE_FAIL\]|\[PURGATORY_EXPIRED\]' | head -5 | 146 | journalctl -u ngit-relay.service | grep -E '\[PARSE_FAIL\]|\[PURGATORY_EXPIRED\]|Invalid announcement' | head -5 |
| 147 | # ^ This should return nothing | 147 | # ^ This should return nothing |
| 148 | 148 | ||
| 149 | # Use the archive service name for Phase 4 | 149 | # Use the archive service name for Phase 4 |
| @@ -448,7 +448,7 @@ The analysis will continue without log data. | |||
| 448 | 448 | ||
| 449 | **Most common cause:** You're querying the wrong service (ngit-relay instead of ngit-grasp). | 449 | **Most common cause:** You're querying the wrong service (ngit-relay instead of ngit-grasp). |
| 450 | 450 | ||
| 451 | Structured logging (`[PARSE_FAIL]`, `[PURGATORY_EXPIRED]`) **only exists in ngit-grasp services**. If you specify an ngit-relay service, Phase 4 will find zero logs. | 451 | Structured logging (`[PARSE_FAIL]`, `[PURGATORY_EXPIRED]`, `Invalid announcement` rejections) **only exists in ngit-grasp services**. If you specify an ngit-relay service, Phase 4 will find zero logs. |
| 452 | 452 | ||
| 453 | **How to diagnose:** | 453 | **How to diagnose:** |
| 454 | 454 | ||
| @@ -464,7 +464,7 @@ systemctl list-units 'ngit-grasp*' --all | |||
| 464 | 464 | ||
| 465 | # 4. Verify the ngit-grasp service has structured logs | 465 | # 4. Verify the ngit-grasp service has structured logs |
| 466 | journalctl -u ngit-grasp-relay-ngit-dev.service --since "7 days ago" | \ | 466 | journalctl -u ngit-grasp-relay-ngit-dev.service --since "7 days ago" | \ |
| 467 | grep -E '\[PARSE_FAIL\]|\[PURGATORY_EXPIRED\]' | head -5 | 467 | grep -E '\[PARSE_FAIL\]|\[PURGATORY_EXPIRED\]|Invalid announcement' | head -5 |
| 468 | ``` | 468 | ``` |
| 469 | 469 | ||
| 470 | **How to fix:** | 470 | **How to fix:** |
| @@ -482,12 +482,14 @@ journalctl -u ngit-grasp-relay-ngit-dev.service --since "7 days ago" | \ | |||
| 482 | 482 | ||
| 483 | 1. **Structured logging not deployed:** If the ngit-grasp instance doesn't have the logging improvements deployed, no structured logs will exist. Check the ngit-grasp version. | 483 | 1. **Structured logging not deployed:** If the ngit-grasp instance doesn't have the logging improvements deployed, no structured logs will exist. Check the ngit-grasp version. |
| 484 | 484 | ||
| 485 | 2. **No events in time window:** If there genuinely were no parse failures or purgatory expiry events, the files will be empty. This is valid - it means everything parsed successfully. | 485 | 2. **No events in time window:** If there genuinely were no parse failures, purgatory expiry events, or invalid announcement rejections, the files will be empty. This is valid - it means everything parsed successfully. |
| 486 | 486 | ||
| 487 | 3. **Wrong time range:** The default is 30 days. If your archive has been running longer, you may need `--since` to extend the range. | 487 | 3. **Wrong time range:** The default is 30 days. If your archive has been running longer, you may need `--since` to extend the range. |
| 488 | 488 | ||
| 489 | **Prevention:** The migration scripts now validate the service name and will error if you specify an ngit-relay service. | 489 | **Prevention:** The migration scripts now validate the service name and will error if you specify an ngit-relay service. |
| 490 | 490 | ||
| 491 | **Note on "Invalid announcement" rejections:** These are announcements (kind 30617) that were rejected by the write policy due to format violations. The most common reason is "multiple clone tags found" - the NIP-34 spec requires a single clone tag with multiple values, not multiple clone tags. These rejections are logged as `Event rejected by write policy ... reason=Invalid announcement: ...`. | ||
| 492 | |||
| 491 | ### Event counts are multiples of 250 | 493 | ### Event counts are multiples of 250 |
| 492 | 494 | ||
| 493 | This suggests pagination may have failed. The scripts use `--paginate` by default, but if you see exactly 250, 500, 750 events, verify the relay is responding correctly. | 495 | This suggests pagination may have failed. The scripts use `--paginate` by default, but if you see exactly 250, 500, 750 events, verify the relay is responding correctly. |
| @@ -529,7 +531,11 @@ The analysis is split into 5 modular phases: | |||
| 529 | ↓ | 531 | ↓ |
| 530 | ┌─────────────────────────────────────────────────────────────────┐ | 532 | ┌─────────────────────────────────────────────────────────────────┐ |
| 531 | │ PHASE 4: Log-Based Categories (VPS required) │ | 533 | │ PHASE 4: Log-Based Categories (VPS required) │ |
| 532 | │ Extracts [PARSE_FAIL] and [PURGATORY_EXPIRED] from logs │ | 534 | │ Extracts structured logs from the archive service: │ |
| 535 | │ - [PARSE_FAIL] - Events that failed to parse │ | ||
| 536 | │ - [PURGATORY_EXPIRED] - Repos where git data never arrived │ | ||
| 537 | │ - "Invalid announcement" - Announcements rejected for format │ | ||
| 538 | │ violations (e.g., multiple clone tags) │ | ||
| 533 | │ Provides context for why repos failed to sync │ | 539 | │ Provides context for why repos failed to sync │ |
| 534 | └─────────────────────────────────────────────────────────────────┘ | 540 | └─────────────────────────────────────────────────────────────────┘ |
| 535 | ↓ | 541 | ↓ |