diff options
| author | DanConwayDev <DanConwayDev@protonmail.com> | 2025-12-11 16:20:23 +0000 |
|---|---|---|
| committer | DanConwayDev <DanConwayDev@protonmail.com> | 2025-12-11 16:20:23 +0000 |
| commit | 6d0447f31eb9f9282e60ac3c90c665a8b3781331 (patch) | |
| tree | 52a15001bda47c1096f82eb0598c8320df0b637c /docs | |
| parent | 497df415749039236126140193af0ea612358cc7 (diff) | |
feat: implement NIP-77 negentropy sync for historical data
Replace EOSE-based sync completion with negentropy reconciliation for:
- Initial connect (fresh sync)
- Daily sync (Layer 1 announcements)
- Stale reconnect (>15 min)
Key changes:
- Add NegentropySyncResult struct with remote_only, local_only, received fields
- Add supports_negentropy() using try-and-fallback approach
- Add negentropy_sync_filter() using nostr-sdk client.sync() API
- Modify handle_connect_or_reconnect() to use negentropy for fresh/stale sync
- Modify daily_sync() to use negentropy for Layer 1
- Single-warning logging per relay when negentropy fails
Quick reconnects (<15 min) unchanged - still use REQ with since filter.
If negentropy unsupported, gracefully falls back to REQ+EOSE flow.
Diffstat (limited to 'docs')
| -rw-r--r-- | docs/explanation/grasp-02-proactive-sync.md | 74 |
1 files changed, 74 insertions, 0 deletions
diff --git a/docs/explanation/grasp-02-proactive-sync.md b/docs/explanation/grasp-02-proactive-sync.md index d6665cc..f11148e 100644 --- a/docs/explanation/grasp-02-proactive-sync.md +++ b/docs/explanation/grasp-02-proactive-sync.md | |||
| @@ -9,6 +9,7 @@ This document explains the proactive sync system that synchronizes repository da | |||
| 9 | 3. **Two subscription paths on reconnect** - Catch-up (retained, with since) vs new items (via compute_actions) | 9 | 3. **Two subscription paths on reconnect** - Catch-up (retained, with since) vs new items (via compute_actions) |
| 10 | 4. **Blank state = fresh sync** - Empty confirmed state triggers full historical fetch | 10 | 4. **Blank state = fresh sync** - Empty confirmed state triggers full historical fetch |
| 11 | 5. **Clear on disconnect, not reconnect** - PendingSyncIndex cleared at event boundary | 11 | 5. **Clear on disconnect, not reconnect** - PendingSyncIndex cleared at event boundary |
| 12 | 6. **NIP-77 negentropy for historical sync** - Efficient set reconciliation, fallback to REQ if unsupported | ||
| 12 | 13 | ||
| 13 | --- | 14 | --- |
| 14 | 15 | ||
| @@ -487,6 +488,79 @@ flowchart TB | |||
| 487 | | Clear on disconnect | Clear PSI on disconnect | Cleanup at event boundary, simpler than on reconnect | | 488 | | Clear on disconnect | Clear PSI on disconnect | Cleanup at event boundary, simpler than on reconnect | |
| 488 | | 15-minute rule | Clear confirmed if disconnected >15min | Matches since filter buffer, prevents stale subscriptions | | 489 | | 15-minute rule | Clear confirmed if disconnected >15min | Matches since filter buffer, prevents stale subscriptions | |
| 489 | | Daily timer | Fresh sync (clears state) | Ensures consistency, detects drift | | 490 | | Daily timer | Fresh sync (clears state) | Ensures consistency, detects drift | |
| 491 | | NIP-77 negentropy | Try first, fallback to REQ | Efficient set reconciliation when supported | | ||
| 492 | |||
| 493 | --- | ||
| 494 | |||
| 495 | ## NIP-77 Negentropy Sync | ||
| 496 | |||
| 497 | The sync system supports NIP-77 negentropy for efficient set reconciliation when syncing with external relays. | ||
| 498 | |||
| 499 | ### What is Negentropy? | ||
| 500 | |||
| 501 | NIP-77 defines the negentropy protocol for efficient event set comparison. Instead of requesting all events matching a filter (REQ+EOSE), negentropy allows relays to compare fingerprints of their event sets and only transfer the differences. | ||
| 502 | |||
| 503 | ### When Negentropy is Used | ||
| 504 | |||
| 505 | Negentropy sync is attempted for: | ||
| 506 | - **Initial connect** - Fresh sync without `last_connected` | ||
| 507 | - **Daily sync** - Periodic full refresh (23-25 hour timer) | ||
| 508 | - **Stale reconnect** - Disconnected for more than 15 minutes | ||
| 509 | |||
| 510 | Negentropy is NOT used for: | ||
| 511 | - **Quick reconnect** - Less than 15 minutes disconnected (uses REQ with `since`) | ||
| 512 | - **Live subscriptions** - Ongoing event streams always use REQ | ||
| 513 | |||
| 514 | ### Implementation | ||
| 515 | |||
| 516 | The [`RelayConnection`](../../src/sync/relay_connection.rs:71) now includes NIP-77 methods: | ||
| 517 | |||
| 518 | ```rust | ||
| 519 | /// Check if negentropy sync should be attempted | ||
| 520 | pub async fn supports_negentropy(&self) -> bool { | ||
| 521 | // Always returns true - we try negentropy and handle failure gracefully | ||
| 522 | true | ||
| 523 | } | ||
| 524 | |||
| 525 | /// Perform negentropy synchronization for a filter | ||
| 526 | pub async fn negentropy_sync_filter(&self, filter: Filter) | ||
| 527 | -> Result<NegentropySyncResult, String> { | ||
| 528 | // Uses nostr-sdk's client.sync() method | ||
| 529 | } | ||
| 530 | ``` | ||
| 531 | |||
| 532 | ### Sync Flow with Negentropy | ||
| 533 | |||
| 534 | ```mermaid | ||
| 535 | flowchart TB | ||
| 536 | CONNECT[Connect to relay] --> NEG{Try negentropy} | ||
| 537 | NEG --> |success| L1[Layer 1 synced via negentropy] | ||
| 538 | NEG --> |failure| FALLBACK[Fall back to REQ+EOSE] | ||
| 539 | |||
| 540 | L1 --> SINCE[Record timestamp = now] | ||
| 541 | FALLBACK --> EOSE[Wait for EOSE] | ||
| 542 | EOSE --> SINCE | ||
| 543 | |||
| 544 | SINCE --> LIVE[Open live REQ with since=now] | ||
| 545 | ``` | ||
| 546 | |||
| 547 | ### Fallback Behavior | ||
| 548 | |||
| 549 | If negentropy fails (relay doesn't support NIP-77, network error, etc.): | ||
| 550 | 1. A warning is logged (once per relay to avoid spam) | ||
| 551 | 2. The sync falls back to traditional REQ+EOSE | ||
| 552 | 3. No error is raised - fallback is automatic | ||
| 553 | |||
| 554 | **Implementation:** [`negentropy_sync_and_process()`](../../src/sync/mod.rs:1549) | ||
| 555 | |||
| 556 | ### Key Design Decisions for Negentropy | ||
| 557 | |||
| 558 | | Decision | Choice | Rationale | | ||
| 559 | |----------|--------|-----------| | ||
| 560 | | Detection approach | Try and fallback | More reliable than NIP-11 document detection | | ||
| 561 | | When to use | Fresh/daily/stale sync only | Quick reconnect with `since` is already efficient | | ||
| 562 | | Error handling | Log once, fallback silently | Avoid log spam while maintaining visibility | | ||
| 563 | | Layer application | Layer 1 first | Announcements are highest priority | | ||
| 490 | 564 | ||
| 491 | --- | 565 | --- |
| 492 | 566 | ||