upleb.uk

Public git repos — served from a NIP-34 GRASP relay at git.upleb.uk

summaryrefslogtreecommitdiff
path: root/docs/explanation/grasp-02-proactive-sync.md
diff options
context:
space:
mode:
authorDanConwayDev <DanConwayDev@protonmail.com>2026-02-23 14:49:30 +0000
committerDanConwayDev <DanConwayDev@protonmail.com>2026-02-23 14:49:30 +0000
commit4848c4029fc58f6f310a2babeae1ee82a7e41656 (patch)
treeccdfdaae41dd2907794a47bbeff562824dd3915b /docs/explanation/grasp-02-proactive-sync.md
parentf19b424e01fc5a682778c5e2bb194d242efd6987 (diff)
docs: update purgatory docs to reflect announcements purgatory implementation
Remove the pre-implementation planning docs (announcements-purgatory-design.md and announcements-purgatory-implementation.md) now that the feature is built. Update the three living docs to reflect what was actually implemented: - purgatory-design.md: expanded to cover all three purgatory stores (announcement, state, PR), including AnnouncementPurgatoryEntry structure, two-phase soft expiry lifecycle, expiry extension triggers, promotion flow, and updated integration points and file structure - grasp-02-proactive-sync.md: added SyncLevel enum (Full/StateOnly) to RepoSyncNeeds, documented the purgatory announcement sync timer as the registration path for purgatory announcements, updated filter building to describe build_sync_level_aware_filters() and StateOnly behaviour - grasp-02-proactive-sync-purgatory-git-data.md: expanded to cover announcement purgatory as a third entry type, added Timeline E showing soft-expiry and revival, replaced the single expiry section with separate hard-expiry (state/PR) and two-phase soft-expiry (announcements) sections with full justification for the 24-hour extended retention window
Diffstat (limited to 'docs/explanation/grasp-02-proactive-sync.md')
-rw-r--r--docs/explanation/grasp-02-proactive-sync.md57
1 files changed, 49 insertions, 8 deletions
diff --git a/docs/explanation/grasp-02-proactive-sync.md b/docs/explanation/grasp-02-proactive-sync.md
index ed8fdbf..6696e27 100644
--- a/docs/explanation/grasp-02-proactive-sync.md
+++ b/docs/explanation/grasp-02-proactive-sync.md
@@ -47,20 +47,37 @@ This state starts afresh when the binary loads.
47### RepoSyncIndex (Source of Truth) 47### RepoSyncIndex (Source of Truth)
48 48
49```rust 49```rust
50/// What we WANT to sync - derived from events received via self-subscription. 50/// What we WANT to sync - derived from events received via self-subscription
51/// Updated immediately when self-subscriber batch fires. 51/// and from purgatory announcements.
52/// Updated immediately when self-subscriber batch fires or purgatory sync timer runs.
52/// Key: repo addressable ref - 30617:pubkey:identifier 53/// Key: repo addressable ref - 30617:pubkey:identifier
53pub type RepoSyncIndex = Arc<RwLock<HashMap<String, RepoSyncNeeds>>>; 54pub type RepoSyncIndex = Arc<RwLock<HashMap<String, RepoSyncNeeds>>>;
54 55
56/// Controls which sync filters are built for a repo
57#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)]
58pub enum SyncLevel {
59 #[default]
60 Full, // Full L2 + L3 sync (promoted repos with git data)
61 StateOnly, // Only state events (kind 30618) — for purgatory announcements
62}
63
55#[derive(Debug, Clone, Default)] 64#[derive(Debug, Clone, Default)]
56pub struct RepoSyncNeeds { 65pub struct RepoSyncNeeds {
57 /// Relay URLs listed in this repo's 30617 announcement 66 /// Relay URLs listed in this repo's 30617 announcement
58 pub relays: HashSet<String>, 67 pub relays: HashSet<String>,
59 /// Root event IDs - 1617/1618/1621 - that reference this repo 68 /// Root event IDs - 1617/1618/1621 - that reference this repo
60 pub root_events: HashSet<EventId>, 69 pub root_events: HashSet<EventId>,
70 /// Controls which filters are built: Full (L2+L3) or StateOnly (kind 30618 only)
71 pub sync_level: SyncLevel,
61} 72}
62``` 73```
63 74
75**Two sources populate `RepoSyncIndex`:**
76
771. **`SelfSubscriber`** — monitors the relay's own event stream for accepted announcements (kinds 30617, 1617, 1618, 1621). Adds entries with `SyncLevel::Full`. When an announcement is promoted from purgatory to the database, the SelfSubscriber sees it and upgrades the entry to `Full`.
78
792. **Purgatory announcement sync timer** (`run_purgatory_announcement_sync`, every 5 seconds) — iterates `purgatory.announcements_for_sync()` and ensures each purgatory announcement has a `SyncLevel::StateOnly` entry in `RepoSyncIndex`. This is the only registration path for purgatory announcements because they are not saved to the database and therefore never seen by the SelfSubscriber.
80
64### RelaySyncIndex (Confirmed State + Connection) 81### RelaySyncIndex (Confirmed State + Connection)
65 82
66```rust 83```rust
@@ -336,7 +353,23 @@ The sync system uses three background tasks that run continuously:
336 353
3371. Queue events to `PendingUpdates` 3541. Queue events to `PendingUpdates`
3382. Timer fires (interval, does not reset on events) 3552. Timer fires (interval, does not reset on events)
3393. Process batch: update RepoSyncIndex → derive targets → send AddFilters to SyncManager 3563. Process batch: update RepoSyncIndex with `SyncLevel::Full` → derive targets → send AddFilters to SyncManager
357
358**Note**: The SelfSubscriber only sees announcements that have been accepted to the database (promoted from purgatory). Purgatory announcements are registered separately by the purgatory sync timer (see below).
359
360### 4. Purgatory Announcement Sync Timer (`run_purgatory_announcement_sync`)
361
362**Purpose**: Register purgatory announcements in `RepoSyncIndex` so state events are synced for them
363
364**Interval**: Every 5 seconds (200ms in test mode)
365
366**Flow**:
367
3681. Iterate `purgatory.announcements_for_sync()`
3692. For each announcement not already in `RepoSyncIndex`: insert with `SyncLevel::StateOnly`
3703. When an announcement is promoted (git data arrives), the SelfSubscriber sees the newly accepted event and upgrades the entry to `SyncLevel::Full`
371
372**Why a separate timer?** Purgatory announcements are never saved to the database, so the SelfSubscriber never sees them. The timer bridges this gap, ensuring state events are synced for repos that may still receive git data.
340 373
341--- 374---
342 375
@@ -602,9 +635,10 @@ flowchart TB
602 635
603- Self-subscriber monitors own relay for 30617, 1617, 1618, 1621 (NOT 1619 or 30618) 636- Self-subscriber monitors own relay for 30617, 1617, 1618, 1621 (NOT 1619 or 30618)
604- Batches events in `PendingUpdates` (5 second window via interval timer) 637- Batches events in `PendingUpdates` (5 second window via interval timer)
605- `process_batch()` updates RepoSyncIndex, then builds AddFilters **directly** (no compute_actions) 638- `process_batch()` updates RepoSyncIndex with `SyncLevel::Full`, then builds AddFilters **directly** (no compute_actions)
606- AddFilters sent via channel to SyncManager, which calls `handle_new_sync_filters()` 639- AddFilters sent via channel to SyncManager, which calls `handle_new_sync_filters()`
607- This path does NOT use compute_actions because it's building fresh filters from the updated index 640- This path does NOT use compute_actions because it's building fresh filters from the updated index
641- Purgatory announcements (not in DB) are registered separately by the purgatory sync timer with `SyncLevel::StateOnly`
608 642
609--- 643---
610 644
@@ -687,16 +721,23 @@ fn compute_actions(
687- **Tags**: lowercase `a`, uppercase `A`, and `q` tags for comprehensive coverage 721- **Tags**: lowercase `a`, uppercase `A`, and `q` tags for comprehensive coverage
688- **Batching**: Per 100 repo refs 722- **Batching**: Per 100 repo refs
689- **Function**: `build_repo_tag_filters(repos, since)` 723- **Function**: `build_repo_tag_filters(repos, since)`
724- **Only for `SyncLevel::Full` repos** — purgatory announcements (`StateOnly`) skip this layer
690 725
691### Layer 3: Events Tagging Our Root Events 726### Layer 3: Events Tagging Our Root Events
692 727
693- **Tags**: lowercase `e`, uppercase `E`, and `q` tags for comprehensive coverage 728- **Tags**: lowercase `e`, uppercase `E`, and `q` tags for comprehensive coverage
694- **Batching**: Per 100 event IDs 729- **Batching**: Per 100 event IDs
695- **Function**: `build_root_event_tag_filters(root_events, since)` 730- **Function**: `build_root_event_tag_filters(root_events, since)`
731- **Only for `SyncLevel::Full` repos** — purgatory announcements (`StateOnly`) skip this layer
732
733### Combined Layer 2+3 (SyncLevel-Aware)
734
735The `build_sync_level_aware_filters()` function combines both layers, partitioning repos by `SyncLevel`:
696 736
697### Combined Layer 2+3 737- **`Full` repos**: state event filters + repo-tag filters + root-event-tag filters
738- **`StateOnly` repos**: state event filters only (kind 30618 with `#d` tags)
698 739
699The `build_layer2_and_layer3_filters()` function combines both layers. Used by: 740Used by:
700 741
701- `recompute_new_sync_filters_for_relay` for new item subscriptions 742- `recompute_new_sync_filters_for_relay` for new item subscriptions
702- `reconstruct_filters` for rebuilding from confirmed state 743- `reconstruct_filters` for rebuilding from confirmed state
@@ -871,9 +912,9 @@ flowchart TB
871 912
872``` 913```
873src/sync/ 914src/sync/
874├── mod.rs # SyncManager, main loop, data structures 915├── mod.rs # SyncManager, main loop, data structures, SyncLevel, run_purgatory_announcement_sync
875├── algorithms.rs # derive_relay_targets(), compute_actions() 916├── algorithms.rs # derive_relay_targets(), compute_actions()
876├── filters.rs # build_announcement_filter(), build_layer2_and_layer3_filters() 917├── filters.rs # build_announcement_filter(), build_sync_level_aware_filters()
877├── health.rs # RelayHealthTracker with exponential backoff 918├── health.rs # RelayHealthTracker with exponential backoff
878├── relay_connection.rs # RelayConnection, RelayEvent handling 919├── relay_connection.rs # RelayConnection, RelayEvent handling
879├── self_subscriber.rs # SelfSubscriber with batching 920├── self_subscriber.rs # SelfSubscriber with batching