diff options
Diffstat (limited to 'docs')
| -rw-r--r-- | docs/explanation/grasp-02-proactive-sync.md | 729 |
1 files changed, 432 insertions, 297 deletions
diff --git a/docs/explanation/grasp-02-proactive-sync.md b/docs/explanation/grasp-02-proactive-sync.md index 2a86126..34b7bb6 100644 --- a/docs/explanation/grasp-02-proactive-sync.md +++ b/docs/explanation/grasp-02-proactive-sync.md | |||
| @@ -4,12 +4,11 @@ | |||
| 4 | 4 | ||
| 5 | This document explains the proactive sync system that synchronizes repository data from external relays based on relay URLs listed in 30617 repository announcements. Key principles: | 5 | This document explains the proactive sync system that synchronizes repository data from external relays based on relay URLs listed in 30617 repository announcements. Key principles: |
| 6 | 6 | ||
| 7 | 1. **Self-subscription as the only mechanism** - No database initialization at startup | 7 | 1. **Triggers call compute_actions → sync_computed_filters** - Self-subscriber batches and connect/reconnect events trigger this flow |
| 8 | 2. **compute_actions as single decision point** - Determines what NEW subscriptions to create | 8 | 2. **Clear separation of live vs historic sync** - Two distinct primitives with different purposes |
| 9 | 3. **Two subscription paths on reconnect** - Catch-up (retained, with since) vs new items (via compute_actions) | 9 | 3. **Layer 1 on connect, Layer 2+3 via AddFilters** - L1 handled at connection time, L2+L3 flow through compute_actions |
| 10 | 4. **Blank state = fresh sync** - Empty confirmed state triggers full historical fetch | 10 | 4. **Always clear PendingSyncIndex first** - Before any reconnect/consolidate operation |
| 11 | 5. **Clear on disconnect, not reconnect** - PendingSyncIndex cleared at event boundary | 11 | 5. **NIP-77 negentropy for historical sync** - Efficient set reconciliation, fallback to REQ if unsupported |
| 12 | 6. **NIP-77 negentropy for historical sync** - Efficient set reconciliation, fallback to REQ if unsupported | ||
| 13 | 12 | ||
| 14 | --- | 13 | --- |
| 15 | 14 | ||
| @@ -90,7 +89,6 @@ impl RelayState { | |||
| 90 | ### PendingSyncIndex (In-Flight Batches) | 89 | ### PendingSyncIndex (In-Flight Batches) |
| 91 | 90 | ||
| 92 | ```rust | 91 | ```rust |
| 93 | |||
| 94 | /// Method used for synchronization | 92 | /// Method used for synchronization |
| 95 | #[derive(Debug, Clone, Copy, PartialEq, Eq)] | 93 | #[derive(Debug, Clone, Copy, PartialEq, Eq)] |
| 96 | pub enum SyncMethod { | 94 | pub enum SyncMethod { |
| @@ -100,7 +98,6 @@ pub enum SyncMethod { | |||
| 100 | Negentropy, | 98 | Negentropy, |
| 101 | } | 99 | } |
| 102 | 100 | ||
| 103 | |||
| 104 | /// Tracks batches of subscriptions that are in-flight, awaiting EOSE. | 101 | /// Tracks batches of subscriptions that are in-flight, awaiting EOSE. |
| 105 | /// Each batch has its own ID and can confirm independently. | 102 | /// Each batch has its own ID and can confirm independently. |
| 106 | /// Key: relay URL | 103 | /// Key: relay URL |
| @@ -118,7 +115,6 @@ pub struct PendingBatch { | |||
| 118 | pub sync_method: SyncMethod, | 115 | pub sync_method: SyncMethod, |
| 119 | } | 116 | } |
| 120 | 117 | ||
| 121 | |||
| 122 | #[derive(Debug, Clone, Default)] | 118 | #[derive(Debug, Clone, Default)] |
| 123 | pub struct PendingItems { | 119 | pub struct PendingItems { |
| 124 | pub repos: HashSet<String>, | 120 | pub repos: HashSet<String>, |
| @@ -145,152 +141,202 @@ stateDiagram-v2 | |||
| 145 | 141 | ||
| 146 | --- | 142 | --- |
| 147 | 143 | ||
| 148 | ## Flow Scenarios | 144 | ## Core Architecture: Live vs Historic Sync |
| 145 | |||
| 146 | The sync system is built on two fundamental primitives that are clearly separated: | ||
| 147 | |||
| 148 | ### Sync Primitives | ||
| 149 | |||
| 150 | | Primitive | Purpose | Filter Modifier | Tracking | | ||
| 151 | | ----------------- | ----------------------- | ---------------- | ---------------- | | ||
| 152 | | `sync_live()` | Ongoing event stream | `limit: 0` | Not tracked | | ||
| 153 | | `historic_sync()` | Catch up on past events | Optional `since` | PendingSyncIndex | | ||
| 154 | |||
| 155 | ### Why `limit: 0` for Live Sync? | ||
| 156 | |||
| 157 | | Approach | Pros | Cons | | ||
| 158 | | ------------ | --------------------------------------- | --------------------------------- | | ||
| 159 | | `since: now` | Intuitive | Time-sensitive, clock skew issues | | ||
| 160 | | `limit: 0` | Deterministic, mirrors filter structure | Less intuitive name | | ||
| 161 | |||
| 162 | `limit: 0` is better because: | ||
| 163 | |||
| 164 | 1. **No time dependency**: Doesn't depend on synchronized clocks | ||
| 165 | 2. **Mirrors historic filters**: Same tag structure, just different limit | ||
| 166 | 3. **State reconstruction**: Can rebuild from repo/event lists without timestamps | ||
| 167 | |||
| 168 | ### Layer Strategy | ||
| 169 | |||
| 170 | | Layer | Content | When Subscribed | Managed By | | ||
| 171 | | ------- | --------------------------------------- | --------------------- | -------------------- | | ||
| 172 | | Layer 1 | 30617 Announcements, 30618 Maintainers | On connect (any type) | Connection lifecycle | | ||
| 173 | | Layer 2 | Events tagging our repos (a/A/q tags) | Via AddFilters | compute_actions | | ||
| 174 | | Layer 3 | Events tagging root events (e/E/q tags) | Via AddFilters | compute_actions | | ||
| 149 | 175 | ||
| 150 | ### Scenario 1: Initial Connect | 176 | **Key insight**: Layer 1 is connection-level (handled at connect time), Layer 2+3 are item-level (flow through compute_actions → sync_computed_filters). |
| 177 | |||
| 178 | --- | ||
| 179 | |||
| 180 | ## Triggers and Flow | ||
| 181 | |||
| 182 | ### What Triggers compute_actions → sync_computed_filters? | ||
| 183 | |||
| 184 | | Trigger | When | What Happens | | ||
| 185 | | --------------------------- | -------------------------------------- | ---------------------------------------- | | ||
| 186 | | Self-subscriber batch fires | New events discovered on own relay | Update RepoSyncIndex → compute_actions | | ||
| 187 | | fresh_start() | Initial connect, long_reconnect, daily | After L1 setup → compute_actions | | ||
| 188 | | quick_reconnect() | Reconnect < 15 minutes | After L1+L2+L3 catchup → compute_actions | | ||
| 189 | | consolidate() | Filter count > threshold | After live rebuild → compute_actions | | ||
| 190 | |||
| 191 | ### The Core Flow | ||
| 151 | 192 | ||
| 152 | ```mermaid | 193 | ```mermaid |
| 153 | flowchart TB | 194 | flowchart TB |
| 154 | START[Startup] --> SS[Self-subscribe to own relay] | 195 | TRIGGER[Trigger fires] --> CA[compute_actions] |
| 155 | SS --> |no since filter| EVENTS[Receive historical events] | 196 | CA --> |derives from| RSI[RepoSyncIndex] |
| 156 | EVENTS --> RSI[Update RepoSyncIndex] | 197 | CA --> |subtracts| RLI[RelaySyncIndex] |
| 157 | RSI --> DT[derive_relay_targets] | 198 | CA --> |subtracts| PSI[PendingSyncIndex] |
| 158 | DT --> CA[compute_actions with targets and empty confirmed] | 199 | CA --> |produces| AF[AddFilters actions] |
| 159 | CA --> AF[AddFilters for each relay] | 200 | AF --> SFRE[sync_computed_filters] |
| 160 | AF --> SPAWN{Relay connected?} | 201 | SFRE --> LIVE[sync_live - L2+L3] |
| 161 | SPAWN --> |no| CONN[spawn_connection] | 202 | SFRE --> HIST[historic_sync - L2+L3] |
| 162 | CONN --> HC[handle_connect_or_reconnect] | 203 | HIST --> PSI_UPDATE[Update PendingSyncIndex] |
| 163 | SPAWN --> |yes| SUB | 204 | PSI_UPDATE --> |EOSE received| CONFIRM[Move to RelaySyncIndex] |
| 164 | 205 | ``` | |
| 165 | subgraph handle_connect_or_reconnect - Fresh Sync | 206 | |
| 166 | HC --> CHECK_FRESH{is_fresh_sync?} | 207 | --- |
| 167 | CHECK_FRESH --> |yes - no last_connected| L1[build_announcement_filter - no since] | 208 | |
| 168 | L1 --> RCA[recompute_actions_for_relay] | 209 | ## Flow Scenarios |
| 169 | end | ||
| 170 | 210 | ||
| 171 | RCA --> SUB[Subscribe Layer 2+3 filters via AddFilters] | 211 | ### Scenario 1: Fresh Start (Initial Connect / Long Reconnect / Daily Sync) |
| 172 | SUB --> PB[Create PendingBatch] | 212 | |
| 213 | ```mermaid | ||
| 214 | flowchart TB | ||
| 215 | START[fresh_start called] --> CLEAR_PSI[Clear PendingSyncIndex] | ||
| 216 | CLEAR_PSI --> CLEAR_RSI[Clear RelaySyncIndex] | ||
| 217 | CLEAR_RSI --> L1_LIVE[L1: sync_live - announcements] | ||
| 218 | L1_LIVE --> L1_HIST[L1: historic_sync - no since] | ||
| 219 | L1_HIST --> NEG{NIP-77 supported?} | ||
| 220 | NEG --> |yes| NEGENTROPY[negentropy sync] | ||
| 221 | NEG --> |no| REQ[REQ+EOSE] | ||
| 222 | NEGENTROPY --> CA[compute_actions] | ||
| 223 | REQ --> CA | ||
| 224 | CA --> |empty RelaySyncIndex| AF[AddFilters for ALL repos] | ||
| 225 | AF --> SFRE[sync_computed_filters] | ||
| 226 | SFRE --> L23_LIVE[L2+L3: sync_live] | ||
| 227 | SFRE --> L23_HIST[L2+L3: historic_sync] | ||
| 228 | L23_HIST --> PB[Create PendingBatch] | ||
| 173 | PB --> EOSE[Wait for EOSE] | 229 | PB --> EOSE[Wait for EOSE] |
| 174 | EOSE --> CONFIRM[Move items to confirmed repos/root_events] | 230 | EOSE --> CONFIRM[Move items to RelaySyncIndex] |
| 175 | ``` | 231 | ``` |
| 176 | 232 | ||
| 177 | **Key points:** | 233 | **Key points:** |
| 178 | 234 | ||
| 179 | - No `since` filter on initial connect - get full history | 235 | - Always clear PendingSyncIndex first, then RelaySyncIndex |
| 180 | - `handle_connect_or_reconnect` detects `is_fresh_sync` via `last_connected.is_none()` | 236 | - L1 live + L1 historic (uses negentropy if available) |
| 181 | - Layer 1: `build_announcement_filter(None)` - subscribed immediately without since | 237 | - Empty RelaySyncIndex means diff produces AddFilters for everything |
| 182 | - Layer 2+3: handled via `recompute_actions_for_relay` → `compute_actions` with PendingBatch tracking | 238 | - L2+L3 flow through sync_computed_filters with proper pending tracking |
| 183 | 239 | ||
| 184 | ### Scenario 2: Quick Reconnect (less than 15 minutes) | 240 | ### Scenario 2: Quick Reconnect (< 15 minutes) |
| 185 | 241 | ||
| 186 | ```mermaid | 242 | ```mermaid |
| 187 | flowchart TB | 243 | flowchart TB |
| 188 | DISC[Connection lost] --> MARK[Set disconnected_at = now] | 244 | DISC[Connection lost] --> MARK[Set disconnected_at = now] |
| 189 | MARK --> CLEAR_PEND[Clear PendingSyncIndex for relay] | 245 | MARK --> WAIT[Wait for reconnection < 15min] |
| 190 | CLEAR_PEND --> WAIT[Wait for reconnection] | ||
| 191 | WAIT --> RECONN[Connection restored] | 246 | WAIT --> RECONN[Connection restored] |
| 192 | RECONN --> HC[handle_connect_or_reconnect] | 247 | RECONN --> CLEAR_PSI[Clear PendingSyncIndex] |
| 193 | 248 | CLEAR_PSI --> L1_LIVE[L1: sync_live - announcements] | |
| 194 | subgraph handle_connect_or_reconnect - Quick Reconnect | 249 | L1_LIVE --> L1_HIST[L1: historic_sync WITH since] |
| 195 | HC --> CHECK{is_fresh_sync?} | 250 | L1_HIST --> RECON[reconstruct_filters from RelaySyncIndex] |
| 196 | CHECK --> |no - last_connected exists AND less than 15min| SINCE[since = last_connected - 15min] | 251 | RECON --> L23_LIVE[L2+L3: sync_live] |
| 197 | SINCE --> L1[build_announcement_filter - with since] | 252 | RECON --> L23_HIST[L2+L3: historic_sync WITH since] |
| 198 | L1 --> L23[rebuild_layer2_and_layer3 - with since] | 253 | L23_HIST --> CA[compute_actions] |
| 199 | L23 --> RCA[recompute_actions_for_relay] | 254 | CA --> |check for new items| AF{New items?} |
| 200 | end | 255 | AF --> |yes| SFRE[sync_computed_filters] |
| 201 | 256 | AF --> |no| DONE[Done] | |
| 202 | RCA --> AF[AddFilters for new items only] | 257 | SFRE --> PB[Create PendingBatch] |
| 203 | AF --> SUB[Subscribe] | ||
| 204 | SUB --> PB[Create PendingBatch] | ||
| 205 | PB --> EOSE[Wait for EOSE] | ||
| 206 | EOSE --> EXTEND[Extend confirmed state] | ||
| 207 | ``` | 258 | ``` |
| 208 | 259 | ||
| 209 | **Key points:** | 260 | **Key points:** |
| 210 | 261 | ||
| 211 | - PendingSyncIndex cleared on disconnect (not reconnect) | 262 | - Clear PendingSyncIndex first (old subscriptions are dead) |
| 212 | - `handle_connect_or_reconnect`: | 263 | - L1 live (always on any connection) |
| 213 | 1. `build_announcement_filter(Some(since))` - Layer 1 with since | 264 | - L1 historic WITH since (catches up missed announcements) |
| 214 | 2. `rebuild_layer2_and_layer3(since)` - Layer 2+3 with since | 265 | - L2+L3 rebuilt from RelaySyncIndex (confirmed state preserved) |
| 215 | 3. `recompute_actions_for_relay` - check for new items | 266 | - compute_actions checks for any NEW items discovered during catchup |
| 216 | - since = last_connected - 15min ensures we catch events during disconnection | ||
| 217 | 267 | ||
| 218 | ### Scenario 3: Stale Reconnect (greater than 15 minutes) | 268 | ### Scenario 3: Long Reconnect (> 15 minutes) |
| 219 | 269 | ||
| 220 | ```mermaid | 270 | ```mermaid |
| 221 | flowchart TB | 271 | flowchart TB |
| 222 | RECONN[Connection restored] --> HC[handle_connect_or_reconnect] | 272 | RECONN[Connection restored > 15min] --> METRIC[Record disconnect/reconnect metric] |
| 273 | METRIC --> FRESH[fresh_start] | ||
| 274 | FRESH --> |same as initial connect| DONE[Full sync initiated] | ||
| 275 | ``` | ||
| 223 | 276 | ||
| 224 | subgraph handle_connect_or_reconnect - Stale Reconnect | 277 | **Key points:** |
| 225 | HC --> CHECK{is_fresh_sync?} | ||
| 226 | CHECK --> |yes - disconnected greater than 15min| CLEAR[clear_sync_state] | ||
| 227 | CLEAR --> L1[build_announcement_filter - no since] | ||
| 228 | L1 --> RCA[recompute_actions_for_relay] | ||
| 229 | end | ||
| 230 | 278 | ||
| 231 | RCA --> CA[compute_actions with empty confirmed] | 279 | - Records disconnect/reconnect as a metric |
| 232 | CA --> AF[AddFilters for everything] | 280 | - Delegates to fresh_start() - same as initial connect |
| 233 | AF --> SUB[Subscribe - no since filter] | 281 | - State too stale to trust, start fresh |
| 234 | SUB --> PB[Create PendingBatch] | 282 | |
| 235 | PB --> EOSE[Wait for EOSE] | 283 | ### Scenario 4: Consolidation (Filter Count > Threshold) |
| 236 | EOSE --> CONFIRM[Populate confirmed state fresh] | 284 | |
| 285 | ```mermaid | ||
| 286 | flowchart TB | ||
| 287 | CHECK[Filter count check] --> THRESHOLD{count > 70?} | ||
| 288 | THRESHOLD --> |yes| CLEAR_PSI[Clear PendingSyncIndex] | ||
| 289 | CLEAR_PSI --> UNSUB[unsubscribe_all] | ||
| 290 | UNSUB --> RECON[reconstruct_filters from RelaySyncIndex] | ||
| 291 | RECON --> L1_LIVE[L1: sync_live] | ||
| 292 | RECON --> L23_LIVE[L2+L3: sync_live] | ||
| 293 | L23_LIVE --> CA[compute_actions] | ||
| 294 | CA --> |check for new items| AF{New items?} | ||
| 295 | AF --> |yes| SFRE[sync_computed_filters] | ||
| 296 | AF --> |no| DONE[Done] | ||
| 297 | THRESHOLD --> |no| SKIP[Continue normally] | ||
| 237 | ``` | 298 | ``` |
| 238 | 299 | ||
| 239 | **Key points:** | 300 | **Key points:** |
| 240 | 301 | ||
| 241 | - `should_clear_state()` returns true → triggers fresh sync | 302 | - Clear PendingSyncIndex first |
| 242 | - Same path as initial connect after clearing state | 303 | - NO historic sync needed - items already synced/syncing |
| 243 | - Layer 1: `build_announcement_filter(None)` - full history | 304 | - Only rebuilds live subscriptions from confirmed state |
| 244 | - Layer 2+3: handled via empty confirmed state → compute_actions generates AddFilters for everything | 305 | - compute_actions catches any new items that need syncing |
| 245 | 306 | ||
| 246 | ### Scenario 4: Consolidation (Triggered on Filter Add) | 307 | ### Scenario 5: Daily Sync (23-25h Random Timer) |
| 247 | 308 | ||
| 248 | ```mermaid | 309 | ```mermaid |
| 249 | flowchart TB | 310 | flowchart TB |
| 250 | AF[handle_add_filters called] --> COUNT{current + new > 70?} | 311 | TIMER[Daily timer fires] --> FRESH[fresh_start] |
| 251 | COUNT --> |yes| CONSOLIDATE[consolidate] | 312 | FRESH --> |NO disconnect metric| DONE[Full sync initiated] |
| 252 | CONSOLIDATE --> WAIT_PEND[wait_pending_complete] | ||
| 253 | WAIT_PEND --> CLOSE[unsubscribe_all] | ||
| 254 | CLOSE --> SINCE[since = now - 15min] | ||
| 255 | SINCE --> L1[build_announcement_filter - with since] | ||
| 256 | L1 --> L23[rebuild_layer2_and_layer3 - with since] | ||
| 257 | COUNT --> |no| SUB[Subscribe new filters] | ||
| 258 | SUB --> PB[Create PendingBatch] | ||
| 259 | ``` | 313 | ``` |
| 260 | 314 | ||
| 261 | **Key points:** | 315 | **Key points:** |
| 262 | 316 | ||
| 263 | - Consolidation checked in `handle_add_filters` BEFORE adding new filters | 317 | - Same as fresh_start() but WITHOUT recording disconnect/reconnect metric |
| 264 | - After closing all subscriptions, re-subscribe: | 318 | - Ensures consistency, detects any drift accumulated over 24 hours |
| 265 | 1. `build_announcement_filter(Some(since))` - Layer 1 stays active with since | ||
| 266 | 2. `rebuild_layer2_and_layer3(since)` - Layer 2+3 with since | ||
| 267 | - `since = now - 15min` prevents re-fetching old events | ||
| 268 | - Keeps confirmed state, just reduces filter count | ||
| 269 | 319 | ||
| 270 | ### Scenario 5: Daily Timer (23-25h Random) | 320 | ### Scenario 6: Self-Subscriber Batch |
| 271 | 321 | ||
| 272 | ```mermaid | 322 | ```mermaid |
| 273 | flowchart TB | 323 | flowchart TB |
| 274 | DAILY[Daily timer fires] --> CLOSE[unsubscribe_all] | 324 | EVENTS[Events from own relay] --> QUEUE[Queue to pending batch] |
| 275 | CLOSE --> CLEAR_PEND[Clear PendingSyncIndex for relay] | 325 | QUEUE --> TIMER[Batch timer fires - 5 seconds] |
| 276 | CLEAR_PEND --> CLEAR_STATE[clear_sync_state] | 326 | TIMER --> UPDATE[Update RepoSyncIndex] |
| 277 | CLEAR_STATE --> L1[build_announcement_filter - no since] | 327 | UPDATE --> CA[compute_actions] |
| 278 | L1 --> RCA[recompute_actions_for_relay] | 328 | CA --> |new repos/events discovered| AF[AddFilters] |
| 279 | RCA --> CA[compute_actions with empty confirmed] | 329 | AF --> SFRE[sync_computed_filters] |
| 280 | CA --> AF[AddFilters for everything] | 330 | SFRE --> LIVE[sync_live - L2+L3] |
| 281 | AF --> SUB[Subscribe - no since filter] | 331 | SFRE --> HIST[historic_sync - L2+L3] |
| 282 | SUB --> PB[Create PendingBatch] | ||
| 283 | PB --> EOSE[Wait for EOSE] | ||
| 284 | EOSE --> CONFIRM[Repopulate confirmed state] | ||
| 285 | ``` | 332 | ``` |
| 286 | 333 | ||
| 287 | **Key points:** | 334 | **Key points:** |
| 288 | 335 | ||
| 289 | - Daily timer is a full fresh sync, NOT consolidation | 336 | - Self-subscriber monitors own relay for 30617, 1617, 1618, 1619, 1621 |
| 290 | - Clears both PendingSyncIndex and confirmed state | 337 | - Batches events (5 second window) |
| 291 | - Layer 1: `build_announcement_filter(None)` - full history | 338 | - Updates RepoSyncIndex, then compute_actions finds new work |
| 292 | - Layer 2+3: via compute_actions with empty confirmed - full history | 339 | - New items flow through sync_computed_filters |
| 293 | - Detects any state drift accumulated over 24 hours | ||
| 294 | 340 | ||
| 295 | --- | 341 | --- |
| 296 | 342 | ||
| @@ -300,8 +346,6 @@ flowchart TB | |||
| 300 | 346 | ||
| 301 | Transforms the repo-centric `RepoSyncIndex` into a relay-centric view. For each relay URL mentioned in any repo's announcements, collects all the repos and root events that should be synced from that relay. | 347 | Transforms the repo-centric `RepoSyncIndex` into a relay-centric view. For each relay URL mentioned in any repo's announcements, collects all the repos and root events that should be synced from that relay. |
| 302 | 348 | ||
| 303 | **Implementation:** [`derive_relay_targets()`](../../src/sync/algorithms.rs:61) | ||
| 304 | |||
| 305 | ```rust | 349 | ```rust |
| 306 | // Conceptual: inverts repo → relays to relay → repos | 350 | // Conceptual: inverts repo → relays to relay → repos |
| 307 | fn derive_relay_targets(repo_index: &HashMap<String, RepoSyncNeeds>) | 351 | fn derive_relay_targets(repo_index: &HashMap<String, RepoSyncNeeds>) |
| @@ -320,104 +364,262 @@ Performs a three-way diff: `target - pending - confirmed = new` | |||
| 320 | 364 | ||
| 321 | Only creates `AddFilters` actions for items not already pending or confirmed. Skips disconnected relays (they will get AddFilters on reconnect). | 365 | Only creates `AddFilters` actions for items not already pending or confirmed. Skips disconnected relays (they will get AddFilters on reconnect). |
| 322 | 366 | ||
| 323 | **Implementation:** [`compute_actions()`](../../src/sync/algorithms.rs:96) | 367 | ```rust |
| 368 | fn compute_actions( | ||
| 369 | targets: &HashMap<String, RelaySyncNeeds>, | ||
| 370 | pending: &PendingSyncIndex, | ||
| 371 | confirmed: &RelaySyncIndex, | ||
| 372 | ) -> Vec<AddFilters> | ||
| 373 | ``` | ||
| 324 | 374 | ||
| 325 | --- | 375 | --- |
| 326 | 376 | ||
| 327 | ## Filter Building (Three-Layer Strategy) | 377 | ## Method Specifications |
| 378 | |||
| 379 | ### Primitives | ||
| 380 | |||
| 381 | #### `sync_live()` - Live Subscriptions | ||
| 382 | |||
| 383 | ```rust | ||
| 384 | /// Set up live subscription (filters with limit: 0) | ||
| 385 | /// | ||
| 386 | /// - Uses `limit: 0` to receive only new events | ||
| 387 | /// - NOT tracked in PendingSyncIndex (state reconstructable) | ||
| 388 | async fn sync_live(&self, relay_url: &str, filters: &[Filter]) | ||
| 389 | ``` | ||
| 390 | |||
| 391 | #### `historic_sync()` - Historical Sync Dispatcher | ||
| 328 | 392 | ||
| 329 | The filter strategy uses three layers to ensure comprehensive event coverage: | 393 | ```rust |
| 394 | /// Dispatch to appropriate historic sync method based on relay capabilities | ||
| 395 | /// | ||
| 396 | /// Both paths update PendingSyncIndex to ensure consistent lifecycle tracking. | ||
| 397 | async fn historic_sync( | ||
| 398 | &mut self, | ||
| 399 | relay_url: &str, | ||
| 400 | filters: Vec<Filter>, | ||
| 401 | items: PendingItems, | ||
| 402 | since: Option<Timestamp>, | ||
| 403 | ) -> Option<u64> // Returns batch_id | ||
| 404 | ``` | ||
| 405 | |||
| 406 | Dispatches to: | ||
| 407 | |||
| 408 | - `historic_sync_negentropy()` - NIP-77 parallel sync (if supported) | ||
| 409 | - `historic_sync_legacy()` - REQ+EOSE fallback | ||
| 410 | |||
| 411 | ### Building Blocks | ||
| 412 | |||
| 413 | #### `reconstruct_filters()` - Rebuild from Confirmed State | ||
| 414 | |||
| 415 | ```rust | ||
| 416 | /// Reconstruct filters from RelaySyncIndex (confirmed state ONLY) | ||
| 417 | /// | ||
| 418 | /// Returns raw Vec<Filter> for L1+L2+L3. | ||
| 419 | /// Used by: quick_reconnect, consolidate | ||
| 420 | /// Does NOT include pending items - those flow through AddFilters path. | ||
| 421 | async fn reconstruct_filters(&self, relay_url: &str) -> Vec<Filter> | ||
| 422 | ``` | ||
| 423 | |||
| 424 | #### `sync_computed_filters()` - Handle New AddFilters | ||
| 425 | |||
| 426 | ```rust | ||
| 427 | /// Process AddFilters action (from compute_actions) | ||
| 428 | /// | ||
| 429 | /// Orchestrates both live and historic sync for NEW items: | ||
| 430 | /// 1. sync_live() - set up permanent L2+L3 subscriptions | ||
| 431 | /// 2. historic_sync() - catch up on past events | ||
| 432 | /// | ||
| 433 | /// This is specifically for NEW filter discovery. | ||
| 434 | async fn sync_computed_filters( | ||
| 435 | &mut self, | ||
| 436 | action: AddFilters, | ||
| 437 | since: Option<Timestamp>, | ||
| 438 | ) -> Option<u64> | ||
| 439 | ``` | ||
| 440 | |||
| 441 | ### Top-Level Entry Points | ||
| 442 | |||
| 443 | #### `fresh_start()` - Clean Slate Sync | ||
| 444 | |||
| 445 | ```rust | ||
| 446 | /// Fresh start - clears state and does full sync | ||
| 447 | /// | ||
| 448 | /// Called by: initial connect, long_reconnect, daily_sync | ||
| 449 | /// | ||
| 450 | /// Flow: | ||
| 451 | /// 1. Clear PendingSyncIndex | ||
| 452 | /// 2. Clear RelaySyncIndex | ||
| 453 | /// 3. L1 live + L1 historic (negentropy if available) | ||
| 454 | /// 4. compute_actions → AddFilters → sync_computed_filters for L2+L3 | ||
| 455 | async fn fresh_start(&mut self, relay_url: &str) | ||
| 456 | ``` | ||
| 457 | |||
| 458 | #### `quick_reconnect()` - Short Disconnection Recovery | ||
| 459 | |||
| 460 | ```rust | ||
| 461 | /// Quick reconnect - for disconnections < 15 minutes | ||
| 462 | /// | ||
| 463 | /// Flow: | ||
| 464 | /// 1. Clear PendingSyncIndex | ||
| 465 | /// 2. L1 live + L1 historic(since) | ||
| 466 | /// 3. reconstruct_filters → L2+L3 live + L2+L3 historic(since) | ||
| 467 | /// 4. compute_actions for any new items | ||
| 468 | async fn quick_reconnect(&mut self, relay_url: &str, since: Timestamp) | ||
| 469 | ``` | ||
| 470 | |||
| 471 | #### `long_reconnect()` - Extended Disconnection Recovery | ||
| 472 | |||
| 473 | ```rust | ||
| 474 | /// Long reconnect - for disconnections > 15 minutes | ||
| 475 | /// | ||
| 476 | /// Flow: | ||
| 477 | /// 1. Record disconnect/reconnect metric | ||
| 478 | /// 2. fresh_start() | ||
| 479 | async fn long_reconnect(&mut self, relay_url: &str) | ||
| 480 | ``` | ||
| 481 | |||
| 482 | #### `daily_sync()` - Scheduled Full Refresh | ||
| 483 | |||
| 484 | ```rust | ||
| 485 | /// Daily sync - full refresh without disconnect metrics | ||
| 486 | /// | ||
| 487 | /// Flow: fresh_start() (no disconnect metric recorded) | ||
| 488 | async fn daily_sync(&mut self, relay_url: &str) | ||
| 489 | ``` | ||
| 490 | |||
| 491 | #### `consolidate()` - Filter Count Reduction | ||
| 492 | |||
| 493 | ```rust | ||
| 494 | /// Consolidate subscriptions when filter count exceeds threshold | ||
| 495 | /// | ||
| 496 | /// Flow: | ||
| 497 | /// 1. Clear PendingSyncIndex | ||
| 498 | /// 2. unsubscribe_all | ||
| 499 | /// 3. reconstruct_filters → sync_live only (L1+L2+L3) | ||
| 500 | /// 4. compute_actions for any new items | ||
| 501 | /// | ||
| 502 | /// NO historic sync - items already synced, just reducing subscriptions | ||
| 503 | async fn consolidate(&mut self, relay_url: &str) | ||
| 504 | ``` | ||
| 505 | |||
| 506 | #### `handle_new_sync_filters()` - New Filter Discovery | ||
| 507 | |||
| 508 | ```rust | ||
| 509 | /// Handle AddFilters action from compute_actions | ||
| 510 | /// | ||
| 511 | /// Flow: | ||
| 512 | /// 1. Check/spawn connection if needed | ||
| 513 | /// 2. maybe_consolidate (check filter threshold) | ||
| 514 | /// 3. sync_computed_filters | ||
| 515 | async fn handle_new_sync_filters(&mut self, action: AddFilters) | ||
| 516 | ``` | ||
| 517 | |||
| 518 | --- | ||
| 519 | |||
| 520 | ## Method Relationships Summary | ||
| 521 | |||
| 522 | ``` | ||
| 523 | fresh_start(relay_url) // Initial/long_reconnect/daily | ||
| 524 | ├──> Clear PendingSyncIndex | ||
| 525 | ├──> Clear RelaySyncIndex | ||
| 526 | ├──> L1: sync_live(announcement_filter) | ||
| 527 | ├──> L1: historic_sync(announcement_filter, None) | ||
| 528 | └──> compute_actions → AddFilters → sync_computed_filters (L2+L3) | ||
| 529 | |||
| 530 | quick_reconnect(relay_url, since) // Disconnected < 15 min | ||
| 531 | ├──> Clear PendingSyncIndex | ||
| 532 | ├──> L1: sync_live(announcement_filter) | ||
| 533 | ├──> L1: historic_sync(announcement_filter, since) | ||
| 534 | ├──> reconstruct_filters() → L2+L3 filters | ||
| 535 | ├──> L2+L3: sync_live(filters) | ||
| 536 | ├──> L2+L3: historic_sync(filters, since) | ||
| 537 | └──> compute_actions → AddFilters → sync_computed_filters (new items only) | ||
| 538 | |||
| 539 | long_reconnect(relay_url) // Disconnected > 15 min | ||
| 540 | ├──> Record disconnect/reconnect metric | ||
| 541 | └──> fresh_start() | ||
| 542 | |||
| 543 | daily_sync(relay_url) // Timer fires | ||
| 544 | └──> fresh_start() // No disconnect metric | ||
| 545 | |||
| 546 | consolidate(relay_url) // Filter count > threshold | ||
| 547 | ├──> Clear PendingSyncIndex | ||
| 548 | ├──> unsubscribe_all() | ||
| 549 | ├──> reconstruct_filters() → L1+L2+L3 filters | ||
| 550 | ├──> sync_live(filters) // Live only, NO historic | ||
| 551 | └──> compute_actions → AddFilters → sync_computed_filters (new items only) | ||
| 552 | |||
| 553 | handle_new_sync_filters(action) // New filter discovery | ||
| 554 | ├──> Check/spawn connection | ||
| 555 | ├──> maybe_consolidate() | ||
| 556 | └──> sync_computed_filters(action, None) | ||
| 557 | |||
| 558 | sync_computed_filters(action, since) // Process AddFilters | ||
| 559 | ├──> sync_live(action.filters) // L2+L3 live | ||
| 560 | └──> historic_sync(action.filters, since) // L2+L3 historic | ||
| 561 | ├── historic_sync_negentropy() // Parallel, updates Pending | ||
| 562 | └── historic_sync_legacy() // REQ+EOSE, updates Pending | ||
| 563 | ``` | ||
| 564 | |||
| 565 | --- | ||
| 566 | |||
| 567 | ## Filter Building (Three-Layer Strategy) | ||
| 330 | 568 | ||
| 331 | ### Layer 1: Announcements | 569 | ### Layer 1: Announcements |
| 332 | 570 | ||
| 333 | - **Kinds**: 30617 (Repository Announcements), 30618 (Maintainer Lists) | 571 | - **Kinds**: 30617 (Repository Announcements), 30618 (Maintainer Lists) |
| 334 | - **When subscribed**: ONCE on connect, NOT rebuilt during consolidation | 572 | - **When subscribed**: On connect (any type) - handled by connection lifecycle |
| 335 | - **Function**: [`build_announcement_filter()`](../../src/sync/filters.rs:20) | 573 | - **Function**: `build_announcement_filter(since: Option<Timestamp>)` |
| 336 | - 30618 is ONLY synced from remote relays, not self-subscribed | 574 | - 30618 is ONLY synced from remote relays, not self-subscribed |
| 337 | 575 | ||
| 338 | ### Layer 2: Events Tagging Our Repos | 576 | ### Layer 2: Events Tagging Our Repos |
| 339 | 577 | ||
| 340 | - **Tags**: lowercase `a`, uppercase `A`, and `q` tags for comprehensive coverage | 578 | - **Tags**: lowercase `a`, uppercase `A`, and `q` tags for comprehensive coverage |
| 341 | - **Batching**: Per 100 repo refs | 579 | - **Batching**: Per 100 repo refs |
| 342 | - **Function**: [`tagged_one_of_our_repo_event_filters()`](../../src/sync/filters.rs:43) | 580 | - **Function**: `build_repo_tag_filters(repos, since)` |
| 343 | 581 | ||
| 344 | ### Layer 3: Events Tagging Our Root Events | 582 | ### Layer 3: Events Tagging Our Root Events |
| 345 | 583 | ||
| 346 | - **Tags**: lowercase `e`, uppercase `E`, and `q` tags for comprehensive coverage | 584 | - **Tags**: lowercase `e`, uppercase `E`, and `q` tags for comprehensive coverage |
| 347 | - **Batching**: Per 100 event IDs | 585 | - **Batching**: Per 100 event IDs |
| 348 | - **Function**: [`tagged_one_of_our_root_event_filters()`](../../src/sync/filters.rs:98) | 586 | - **Function**: `build_root_event_tag_filters(root_events, since)` |
| 349 | 587 | ||
| 350 | ### Combined Layer 2+3 | 588 | ### Combined Layer 2+3 |
| 351 | 589 | ||
| 352 | The [`build_layer2_and_layer3_filters()`](../../src/sync/filters.rs:152) function combines both layers. Used by: | 590 | The `build_layer2_and_layer3_filters()` function combines both layers. Used by: |
| 353 | |||
| 354 | - `compute_actions` for incremental subscriptions | ||
| 355 | - `rebuild_layer2_and_layer3` during reconnection | ||
| 356 | - Consolidation rebuilds (Layer 1 remains active separately) | ||
| 357 | 591 | ||
| 358 | **Key insight**: Layer 1 is connection-level (subscribe once), Layer 2+3 are item-level (managed by compute_actions and PendingBatch). | 592 | - `sync_computed_filters` for new item subscriptions |
| 593 | - `reconstruct_filters` for rebuilding from confirmed state | ||
| 359 | 594 | ||
| 360 | --- | 595 | --- |
| 361 | 596 | ||
| 362 | ## SyncManager Key Methods | 597 | ## NIP-77 Negentropy Sync |
| 363 | |||
| 364 | The [`SyncManager`](../../src/sync/mod.rs:308) orchestrates all sync operations. Key methods: | ||
| 365 | |||
| 366 | ### Connection Lifecycle | ||
| 367 | |||
| 368 | | Method | Purpose | | ||
| 369 | | ------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------- | | ||
| 370 | | `handle_connect_or_reconnect()` | Unified handler for initial connect and reconnect. Determines fresh vs quick reconnect based on `last_connected` and 15-minute rule | | ||
| 371 | | `handle_disconnect()` | Updates RelayState to Disconnected, sets disconnected_at, clears pending batches, records failure in health tracker | | ||
| 372 | | `spawn_relay_connection()` | Creates RelayConnection, subscribes to Layer 1, spawns event loop task | | ||
| 373 | |||
| 374 | ### Sync Operations | ||
| 375 | |||
| 376 | | Method | Purpose | | ||
| 377 | | ------------------------------- | ------------------------------------------------------------------------------------------------------------------- | | ||
| 378 | | `handle_add_filters()` | Auto-spawns connection if needed, checks consolidation threshold (>70 filters), subscribes and creates PendingBatch | | ||
| 379 | | `handle_eose()` | Processes EOSE for subscription, moves items from pending to confirmed when batch completes | | ||
| 380 | | `recompute_actions_for_relay()` | Runs derive_relay_targets → compute_actions for a specific relay to find new items | | ||
| 381 | | `rebuild_layer2_and_layer3()` | Rebuilds subscriptions from confirmed state with optional since filter | | ||
| 382 | |||
| 383 | ### Maintenance | ||
| 384 | |||
| 385 | | Method | Purpose | | ||
| 386 | | --------------------- | -------------------------------------------------------------------------- | | ||
| 387 | | `daily_sync()` | Full fresh sync - unsubscribes all, clears state, recomputes actions | | ||
| 388 | | `consolidate()` | Reduces filter count by unsubscribing and rebuilding with combined filters | | ||
| 389 | | `check_disconnects()` | Periodic check for empty relays (no repos) to disconnect | | ||
| 390 | | `check_reconnects()` | Attempts reconnection for disconnected relays with pending work | | ||
| 391 | 598 | ||
| 392 | --- | 599 | ### What is Negentropy? |
| 393 | 600 | ||
| 394 | ## Self-Subscriber | 601 | NIP-77 defines the negentropy protocol for efficient event set comparison. Instead of requesting all events matching a filter (REQ+EOSE), negentropy allows relays to compare fingerprints of their event sets and only transfer the differences. |
| 395 | 602 | ||
| 396 | The [`SelfSubscriber`](../../src/sync/self_subscriber.rs:86) monitors our own relay for repository announcements and root events, updating the `RepoSyncIndex`. | 603 | ### When Negentropy is Used |
| 397 | 604 | ||
| 398 | ### Event Kinds Monitored | 605 | Negentropy sync is attempted for: |
| 399 | 606 | ||
| 400 | - **30617** - Repository Announcements (triggers discovery of repos listing our relay) | 607 | - **fresh_start()** - Full sync without `since` |
| 401 | - **1617** - Patches (root events referencing repos) | 608 | - **daily_sync()** - Periodic full refresh (via fresh_start) |
| 402 | - **1618** - Issues | 609 | - **long_reconnect()** - Via fresh_start |
| 403 | - **1619** - Replies/Status | ||
| 404 | - **1621** - Pull Requests | ||
| 405 | 610 | ||
| 406 | Note: 30618 (Maintainer Lists) is NOT self-subscribed - only synced from remote relays. | 611 | Negentropy is NOT used for: |
| 407 | 612 | ||
| 408 | ### Batching Flow | 613 | - **quick_reconnect()** - Uses REQ with `since` (more efficient for small gaps) |
| 614 | - **Live subscriptions** - Always use REQ with `limit: 0` | ||
| 409 | 615 | ||
| 410 | 1. **Receive events** from own relay subscription | 616 | ### Fallback Behavior |
| 411 | 2. **Queue to pending** - announcements get repo ID + relay URLs; root events get repo ref + event ID | ||
| 412 | 3. **Timer fires** (configurable window, default 5 seconds) - does NOT reset on new events | ||
| 413 | 4. **Process batch**: | ||
| 414 | - Update `RepoSyncIndex` with discovered repos and root events | ||
| 415 | - Call `derive_relay_targets()` → `compute_actions()` | ||
| 416 | - Send `AddFilters` actions to SyncManager | ||
| 417 | 617 | ||
| 418 | ### Reconnection | 618 | If negentropy fails (relay doesn't support NIP-77, network error, etc.): |
| 419 | 619 | ||
| 420 | Uses `last_connected` timestamp to apply since filter on reconnect (15-minute buffer), similar to external relay reconnection logic. | 620 | 1. A warning is logged (once per relay to avoid spam) |
| 621 | 2. The sync falls back to traditional REQ+EOSE | ||
| 622 | 3. No error is raised - fallback is automatic | ||
| 421 | 623 | ||
| 422 | --- | 624 | --- |
| 423 | 625 | ||
| @@ -431,12 +633,14 @@ flowchart TB | |||
| 431 | end | 633 | end |
| 432 | 634 | ||
| 433 | subgraph RepoSyncIndex - What We Want | 635 | subgraph RepoSyncIndex - What We Want |
| 434 | RSI[HashMap: Repo to Relays+Events] | 636 | RSI[HashMap: Repo → Relays+Events] |
| 435 | end | 637 | end |
| 436 | 638 | ||
| 437 | subgraph Derived Target | 639 | subgraph Triggers |
| 438 | DT[derive_relay_targets fn] | 640 | T1[Self-subscriber batch] |
| 439 | TGT[Per-relay: repos + events we should sync] | 641 | T2[fresh_start after L1] |
| 642 | T3[quick_reconnect after catchup] | ||
| 643 | T4[consolidate after live rebuild] | ||
| 440 | end | 644 | end |
| 441 | 645 | ||
| 442 | subgraph compute_actions - Decision Point | 646 | subgraph compute_actions - Decision Point |
| @@ -449,136 +653,41 @@ flowchart TB | |||
| 449 | 653 | ||
| 450 | subgraph RelaySyncIndex - Confirmed State | 654 | subgraph RelaySyncIndex - Confirmed State |
| 451 | RLI[RelayState per relay] | 655 | RLI[RelayState per relay] |
| 452 | CONN[connection_status] | ||
| 453 | REPOS[repos + root_events] | ||
| 454 | TIMES[last_connected + disconnected_at] | ||
| 455 | end | 656 | end |
| 456 | 657 | ||
| 457 | SS -->|subscribe| OWN | 658 | SS -->|subscribe| OWN |
| 458 | OWN -->|events| SS | 659 | OWN -->|events| SS |
| 459 | SS -->|batch fires| RSI | 660 | SS -->|batch fires| RSI |
| 460 | RSI --> DT | 661 | RSI --> T1 |
| 461 | DT --> TGT | 662 | T1 --> CA |
| 462 | TGT --> CA | 663 | T2 --> CA |
| 664 | T3 --> CA | ||
| 665 | T4 --> CA | ||
| 463 | PSI --> CA | 666 | PSI --> CA |
| 464 | RLI --> CA | 667 | RLI --> CA |
| 465 | CA -->|Layer 2+3 new items| AF[AddFilters] | 668 | CA -->|new items| AF[AddFilters] |
| 466 | AF -->|check filter count| CONSOL{count + new > 70?} | 669 | AF --> SFRE[sync_computed_filters] |
| 467 | CONSOL -->|yes| CONSOLIDATE[consolidate] | 670 | SFRE --> LIVE[sync_live L2+L3] |
| 468 | CONSOLIDATE --> L1_CONSOL[build_announcement_filter with since] | 671 | SFRE --> HIST[historic_sync L2+L3] |
| 469 | L1_CONSOL --> L23_CONSOL[rebuild_layer2_and_layer3 with since] | 672 | HIST --> PSI |
| 470 | CONSOL -->|no| SUB[subscribe] | 673 | PSI -->|EOSE| RLI |
| 471 | AF -->|spawn if needed| CONN | ||
| 472 | SUB --> PSI | ||
| 473 | PSI -->|EOSE| REPOS | ||
| 474 | |||
| 475 | CONN -->|disconnect| DISC[Clear PSI + set disconnected_at] | ||
| 476 | DISC -->|any reconnect| HC[handle_connect_or_reconnect] | ||
| 477 | |||
| 478 | subgraph handle_connect_or_reconnect | ||
| 479 | HC --> FRESH_CHECK{is_fresh_sync?} | ||
| 480 | FRESH_CHECK -->|yes: no last_connected OR >15min| L1_FRESH[build_announcement_filter - no since] | ||
| 481 | FRESH_CHECK -->|no: quick reconnect| L1_QUICK[build_announcement_filter - with since] | ||
| 482 | L1_FRESH --> RCA1[recompute_actions_for_relay] | ||
| 483 | L1_QUICK --> L23_QUICK[rebuild_layer2_and_layer3 - with since] | ||
| 484 | L23_QUICK --> RCA2[recompute_actions_for_relay] | ||
| 485 | end | ||
| 486 | ``` | 674 | ``` |
| 487 | 675 | ||
| 488 | --- | 676 | --- |
| 489 | 677 | ||
| 490 | ## Key Design Decisions | 678 | ## Key Design Decisions |
| 491 | 679 | ||
| 492 | | Decision | Choice | Rationale | | 680 | | Decision | Choice | Rationale | |
| 493 | | -------------------------- | -------------------------------------- | --------------------------------------------------------------------------- | | 681 | | ----------------------------- | ------------------------------------------- | ------------------------------------------------------------------ | |
| 494 | | Startup mechanism | Self-subscription only | Single code path, fresh DB behaves same as reconnect | | 682 | | Live vs Historic separation | Two distinct primitives | Clear responsibilities, easier reasoning about state | |
| 495 | | Connect/reconnect handling | Unified handle_connect_or_reconnect | Single entry point for both initial and reconnect | | 683 | | Live sync method | `limit: 0` not `since: now` | No clock dependency, deterministic, mirrors filter structure | |
| 496 | | Layer 1 handling | Separate build_announcement_filter | Connection-level: subscribe ONCE on connect, NOT rebuilt in consolidation | | 684 | | Layer 1 handling | On connect, separate from AddFilters | Connection-level concern, not item-level | |
| 497 | | Layer 2+3 handling | Separate rebuild_layer2_and_layer3 | Item-level: managed by compute_actions, consolidated when filter count > 70 | | 685 | | Layer 2+3 handling | Via compute_actions → sync_computed_filters | Item-level, proper pending tracking | |
| 498 | | Filter functions | since as Option parameter | Allows same functions for fresh sync and catch-up | | 686 | | Clear PendingSyncIndex | Always first | Old subscriptions are dead, must clear before any operation | |
| 499 | | Since filter | Only on catch-up paths | Initial/stale gets full history, quick reconnect catches up | | 687 | | fresh_start vs long_reconnect | Same flow, different metrics | Reuse logic, distinguish intentional refresh from failure recovery | |
| 500 | | compute_actions role | ONLY for new Layer 2+3 items | Does NOT handle Layer 1 or catch-up | | 688 | | Consolidation | Live only, no historic | Items already synced, just reducing subscription count | |
| 501 | | Catch-up pending tracking | No PendingBatch | Items already confirmed, don't need re-confirmation | | 689 | | compute_actions role | ONLY decision point for new work | Single place to reason about what needs syncing | |
| 502 | | Consolidation trigger | On filter add, not periodic | Check in handle_add_filters before adding new filters | | 690 | | NIP-77 negentropy | Try first on full sync, fallback | Efficient for large sets, graceful degradation | |
| 503 | | Clear on disconnect | Clear PSI on disconnect | Cleanup at event boundary, simpler than on reconnect | | ||
| 504 | | 15-minute rule | Clear confirmed if disconnected >15min | Matches since filter buffer, prevents stale subscriptions | | ||
| 505 | | Daily timer | Fresh sync (clears state) | Ensures consistency, detects drift | | ||
| 506 | | NIP-77 negentropy | Try first, fallback to REQ | Efficient set reconciliation when supported | | ||
| 507 | |||
| 508 | --- | ||
| 509 | |||
| 510 | ## NIP-77 Negentropy Sync | ||
| 511 | |||
| 512 | The sync system supports NIP-77 negentropy for efficient set reconciliation when syncing with external relays. | ||
| 513 | |||
| 514 | ### What is Negentropy? | ||
| 515 | |||
| 516 | NIP-77 defines the negentropy protocol for efficient event set comparison. Instead of requesting all events matching a filter (REQ+EOSE), negentropy allows relays to compare fingerprints of their event sets and only transfer the differences. | ||
| 517 | |||
| 518 | ### When Negentropy is Used | ||
| 519 | |||
| 520 | Negentropy sync is attempted for: | ||
| 521 | |||
| 522 | - **Initial connect** - Fresh sync without `last_connected` | ||
| 523 | - **Daily sync** - Periodic full refresh (23-25 hour timer) | ||
| 524 | - **Stale reconnect** - Disconnected for more than 15 minutes | ||
| 525 | |||
| 526 | Negentropy is NOT used for: | ||
| 527 | |||
| 528 | - **Quick reconnect** - Less than 15 minutes disconnected (uses REQ with `since`) | ||
| 529 | - **Live subscriptions** - Ongoing event streams always use REQ | ||
| 530 | |||
| 531 | ### Implementation | ||
| 532 | |||
| 533 | The [`RelayConnection`](../../src/sync/relay_connection.rs:71) now includes NIP-77 methods: | ||
| 534 | |||
| 535 | ```rust | ||
| 536 | /// Check if negentropy sync should be attempted | ||
| 537 | pub async fn supports_negentropy(&self) -> bool { | ||
| 538 | // Always returns true - we try negentropy and handle failure gracefully | ||
| 539 | true | ||
| 540 | } | ||
| 541 | |||
| 542 | /// Perform negentropy synchronization for a filter | ||
| 543 | pub async fn negentropy_sync_filter(&self, filter: Filter) | ||
| 544 | -> Result<NegentropySyncResult, String> { | ||
| 545 | // Uses nostr-sdk's client.sync() method | ||
| 546 | } | ||
| 547 | ``` | ||
| 548 | |||
| 549 | ### Sync Flow with Negentropy | ||
| 550 | |||
| 551 | ```mermaid | ||
| 552 | flowchart TB | ||
| 553 | CONNECT[Connect to relay] --> NEG{Try negentropy} | ||
| 554 | NEG --> |success| L1[Layer 1 synced via negentropy] | ||
| 555 | NEG --> |failure| FALLBACK[Fall back to REQ+EOSE] | ||
| 556 | |||
| 557 | L1 --> SINCE[Record timestamp = now] | ||
| 558 | FALLBACK --> EOSE[Wait for EOSE] | ||
| 559 | EOSE --> SINCE | ||
| 560 | |||
| 561 | SINCE --> LIVE[Open live REQ with since=now] | ||
| 562 | ``` | ||
| 563 | |||
| 564 | ### Fallback Behavior | ||
| 565 | |||
| 566 | If negentropy fails (relay doesn't support NIP-77, network error, etc.): | ||
| 567 | |||
| 568 | 1. A warning is logged (once per relay to avoid spam) | ||
| 569 | 2. The sync falls back to traditional REQ+EOSE | ||
| 570 | 3. No error is raised - fallback is automatic | ||
| 571 | |||
| 572 | **Implementation:** [`negentropy_sync_and_process()`](../../src/sync/mod.rs:1549) | ||
| 573 | |||
| 574 | ### Key Design Decisions for Negentropy | ||
| 575 | |||
| 576 | | Decision | Choice | Rationale | | ||
| 577 | | ------------------ | --------------------------- | ------------------------------------------------- | | ||
| 578 | | Detection approach | Try and fallback | More reliable than NIP-11 document detection | | ||
| 579 | | When to use | Fresh/daily/stale sync only | Quick reconnect with `since` is already efficient | | ||
| 580 | | Error handling | Log once, fallback silently | Avoid log spam while maintaining visibility | | ||
| 581 | | Layer application | Layer 1 first | Announcements are highest priority | | ||
| 582 | 691 | ||
| 583 | --- | 692 | --- |
| 584 | 693 | ||
| @@ -586,8 +695,8 @@ If negentropy fails (relay doesn't support NIP-77, network error, etc.): | |||
| 586 | 695 | ||
| 587 | ``` | 696 | ``` |
| 588 | src/sync/ | 697 | src/sync/ |
| 589 | ├── mod.rs # SyncManager, main loop, data structures (RepoSyncNeeds, RelayState, etc.) | 698 | ├── mod.rs # SyncManager, main loop, data structures |
| 590 | ├── algorithms.rs # derive_relay_targets(), compute_actions(), AddFilters | 699 | ├── algorithms.rs # derive_relay_targets(), compute_actions() |
| 591 | ├── filters.rs # build_announcement_filter(), build_layer2_and_layer3_filters() | 700 | ├── filters.rs # build_announcement_filter(), build_layer2_and_layer3_filters() |
| 592 | ├── health.rs # RelayHealthTracker with exponential backoff | 701 | ├── health.rs # RelayHealthTracker with exponential backoff |
| 593 | ├── relay_connection.rs # RelayConnection, RelayEvent handling | 702 | ├── relay_connection.rs # RelayConnection, RelayEvent handling |
| @@ -599,7 +708,7 @@ src/sync/ | |||
| 599 | 708 | ||
| 600 | ## Health Tracking | 709 | ## Health Tracking |
| 601 | 710 | ||
| 602 | The [`RelayHealthTracker`](../../src/sync/health.rs:93) manages connection health with exponential backoff: | 711 | The `RelayHealthTracker` manages connection health with exponential backoff: |
| 603 | 712 | ||
| 604 | - **States**: Healthy, Degraded, Dead | 713 | - **States**: Healthy, Degraded, Dead |
| 605 | - **Backoff**: `base * 2^(failures-1)`, capped at max_backoff | 714 | - **Backoff**: `base * 2^(failures-1)`, capped at max_backoff |
| @@ -610,6 +719,32 @@ Bootstrap relays are never disconnected by the cleanup system, even if empty. | |||
| 610 | 719 | ||
| 611 | --- | 720 | --- |
| 612 | 721 | ||
| 722 | ## Self-Subscriber | ||
| 723 | |||
| 724 | The `SelfSubscriber` monitors our own relay for repository announcements and root events, updating the `RepoSyncIndex`. | ||
| 725 | |||
| 726 | ### Event Kinds Monitored | ||
| 727 | |||
| 728 | - **30617** - Repository Announcements (triggers discovery of repos listing our relay) | ||
| 729 | - **1617** - Patches (root events referencing repos) | ||
| 730 | - **1618** - Issues | ||
| 731 | - **1619** - Replies/Status | ||
| 732 | - **1621** - Pull Requests | ||
| 733 | |||
| 734 | Note: 30618 (Maintainer Lists) is NOT self-subscribed - only synced from remote relays. | ||
| 735 | |||
| 736 | ### Batching Flow | ||
| 737 | |||
| 738 | 1. **Receive events** from own relay subscription | ||
| 739 | 2. **Queue to pending** - announcements get repo ID + relay URLs; root events get repo ref + event ID | ||
| 740 | 3. **Timer fires** (configurable window, default 5 seconds) - does NOT reset on new events | ||
| 741 | 4. **Process batch**: | ||
| 742 | - Update `RepoSyncIndex` with discovered repos and root events | ||
| 743 | - Call `compute_actions()` | ||
| 744 | - Send `AddFilters` actions to SyncManager → `sync_computed_filters()` | ||
| 745 | |||
| 746 | --- | ||
| 747 | |||
| 613 | ## Disconnect Handling | 748 | ## Disconnect Handling |
| 614 | 749 | ||
| 615 | The disconnect checker runs periodically (default: 60 seconds) to clean up empty relays: | 750 | The disconnect checker runs periodically (default: 60 seconds) to clean up empty relays: |