diff options
| -rw-r--r-- | docs/explanation/grasp-02-proactive-sync.md | 207 |
1 files changed, 130 insertions, 77 deletions
diff --git a/docs/explanation/grasp-02-proactive-sync.md b/docs/explanation/grasp-02-proactive-sync.md index 64fa096..64193d3 100644 --- a/docs/explanation/grasp-02-proactive-sync.md +++ b/docs/explanation/grasp-02-proactive-sync.md | |||
| @@ -4,9 +4,9 @@ | |||
| 4 | 4 | ||
| 5 | This document explains the proactive sync system that synchronizes repository data from external relays based on relay URLs listed in 30617 repository announcements. Key principles: | 5 | This document explains the proactive sync system that synchronizes repository data from external relays based on relay URLs listed in 30617 repository announcements. Key principles: |
| 6 | 6 | ||
| 7 | 1. **Triggers call compute_actions → sync_computed_filters** - Self-subscriber batches and connect/reconnect events trigger this flow | 7 | 1. **Two paths to AddFilters → handle_new_sync_filters** - Self-subscriber sends directly via channel; connect/reconnect uses `recompute_new_sync_filters_for_relay` |
| 8 | 2. **Clear separation of live vs historic sync** - Two distinct primitives with different purposes | 8 | 2. **Clear separation of live vs historic sync** - Two distinct primitives with different purposes |
| 9 | 3. **Layer 1 on connect, Layer 2+3 via AddFilters** - L1 handled at connection time, L2+L3 flow through compute_actions | 9 | 3. **Layer 1 on connect, Layer 2+3 via AddFilters** - L1 handled at connection time, L2+L3 flow through AddFilters |
| 10 | 4. **Always clear PendingSyncIndex first** - Before any reconnect/consolidate operation | 10 | 4. **Always clear PendingSyncIndex first** - Before any reconnect/consolidate operation |
| 11 | 5. **NIP-77 negentropy for historical sync** - Efficient set reconciliation, fallback to REQ if unsupported | 11 | 5. **NIP-77 negentropy for historical sync** - Efficient set reconciliation, fallback to REQ if unsupported |
| 12 | 12 | ||
| @@ -238,43 +238,90 @@ The sync system is built on two fundamental primitives that are clearly separate | |||
| 238 | 238 | ||
| 239 | ### Layer Strategy | 239 | ### Layer Strategy |
| 240 | 240 | ||
| 241 | | Layer | Content | When Subscribed | Managed By | | 241 | | Layer | Content | When Subscribed | Managed By | |
| 242 | | ------- | --------------------------------------- | --------------------- | -------------------- | | 242 | | ------- | --------------------------------------- | --------------------- | ----------------------- | |
| 243 | | Layer 1 | 30617 Announcements, 30618 Maintainers | On connect (any type) | Connection lifecycle | | 243 | | Layer 1 | 30617 Announcements, 30618 Maintainers | On connect (any type) | Connection lifecycle | |
| 244 | | Layer 2 | Events tagging our repos (a/A/q tags) | Via AddFilters | compute_actions | | 244 | | Layer 2 | Events tagging our repos (a/A/q tags) | Via AddFilters | handle_new_sync_filters | |
| 245 | | Layer 3 | Events tagging root events (e/E/q tags) | Via AddFilters | compute_actions | | 245 | | Layer 3 | Events tagging root events (e/E/q tags) | Via AddFilters | handle_new_sync_filters | |
| 246 | 246 | ||
| 247 | **Key insight**: Layer 1 is connection-level (handled at connect time), Layer 2+3 are item-level (flow through compute_actions → sync_computed_filters). | 247 | **Key insight**: Layer 1 is connection-level (handled at connect time), Layer 2+3 are item-level (flow through AddFilters → handle_new_sync_filters via two paths). |
| 248 | 248 | ||
| 249 | --- | 249 | --- |
| 250 | 250 | ||
| 251 | ## Triggers and Flow | 251 | ## Triggers and Flow |
| 252 | 252 | ||
| 253 | ### What Triggers compute_actions → sync_computed_filters? | 253 | ### Two Paths to AddFilters |
| 254 | 254 | ||
| 255 | | Trigger | When | What Happens | | 255 | The system has **two independent paths** that create and process AddFilters actions: |
| 256 | | --------------------------- | -------------------------------------- | ---------------------------------------- | | ||
| 257 | | Self-subscriber batch fires | New events discovered on own relay | Update RepoSyncIndex → compute_actions | | ||
| 258 | | fresh_start() | Initial connect, long_reconnect, daily | After L1 setup → compute_actions | | ||
| 259 | | quick_reconnect() | Reconnect < 15 minutes | After L1+L2+L3 catchup → compute_actions | | ||
| 260 | | consolidate() | Filter count > threshold | After live rebuild → compute_actions | | ||
| 261 | 256 | ||
| 262 | ### The Core Flow | 257 | | Source | When | Flow | |
| 258 | | -------------------------- | ----------------------------------- | -------------------------------------------------------------------------------- | | ||
| 259 | | Self-subscriber batch | New events discovered on own relay | Build AddFilters directly → send via channel → handle_new_sync_filters | | ||
| 260 | | Connect/reconnect triggers | fresh_start, quick_reconnect, daily | recompute_new_sync_filters_for_relay → compute_actions → handle_new_sync_filters | | ||
| 261 | |||
| 262 | **Path 1: Self-Subscriber (direct AddFilters construction)** | ||
| 263 | |||
| 264 | The [`SelfSubscriber::process_batch()`](src/sync/self_subscriber.rs:452) method: | ||
| 265 | |||
| 266 | 1. Updates `RepoSyncIndex` with discovered repos | ||
| 267 | 2. Calls `derive_relay_targets()` to get per-relay targets | ||
| 268 | 3. Builds `AddFilters` directly using `build_layer2_and_layer3_filters()` | ||
| 269 | 4. Sends via `action_tx` channel to SyncManager | ||
| 270 | 5. SyncManager receives via `action_rx` and calls `handle_new_sync_filters()` | ||
| 271 | |||
| 272 | **Path 2: Connect/Reconnect (via compute_actions)** | ||
| 273 | |||
| 274 | The [`SyncManager::recompute_new_sync_filters_for_relay()`](src/sync/mod.rs:1374) method: | ||
| 275 | |||
| 276 | 1. Calls `derive_relay_targets()` from `RepoSyncIndex` | ||
| 277 | 2. Calls `compute_actions(targets, pending, confirmed)` - three-way diff | ||
| 278 | 3. Calls `handle_new_sync_filters()` for each resulting AddFilters action | ||
| 279 | |||
| 280 | ### When Each Path is Used | ||
| 281 | |||
| 282 | | Trigger | Path Used | Why | | ||
| 283 | | --------------------------- | --------------------------- | -------------------------------------------- | | ||
| 284 | | Self-subscriber batch fires | Direct (no compute_actions) | Building from scratch, no diff needed | | ||
| 285 | | fresh_start() | compute_actions | Diff against pending/confirmed state | | ||
| 286 | | quick_reconnect() | compute_actions | Check for NEW items discovered while offline | | ||
| 287 | | consolidate() | compute_actions | Check for new items during filter rebuild | | ||
| 288 | |||
| 289 | ### The Core Flow (Path 2: Connect/Reconnect) | ||
| 263 | 290 | ||
| 264 | ```mermaid | 291 | ```mermaid |
| 265 | flowchart TB | 292 | flowchart TB |
| 266 | TRIGGER[Trigger fires] --> CA[compute_actions] | 293 | TRIGGER[Connect/Reconnect trigger] --> RECOMPUTE[recompute_new_sync_filters_for_relay] |
| 267 | CA --> |derives from| RSI[RepoSyncIndex] | 294 | RECOMPUTE --> DRT[derive_relay_targets] |
| 268 | CA --> |subtracts| RLI[RelaySyncIndex] | 295 | DRT --> |derives from| RSI[RepoSyncIndex] |
| 296 | DRT --> CA[compute_actions] | ||
| 269 | CA --> |subtracts| PSI[PendingSyncIndex] | 297 | CA --> |subtracts| PSI[PendingSyncIndex] |
| 298 | CA --> |subtracts| RLI[RelaySyncIndex] | ||
| 270 | CA --> |produces| AF[AddFilters actions] | 299 | CA --> |produces| AF[AddFilters actions] |
| 271 | AF --> SFRE[sync_computed_filters] | 300 | AF --> HNSF[handle_new_sync_filters] |
| 272 | SFRE --> LIVE[sync_live - L2+L3] | 301 | HNSF --> LIVE[sync_live - L2+L3] |
| 273 | SFRE --> HIST[historic_sync - L2+L3] | 302 | HNSF --> HIST[historic_sync - L2+L3] |
| 274 | HIST --> PSI_UPDATE[Update PendingSyncIndex] | 303 | HIST --> PSI_UPDATE[Update PendingSyncIndex] |
| 275 | PSI_UPDATE --> |EOSE received| CONFIRM[Move to RelaySyncIndex] | 304 | PSI_UPDATE --> |EOSE received| CONFIRM[Move to RelaySyncIndex] |
| 276 | ``` | 305 | ``` |
| 277 | 306 | ||
| 307 | ### The Self-Subscriber Flow (Path 1: Direct) | ||
| 308 | |||
| 309 | ```mermaid | ||
| 310 | flowchart TB | ||
| 311 | EVENTS[Events from own relay] --> QUEUE[Queue to PendingUpdates] | ||
| 312 | QUEUE --> TIMER[Batch timer fires - 5 seconds] | ||
| 313 | TIMER --> PB[process_batch] | ||
| 314 | PB --> UPDATE[Update RepoSyncIndex] | ||
| 315 | UPDATE --> DRT[derive_relay_targets] | ||
| 316 | DRT --> BUILD[build_layer2_and_layer3_filters] | ||
| 317 | BUILD --> AF[Create AddFilters] | ||
| 318 | AF --> CHAN[Send via action_tx channel] | ||
| 319 | CHAN --> RX[SyncManager receives via action_rx] | ||
| 320 | RX --> HNSF[handle_new_sync_filters] | ||
| 321 | HNSF --> LIVE[sync_live - L2+L3] | ||
| 322 | HNSF --> HIST[historic_sync - L2+L3] | ||
| 323 | ``` | ||
| 324 | |||
| 278 | --- | 325 | --- |
| 279 | 326 | ||
| 280 | ## Flow Scenarios | 327 | ## Flow Scenarios |
| @@ -302,12 +349,13 @@ flowchart TB | |||
| 302 | L1_HIST --> NEG{NIP-77 supported?} | 349 | L1_HIST --> NEG{NIP-77 supported?} |
| 303 | NEG --> |yes| NEGENTROPY[negentropy sync] | 350 | NEG --> |yes| NEGENTROPY[negentropy sync] |
| 304 | NEG --> |no| REQ[REQ+EOSE] | 351 | NEG --> |no| REQ[REQ+EOSE] |
| 305 | NEGENTROPY --> CA[compute_actions] | 352 | NEGENTROPY --> RECOMPUTE[recompute_new_sync_filters_for_relay] |
| 306 | REQ --> CA | 353 | REQ --> RECOMPUTE |
| 354 | RECOMPUTE --> CA[compute_actions] | ||
| 307 | CA --> |empty RelaySyncIndex| AF[AddFilters for ALL repos] | 355 | CA --> |empty RelaySyncIndex| AF[AddFilters for ALL repos] |
| 308 | AF --> SFRE[sync_computed_filters] | 356 | AF --> HNSF[handle_new_sync_filters] |
| 309 | SFRE --> L23_LIVE[L2+L3: sync_live] | 357 | HNSF --> L23_LIVE[L2+L3: sync_live] |
| 310 | SFRE --> L23_HIST[L2+L3: historic_sync] | 358 | HNSF --> L23_HIST[L2+L3: historic_sync] |
| 311 | L23_HIST --> PB[Create PendingBatch] | 359 | L23_HIST --> PB[Create PendingBatch] |
| 312 | PB --> EOSE[Wait for EOSE] | 360 | PB --> EOSE[Wait for EOSE] |
| 313 | EOSE --> CONFIRM[Move items to RelaySyncIndex] | 361 | EOSE --> CONFIRM[Move items to RelaySyncIndex] |
| @@ -318,7 +366,7 @@ flowchart TB | |||
| 318 | - Always clear PendingSyncIndex first, then RelaySyncIndex | 366 | - Always clear PendingSyncIndex first, then RelaySyncIndex |
| 319 | - L1 live + L1 historic (uses negentropy if available) | 367 | - L1 live + L1 historic (uses negentropy if available) |
| 320 | - Empty RelaySyncIndex means diff produces AddFilters for everything | 368 | - Empty RelaySyncIndex means diff produces AddFilters for everything |
| 321 | - L2+L3 flow through sync_computed_filters with proper pending tracking | 369 | - L2+L3 flow through `recompute_new_sync_filters_for_relay` → `handle_new_sync_filters` with proper pending tracking |
| 322 | 370 | ||
| 323 | ### Scenario 2: Quick Reconnect (< 15 minutes) | 371 | ### Scenario 2: Quick Reconnect (< 15 minutes) |
| 324 | 372 | ||
| @@ -348,11 +396,12 @@ flowchart TB | |||
| 348 | L1_HIST --> RECON[reconstruct_filters from RelaySyncIndex] | 396 | L1_HIST --> RECON[reconstruct_filters from RelaySyncIndex] |
| 349 | RECON --> L23_LIVE[L2+L3: sync_live] | 397 | RECON --> L23_LIVE[L2+L3: sync_live] |
| 350 | RECON --> L23_HIST[L2+L3: historic_sync WITH since] | 398 | RECON --> L23_HIST[L2+L3: historic_sync WITH since] |
| 351 | L23_HIST --> CA[compute_actions] | 399 | L23_HIST --> RECOMPUTE[recompute_new_sync_filters_for_relay] |
| 400 | RECOMPUTE --> CA[compute_actions] | ||
| 352 | CA --> |check for new items| AF{New items?} | 401 | CA --> |check for new items| AF{New items?} |
| 353 | AF --> |yes| SFRE[sync_computed_filters] | 402 | AF --> |yes| HNSF[handle_new_sync_filters] |
| 354 | AF --> |no| DONE[Done] | 403 | AF --> |no| DONE[Done] |
| 355 | SFRE --> PB[Create PendingBatch] | 404 | HNSF --> PB[Create PendingBatch] |
| 356 | ``` | 405 | ``` |
| 357 | 406 | ||
| 358 | **Key points:** | 407 | **Key points:** |
| @@ -361,7 +410,7 @@ flowchart TB | |||
| 361 | - L1 live (always on any connection) | 410 | - L1 live (always on any connection) |
| 362 | - L1 historic WITH since (catches up missed announcements) | 411 | - L1 historic WITH since (catches up missed announcements) |
| 363 | - L2+L3 rebuilt from RelaySyncIndex (confirmed state preserved) | 412 | - L2+L3 rebuilt from RelaySyncIndex (confirmed state preserved) |
| 364 | - compute_actions checks for any NEW items discovered during catchup | 413 | - `recompute_new_sync_filters_for_relay` → `compute_actions` checks for any NEW items discovered during catchup |
| 365 | 414 | ||
| 366 | ### Scenario 3: Long Reconnect (> 15 minutes) | 415 | ### Scenario 3: Long Reconnect (> 15 minutes) |
| 367 | 416 | ||
| @@ -388,9 +437,10 @@ flowchart TB | |||
| 388 | UNSUB --> RECON[reconstruct_filters from RelaySyncIndex] | 437 | UNSUB --> RECON[reconstruct_filters from RelaySyncIndex] |
| 389 | RECON --> L1_LIVE[L1: sync_live] | 438 | RECON --> L1_LIVE[L1: sync_live] |
| 390 | RECON --> L23_LIVE[L2+L3: sync_live] | 439 | RECON --> L23_LIVE[L2+L3: sync_live] |
| 391 | L23_LIVE --> CA[compute_actions] | 440 | L23_LIVE --> RECOMPUTE[recompute_new_sync_filters_for_relay] |
| 441 | RECOMPUTE --> CA[compute_actions] | ||
| 392 | CA --> |check for new items| AF{New items?} | 442 | CA --> |check for new items| AF{New items?} |
| 393 | AF --> |yes| SFRE[sync_computed_filters] | 443 | AF --> |yes| HNSF[handle_new_sync_filters] |
| 394 | AF --> |no| DONE[Done] | 444 | AF --> |no| DONE[Done] |
| 395 | THRESHOLD --> |no| SKIP[Continue normally] | 445 | THRESHOLD --> |no| SKIP[Continue normally] |
| 396 | ``` | 446 | ``` |
| @@ -400,7 +450,7 @@ flowchart TB | |||
| 400 | - Clear PendingSyncIndex first | 450 | - Clear PendingSyncIndex first |
| 401 | - NO historic sync needed - items already synced/syncing | 451 | - NO historic sync needed - items already synced/syncing |
| 402 | - Only rebuilds live subscriptions from confirmed state | 452 | - Only rebuilds live subscriptions from confirmed state |
| 403 | - compute_actions catches any new items that need syncing | 453 | - `recompute_new_sync_filters_for_relay` → `compute_actions` catches any new items that need syncing |
| 404 | 454 | ||
| 405 | ### Scenario 5: Daily Sync (23-25h Random Timer) | 455 | ### Scenario 5: Daily Sync (23-25h Random Timer) |
| 406 | 456 | ||
| @@ -419,22 +469,27 @@ flowchart TB | |||
| 419 | 469 | ||
| 420 | ```mermaid | 470 | ```mermaid |
| 421 | flowchart TB | 471 | flowchart TB |
| 422 | EVENTS[Events from own relay] --> QUEUE[Queue to pending batch] | 472 | EVENTS[Events from own relay] --> QUEUE[Queue to PendingUpdates] |
| 423 | QUEUE --> TIMER[Batch timer fires - 5 seconds] | 473 | QUEUE --> TIMER[Batch timer fires - 5 seconds] |
| 424 | TIMER --> UPDATE[Update RepoSyncIndex] | 474 | TIMER --> PB[process_batch] |
| 425 | UPDATE --> CA[compute_actions] | 475 | PB --> UPDATE[Update RepoSyncIndex] |
| 426 | CA --> |new repos/events discovered| AF[AddFilters] | 476 | UPDATE --> DRT[derive_relay_targets] |
| 427 | AF --> SFRE[sync_computed_filters] | 477 | DRT --> BUILD[build_layer2_and_layer3_filters] |
| 428 | SFRE --> LIVE[sync_live - L2+L3] | 478 | BUILD --> AF[Create AddFilters directly] |
| 429 | SFRE --> HIST[historic_sync - L2+L3] | 479 | AF --> CHAN[Send via action_tx channel] |
| 480 | CHAN --> RX[SyncManager receives] | ||
| 481 | RX --> HNSF[handle_new_sync_filters] | ||
| 482 | HNSF --> LIVE[sync_live - L2+L3] | ||
| 483 | HNSF --> HIST[historic_sync - L2+L3] | ||
| 430 | ``` | 484 | ``` |
| 431 | 485 | ||
| 432 | **Key points:** | 486 | **Key points:** |
| 433 | 487 | ||
| 434 | - Self-subscriber monitors own relay for 30617, 1617, 1618, 1619, 1621 | 488 | - Self-subscriber monitors own relay for 30617, 1617, 1618, 1621 (NOT 1619 or 30618) |
| 435 | - Batches events (5 second window) | 489 | - Batches events in `PendingUpdates` (5 second window via interval timer) |
| 436 | - Updates RepoSyncIndex, then compute_actions finds new work | 490 | - `process_batch()` updates RepoSyncIndex, then builds AddFilters **directly** (no compute_actions) |
| 437 | - New items flow through sync_computed_filters | 491 | - AddFilters sent via channel to SyncManager, which calls `handle_new_sync_filters()` |
| 492 | - This path does NOT use compute_actions because it's building fresh filters from the updated index | ||
| 438 | 493 | ||
| 439 | --- | 494 | --- |
| 440 | 495 | ||
| @@ -508,21 +563,19 @@ Dispatches to: | |||
| 508 | 563 | ||
| 509 | ### Building Blocks | 564 | ### Building Blocks |
| 510 | 565 | ||
| 511 | #### `sync_computed_filters()` - Handle New AddFilters | 566 | #### `handle_new_sync_filters()` - Handle New AddFilters |
| 512 | 567 | ||
| 513 | ```rust | 568 | ```rust |
| 514 | /// Process AddFilters action (from compute_actions) | 569 | /// Handle AddFilters action (from self-subscriber channel OR compute_actions) |
| 515 | /// | 570 | /// |
| 516 | /// Orchestrates both live and historic sync for NEW items: | 571 | /// Orchestrates both live and historic sync for NEW items: |
| 517 | /// 1. sync_live() - set up permanent L2+L3 subscriptions | 572 | /// 1. Check/spawn connection if needed (for unknown relays) |
| 518 | /// 2. historic_sync() - catch up on past events | 573 | /// 2. maybe_consolidate() - check filter threshold |
| 574 | /// 3. sync_live() - set up permanent L2+L3 subscriptions | ||
| 575 | /// 4. historic_sync() - catch up on past events | ||
| 519 | /// | 576 | /// |
| 520 | /// This is specifically for NEW filter discovery. | 577 | /// This is the SINGLE entry point for processing AddFilters from BOTH paths. |
| 521 | async fn sync_computed_filters( | 578 | async fn handle_new_sync_filters(&mut self, action: AddFilters) |
| 522 | &mut self, | ||
| 523 | action: AddFilters, | ||
| 524 | since: Option<Timestamp>, | ||
| 525 | ) -> Option<u64> | ||
| 526 | ``` | 579 | ``` |
| 527 | 580 | ||
| 528 | ### Top-Level Entry Points | 581 | ### Top-Level Entry Points |
| @@ -538,7 +591,7 @@ async fn sync_computed_filters( | |||
| 538 | /// 1. Clear PendingSyncIndex | 591 | /// 1. Clear PendingSyncIndex |
| 539 | /// 2. Clear RelaySyncIndex | 592 | /// 2. Clear RelaySyncIndex |
| 540 | /// 3. L1 live + L1 historic (negentropy if available) | 593 | /// 3. L1 live + L1 historic (negentropy if available) |
| 541 | /// 4. compute_actions → AddFilters → sync_computed_filters for L2+L3 | 594 | /// 4. recompute_new_sync_filters_for_relay → compute_actions → handle_new_sync_filters for L2+L3 |
| 542 | async fn fresh_start(&mut self, relay_url: &str) | 595 | async fn fresh_start(&mut self, relay_url: &str) |
| 543 | ``` | 596 | ``` |
| 544 | 597 | ||
| @@ -598,7 +651,7 @@ async fn consolidate(&mut self, relay_url: &str) | |||
| 598 | /// Flow: | 651 | /// Flow: |
| 599 | /// 1. Check/spawn connection if needed | 652 | /// 1. Check/spawn connection if needed |
| 600 | /// 2. maybe_consolidate (check filter threshold) | 653 | /// 2. maybe_consolidate (check filter threshold) |
| 601 | /// 3. sync_computed_filters | 654 | /// 3. recompute_new_sync_filters_for_relay |
| 602 | async fn handle_new_sync_filters(&mut self, action: AddFilters) | 655 | async fn handle_new_sync_filters(&mut self, action: AddFilters) |
| 603 | ``` | 656 | ``` |
| 604 | 657 | ||
| @@ -612,7 +665,7 @@ fresh_start(relay_url) // Initial/long_reconnect/daily | |||
| 612 | ├──> Clear RelaySyncIndex | 665 | ├──> Clear RelaySyncIndex |
| 613 | ├──> L1: sync_live(announcement_filter) | 666 | ├──> L1: sync_live(announcement_filter) |
| 614 | ├──> L1: historic_sync(announcement_filter, None) | 667 | ├──> L1: historic_sync(announcement_filter, None) |
| 615 | └──> compute_actions → AddFilters → sync_computed_filters (L2+L3) | 668 | └──> compute_actions → AddFilters → recompute_new_sync_filters_for_relay (L2+L3) |
| 616 | 669 | ||
| 617 | quick_reconnect(relay_url, since) // Disconnected < 15 min | 670 | quick_reconnect(relay_url, since) // Disconnected < 15 min |
| 618 | ├──> Clear PendingSyncIndex | 671 | ├──> Clear PendingSyncIndex |
| @@ -621,7 +674,7 @@ quick_reconnect(relay_url, since) // Disconnected < 15 min | |||
| 621 | ├──> reconstruct_filters() → L2+L3 filters | 674 | ├──> reconstruct_filters() → L2+L3 filters |
| 622 | ├──> L2+L3: sync_live(filters) | 675 | ├──> L2+L3: sync_live(filters) |
| 623 | ├──> L2+L3: historic_sync(filters, since) | 676 | ├──> L2+L3: historic_sync(filters, since) |
| 624 | └──> compute_actions → AddFilters → sync_computed_filters (new items only) | 677 | └──> compute_actions → AddFilters → recompute_new_sync_filters_for_relay (new items only) |
| 625 | 678 | ||
| 626 | long_reconnect(relay_url) // Disconnected > 15 min | 679 | long_reconnect(relay_url) // Disconnected > 15 min |
| 627 | ├──> Record disconnect/reconnect metric | 680 | ├──> Record disconnect/reconnect metric |
| @@ -635,14 +688,14 @@ consolidate(relay_url) // Filter count > threshold | |||
| 635 | ├──> unsubscribe_all() | 688 | ├──> unsubscribe_all() |
| 636 | ├──> reconstruct_filters() → L1+L2+L3 filters | 689 | ├──> reconstruct_filters() → L1+L2+L3 filters |
| 637 | ├──> sync_live(filters) // Live only, NO historic | 690 | ├──> sync_live(filters) // Live only, NO historic |
| 638 | └──> compute_actions → AddFilters → sync_computed_filters (new items only) | 691 | └──> compute_actions → AddFilters → recompute_new_sync_filters_for_relay (new items only) |
| 639 | 692 | ||
| 640 | handle_new_sync_filters(action) // New filter discovery | 693 | handle_new_sync_filters(action) // New filter discovery |
| 641 | ├──> Check/spawn connection | 694 | ├──> Check/spawn connection |
| 642 | ├──> maybe_consolidate() | 695 | ├──> maybe_consolidate() |
| 643 | └──> sync_computed_filters(action, None) | 696 | └──> recompute_new_sync_filters_for_relay(action, None) |
| 644 | 697 | ||
| 645 | sync_computed_filters(action, since) // Process AddFilters | 698 | recompute_new_sync_filters_for_relay(action, since) // Process AddFilters |
| 646 | ├──> sync_live(action.filters) // L2+L3 live | 699 | ├──> sync_live(action.filters) // L2+L3 live |
| 647 | └──> historic_sync(action.filters, since) // L2+L3 historic | 700 | └──> historic_sync(action.filters, since) // L2+L3 historic |
| 648 | ├── historic_sync_negentropy() // Parallel, updates Pending | 701 | ├── historic_sync_negentropy() // Parallel, updates Pending |
| @@ -676,7 +729,7 @@ sync_computed_filters(action, since) // Process AddFilters | |||
| 676 | 729 | ||
| 677 | The `build_layer2_and_layer3_filters()` function combines both layers. Used by: | 730 | The `build_layer2_and_layer3_filters()` function combines both layers. Used by: |
| 678 | 731 | ||
| 679 | - `sync_computed_filters` for new item subscriptions | 732 | - `recompute_new_sync_filters_for_relay` for new item subscriptions |
| 680 | - `reconstruct_filters` for rebuilding from confirmed state | 733 | - `reconstruct_filters` for rebuilding from confirmed state |
| 681 | 734 | ||
| 682 | --- | 735 | --- |
| @@ -802,7 +855,7 @@ flowchart TB | |||
| 802 | PSI --> CA | 855 | PSI --> CA |
| 803 | RLI --> CA | 856 | RLI --> CA |
| 804 | CA -->|new items| AF[AddFilters] | 857 | CA -->|new items| AF[AddFilters] |
| 805 | AF --> SFRE[sync_computed_filters] | 858 | AF --> SFRE[recompute_new_sync_filters_for_relay] |
| 806 | SFRE --> LIVE[sync_live L2+L3] | 859 | SFRE --> LIVE[sync_live L2+L3] |
| 807 | SFRE --> HIST[historic_sync L2+L3] | 860 | SFRE --> HIST[historic_sync L2+L3] |
| 808 | HIST --> PSI | 861 | HIST --> PSI |
| @@ -813,17 +866,17 @@ flowchart TB | |||
| 813 | 866 | ||
| 814 | ## Key Design Decisions | 867 | ## Key Design Decisions |
| 815 | 868 | ||
| 816 | | Decision | Choice | Rationale | | 869 | | Decision | Choice | Rationale | |
| 817 | | ----------------------------- | ------------------------------------------- | ------------------------------------------------------------------ | | 870 | | ----------------------------- | ---------------------------------------------------------- | ------------------------------------------------------------------ | |
| 818 | | Live vs Historic separation | Two distinct primitives | Clear responsibilities, easier reasoning about state | | 871 | | Live vs Historic separation | Two distinct primitives | Clear responsibilities, easier reasoning about state | |
| 819 | | Live sync method | `limit: 0` not `since: now` | No clock dependency, deterministic, mirrors filter structure | | 872 | | Live sync method | `limit: 0` not `since: now` | No clock dependency, deterministic, mirrors filter structure | |
| 820 | | Layer 1 handling | On connect, separate from AddFilters | Connection-level concern, not item-level | | 873 | | Layer 1 handling | On connect, separate from AddFilters | Connection-level concern, not item-level | |
| 821 | | Layer 2+3 handling | Via compute_actions → sync_computed_filters | Item-level, proper pending tracking | | 874 | | Layer 2+3 handling | Via compute_actions → recompute_new_sync_filters_for_relay | Item-level, proper pending tracking | |
| 822 | | Clear PendingSyncIndex | Always first | Old subscriptions are dead, must clear before any operation | | 875 | | Clear PendingSyncIndex | Always first | Old subscriptions are dead, must clear before any operation | |
| 823 | | fresh_start vs long_reconnect | Same flow, different metrics | Reuse logic, distinguish intentional refresh from failure recovery | | 876 | | fresh_start vs long_reconnect | Same flow, different metrics | Reuse logic, distinguish intentional refresh from failure recovery | |
| 824 | | Consolidation | Live only, no historic | Items already synced, just reducing subscription count | | 877 | | Consolidation | Live only, no historic | Items already synced, just reducing subscription count | |
| 825 | | compute_actions role | ONLY decision point for new work | Single place to reason about what needs syncing | | 878 | | compute_actions role | ONLY decision point for new work | Single place to reason about what needs syncing | |
| 826 | | NIP-77 negentropy | Try first on full sync, fallback | Efficient for large sets, graceful degradation | | 879 | | NIP-77 negentropy | Try first on full sync, fallback | Efficient for large sets, graceful degradation | |
| 827 | 880 | ||
| 828 | --- | 881 | --- |
| 829 | 882 | ||
| @@ -877,7 +930,7 @@ Note: 30618 (Maintainer Lists) is NOT self-subscribed - only synced from remote | |||
| 877 | 4. **Process batch**: | 930 | 4. **Process batch**: |
| 878 | - Update `RepoSyncIndex` with discovered repos and root events | 931 | - Update `RepoSyncIndex` with discovered repos and root events |
| 879 | - Call `compute_actions()` | 932 | - Call `compute_actions()` |
| 880 | - Send `AddFilters` actions to SyncManager → `sync_computed_filters()` | 933 | - Send `AddFilters` actions to SyncManager → `recompute_new_sync_filters_for_relay()` |
| 881 | 934 | ||
| 882 | --- | 935 | --- |
| 883 | 936 | ||