diff options
| author | DanConwayDev <DanConwayDev@protonmail.com> | 2025-12-09 09:28:12 +0000 |
|---|---|---|
| committer | DanConwayDev <DanConwayDev@protonmail.com> | 2025-12-09 09:28:18 +0000 |
| commit | efaad1e2857914b87307cf78903a957a604697a8 (patch) | |
| tree | dadd0285727b324328166d06d86a6e1e6fb935cf /docs/explanation | |
| parent | 91dc5e8d718475a73815892452a58e1dbf56c8d9 (diff) | |
basic sync stub
Diffstat (limited to 'docs/explanation')
| -rw-r--r-- | docs/explanation/grasp-02-proactive-sync-v2.md | 47 |
1 files changed, 31 insertions, 16 deletions
diff --git a/docs/explanation/grasp-02-proactive-sync-v2.md b/docs/explanation/grasp-02-proactive-sync-v2.md index d1bbe14..311e93c 100644 --- a/docs/explanation/grasp-02-proactive-sync-v2.md +++ b/docs/explanation/grasp-02-proactive-sync-v2.md | |||
| @@ -15,14 +15,14 @@ This document presents a simplified redesign of the proactive sync module. The k | |||
| 15 | 15 | ||
| 16 | This design targets the following scale: | 16 | This design targets the following scale: |
| 17 | 17 | ||
| 18 | | Metric | Target | Notes | | 18 | | Metric | Target | Notes | |
| 19 | |--------|--------|-------| | 19 | | ----------------------------- | -------- | ----------------------------------------- | |
| 20 | | **Repositories** | 1,000 | Repos we host/track | | 20 | | **Repositories** | 1,000 | Repos we host/track | |
| 21 | | **Root events per repo** | 50 (avg) | PRs, Issues, Patches per repo | | 21 | | **Root events per repo** | 50 (avg) | PRs, Issues, Patches per repo | |
| 22 | | **Total relays in ecosystem** | 100 | Unique relays across all repos | | 22 | | **Total relays in ecosystem** | 100 | Unique relays across all repos | |
| 23 | | **Relays per repo** | 5 (avg) | Relays listed in each repo's announcement | | 23 | | **Relays per repo** | 5 (avg) | Relays listed in each repo's announcement | |
| 24 | | **Total root events** | ~50,000 | 1,000 repos × 50 events | | 24 | | **Total root events** | ~50,000 | 1,000 repos × 50 events | |
| 25 | | **Sync connections** | ~50-100 | Based on relay overlap | | 25 | | **Sync connections** | ~50-100 | Based on relay overlap | |
| 26 | 26 | ||
| 27 | **Memory Estimate (in-memory HashMaps):** | 27 | **Memory Estimate (in-memory HashMaps):** |
| 28 | 28 | ||
| @@ -95,6 +95,21 @@ flowchart TB | |||
| 95 | AP -->|store| DB | 95 | AP -->|store| DB |
| 96 | ``` | 96 | ``` |
| 97 | 97 | ||
| 98 | ## Module Structure | ||
| 99 | |||
| 100 | The sync module is organized following the pattern used by `src/http/mod.rs` and `src/metrics/mod.rs` where the primary struct lives in `mod.rs`: | ||
| 101 | |||
| 102 | ``` | ||
| 103 | src/sync/ | ||
| 104 | ├── mod.rs # SyncManager + state types (FollowingRepoRootEvents, SyncRelays) | ||
| 105 | ├── self_subscriber.rs # SelfSubscriber struct and batching logic | ||
| 106 | ├── relay_connection.rs # Per-relay WebSocket connection management | ||
| 107 | ├── health.rs # RelayHealthTracker for backoff and dead relay detection | ||
| 108 | └── metrics.rs # SyncMetrics for Prometheus integration | ||
| 109 | ``` | ||
| 110 | |||
| 111 | **Rationale:** The state type aliases (`FollowingRepoRootEvents`, `SyncRelays`) are simple `Arc<RwLock<HashMap<...>>>` wrappers owned by SyncManager. Rather than creating a separate `state.rs` for two type aliases, they are colocated with SyncManager in `mod.rs` to reduce file count while maintaining clarity. | ||
| 112 | |||
| 98 | ## Design Decision: No Jitter | 113 | ## Design Decision: No Jitter |
| 99 | 114 | ||
| 100 | We considered adding jitter to prevent thundering herd scenarios when: | 115 | We considered adding jitter to prevent thundering herd scenarios when: |
| @@ -468,18 +483,18 @@ impl SyncManager { | |||
| 468 | async fn daily_catchup(&mut self, relay_url: &str) { | 483 | async fn daily_catchup(&mut self, relay_url: &str) { |
| 469 | // Close all current subscriptions for this relay | 484 | // Close all current subscriptions for this relay |
| 470 | self.close_all_subscriptions(relay_url).await; | 485 | self.close_all_subscriptions(relay_url).await; |
| 471 | 486 | ||
| 472 | // Rebuild fresh filters from current HashMap state | 487 | // Rebuild fresh filters from current HashMap state |
| 473 | let filters = self.build_three_layer_filters_for_relay(relay_url).await; | 488 | let filters = self.build_three_layer_filters_for_relay(relay_url).await; |
| 474 | 489 | ||
| 475 | // Subscribe WITHOUT since filter to get full historical sync | 490 | // Subscribe WITHOUT since filter to get full historical sync |
| 476 | for filter in filters { | 491 | for filter in filters { |
| 477 | self.subscribe_to_relay(relay_url, filter).await; | 492 | self.subscribe_to_relay(relay_url, filter).await; |
| 478 | } | 493 | } |
| 479 | 494 | ||
| 480 | // After EOSE, switch back to live mode with since filter | 495 | // After EOSE, switch back to live mode with since filter |
| 481 | self.wait_for_eose(relay_url).await; | 496 | self.wait_for_eose(relay_url).await; |
| 482 | 497 | ||
| 483 | // Re-add since filter for ongoing live sync | 498 | // Re-add since filter for ongoing live sync |
| 484 | let since = Timestamp::now() - 900; // 15 minutes ago | 499 | let since = Timestamp::now() - 900; // 15 minutes ago |
| 485 | self.resubscribe_with_since(relay_url, since).await; | 500 | self.resubscribe_with_since(relay_url, since).await; |
| @@ -663,15 +678,15 @@ async fn check_relay_removal(&mut self) { | |||
| 663 | 678 | ||
| 664 | ``` | 679 | ``` |
| 665 | src/sync/ | 680 | src/sync/ |
| 666 | ├── mod.rs # Module exports, constants | 681 | ├── mod.rs # SyncManager + state types (FollowingRepoRootEvents, SyncRelays) |
| 667 | ├── manager.rs # SyncManager - orchestrates sync | 682 | ├── self_subscriber.rs # SelfSubscriber + batching logic |
| 668 | ├── state.rs # FollowingRepoRootEvents + SyncRelays HashMaps | ||
| 669 | ├── self_subscriber.rs # Self-subscriber + batching logic | ||
| 670 | ├── relay_connection.rs # Per-relay WebSocket + filters | 683 | ├── relay_connection.rs # Per-relay WebSocket + filters |
| 671 | ├── health.rs # RelayHealthTracker (reuse from v1) | 684 | ├── health.rs # RelayHealthTracker (reuse from v1) |
| 672 | └── metrics.rs # SyncMetrics (reuse from v1) | 685 | └── metrics.rs # SyncMetrics (reuse from v1) |
| 673 | ``` | 686 | ``` |
| 674 | 687 | ||
| 688 | > **Note:** SyncManager and state type aliases are colocated in `mod.rs` following the pattern of `src/http/mod.rs` (HttpService) and `src/metrics/mod.rs` (Metrics). See the earlier "Module Structure" section for rationale. | ||
| 689 | |||
| 675 | ## Comparison: v1 vs v2 | 690 | ## Comparison: v1 vs v2 |
| 676 | 691 | ||
| 677 | | Aspect | v1 (Current) | v2 (Simplified) | | 692 | | Aspect | v1 (Current) | v2 (Simplified) | |