upleb.uk

Public git repos — served from a NIP-34 GRASP relay at git.upleb.uk

summaryrefslogtreecommitdiff
path: root/docs/explanation
diff options
context:
space:
mode:
authorDanConwayDev <DanConwayDev@protonmail.com>2025-12-09 09:28:12 +0000
committerDanConwayDev <DanConwayDev@protonmail.com>2025-12-09 09:28:18 +0000
commitefaad1e2857914b87307cf78903a957a604697a8 (patch)
treedadd0285727b324328166d06d86a6e1e6fb935cf /docs/explanation
parent91dc5e8d718475a73815892452a58e1dbf56c8d9 (diff)
basic sync stub
Diffstat (limited to 'docs/explanation')
-rw-r--r--docs/explanation/grasp-02-proactive-sync-v2.md47
1 files changed, 31 insertions, 16 deletions
diff --git a/docs/explanation/grasp-02-proactive-sync-v2.md b/docs/explanation/grasp-02-proactive-sync-v2.md
index d1bbe14..311e93c 100644
--- a/docs/explanation/grasp-02-proactive-sync-v2.md
+++ b/docs/explanation/grasp-02-proactive-sync-v2.md
@@ -15,14 +15,14 @@ This document presents a simplified redesign of the proactive sync module. The k
15 15
16This design targets the following scale: 16This design targets the following scale:
17 17
18| Metric | Target | Notes | 18| Metric | Target | Notes |
19|--------|--------|-------| 19| ----------------------------- | -------- | ----------------------------------------- |
20| **Repositories** | 1,000 | Repos we host/track | 20| **Repositories** | 1,000 | Repos we host/track |
21| **Root events per repo** | 50 (avg) | PRs, Issues, Patches per repo | 21| **Root events per repo** | 50 (avg) | PRs, Issues, Patches per repo |
22| **Total relays in ecosystem** | 100 | Unique relays across all repos | 22| **Total relays in ecosystem** | 100 | Unique relays across all repos |
23| **Relays per repo** | 5 (avg) | Relays listed in each repo's announcement | 23| **Relays per repo** | 5 (avg) | Relays listed in each repo's announcement |
24| **Total root events** | ~50,000 | 1,000 repos × 50 events | 24| **Total root events** | ~50,000 | 1,000 repos × 50 events |
25| **Sync connections** | ~50-100 | Based on relay overlap | 25| **Sync connections** | ~50-100 | Based on relay overlap |
26 26
27**Memory Estimate (in-memory HashMaps):** 27**Memory Estimate (in-memory HashMaps):**
28 28
@@ -95,6 +95,21 @@ flowchart TB
95 AP -->|store| DB 95 AP -->|store| DB
96``` 96```
97 97
98## Module Structure
99
100The sync module is organized following the pattern used by `src/http/mod.rs` and `src/metrics/mod.rs` where the primary struct lives in `mod.rs`:
101
102```
103src/sync/
104├── mod.rs # SyncManager + state types (FollowingRepoRootEvents, SyncRelays)
105├── self_subscriber.rs # SelfSubscriber struct and batching logic
106├── relay_connection.rs # Per-relay WebSocket connection management
107├── health.rs # RelayHealthTracker for backoff and dead relay detection
108└── metrics.rs # SyncMetrics for Prometheus integration
109```
110
111**Rationale:** The state type aliases (`FollowingRepoRootEvents`, `SyncRelays`) are simple `Arc<RwLock<HashMap<...>>>` wrappers owned by SyncManager. Rather than creating a separate `state.rs` for two type aliases, they are colocated with SyncManager in `mod.rs` to reduce file count while maintaining clarity.
112
98## Design Decision: No Jitter 113## Design Decision: No Jitter
99 114
100We considered adding jitter to prevent thundering herd scenarios when: 115We considered adding jitter to prevent thundering herd scenarios when:
@@ -468,18 +483,18 @@ impl SyncManager {
468 async fn daily_catchup(&mut self, relay_url: &str) { 483 async fn daily_catchup(&mut self, relay_url: &str) {
469 // Close all current subscriptions for this relay 484 // Close all current subscriptions for this relay
470 self.close_all_subscriptions(relay_url).await; 485 self.close_all_subscriptions(relay_url).await;
471 486
472 // Rebuild fresh filters from current HashMap state 487 // Rebuild fresh filters from current HashMap state
473 let filters = self.build_three_layer_filters_for_relay(relay_url).await; 488 let filters = self.build_three_layer_filters_for_relay(relay_url).await;
474 489
475 // Subscribe WITHOUT since filter to get full historical sync 490 // Subscribe WITHOUT since filter to get full historical sync
476 for filter in filters { 491 for filter in filters {
477 self.subscribe_to_relay(relay_url, filter).await; 492 self.subscribe_to_relay(relay_url, filter).await;
478 } 493 }
479 494
480 // After EOSE, switch back to live mode with since filter 495 // After EOSE, switch back to live mode with since filter
481 self.wait_for_eose(relay_url).await; 496 self.wait_for_eose(relay_url).await;
482 497
483 // Re-add since filter for ongoing live sync 498 // Re-add since filter for ongoing live sync
484 let since = Timestamp::now() - 900; // 15 minutes ago 499 let since = Timestamp::now() - 900; // 15 minutes ago
485 self.resubscribe_with_since(relay_url, since).await; 500 self.resubscribe_with_since(relay_url, since).await;
@@ -663,15 +678,15 @@ async fn check_relay_removal(&mut self) {
663 678
664``` 679```
665src/sync/ 680src/sync/
666├── mod.rs # Module exports, constants 681├── mod.rs # SyncManager + state types (FollowingRepoRootEvents, SyncRelays)
667├── manager.rs # SyncManager - orchestrates sync 682├── self_subscriber.rs # SelfSubscriber + batching logic
668├── state.rs # FollowingRepoRootEvents + SyncRelays HashMaps
669├── self_subscriber.rs # Self-subscriber + batching logic
670├── relay_connection.rs # Per-relay WebSocket + filters 683├── relay_connection.rs # Per-relay WebSocket + filters
671├── health.rs # RelayHealthTracker (reuse from v1) 684├── health.rs # RelayHealthTracker (reuse from v1)
672└── metrics.rs # SyncMetrics (reuse from v1) 685└── metrics.rs # SyncMetrics (reuse from v1)
673``` 686```
674 687
688> **Note:** SyncManager and state type aliases are colocated in `mod.rs` following the pattern of `src/http/mod.rs` (HttpService) and `src/metrics/mod.rs` (Metrics). See the earlier "Module Structure" section for rationale.
689
675## Comparison: v1 vs v2 690## Comparison: v1 vs v2
676 691
677| Aspect | v1 (Current) | v2 (Simplified) | 692| Aspect | v1 (Current) | v2 (Simplified) |