diff options
Diffstat (limited to 'docs/explanation/grasp-02-proactive-sync.md')
| -rw-r--r-- | docs/explanation/grasp-02-proactive-sync.md | 57 |
1 files changed, 49 insertions, 8 deletions
diff --git a/docs/explanation/grasp-02-proactive-sync.md b/docs/explanation/grasp-02-proactive-sync.md index ed8fdbf..6696e27 100644 --- a/docs/explanation/grasp-02-proactive-sync.md +++ b/docs/explanation/grasp-02-proactive-sync.md | |||
| @@ -47,20 +47,37 @@ This state starts afresh when the binary loads. | |||
| 47 | ### RepoSyncIndex (Source of Truth) | 47 | ### RepoSyncIndex (Source of Truth) |
| 48 | 48 | ||
| 49 | ```rust | 49 | ```rust |
| 50 | /// What we WANT to sync - derived from events received via self-subscription. | 50 | /// What we WANT to sync - derived from events received via self-subscription |
| 51 | /// Updated immediately when self-subscriber batch fires. | 51 | /// and from purgatory announcements. |
| 52 | /// Updated immediately when self-subscriber batch fires or purgatory sync timer runs. | ||
| 52 | /// Key: repo addressable ref - 30617:pubkey:identifier | 53 | /// Key: repo addressable ref - 30617:pubkey:identifier |
| 53 | pub type RepoSyncIndex = Arc<RwLock<HashMap<String, RepoSyncNeeds>>>; | 54 | pub type RepoSyncIndex = Arc<RwLock<HashMap<String, RepoSyncNeeds>>>; |
| 54 | 55 | ||
| 56 | /// Controls which sync filters are built for a repo | ||
| 57 | #[derive(Debug, Clone, Copy, PartialEq, Eq, Default)] | ||
| 58 | pub enum SyncLevel { | ||
| 59 | #[default] | ||
| 60 | Full, // Full L2 + L3 sync (promoted repos with git data) | ||
| 61 | StateOnly, // Only state events (kind 30618) — for purgatory announcements | ||
| 62 | } | ||
| 63 | |||
| 55 | #[derive(Debug, Clone, Default)] | 64 | #[derive(Debug, Clone, Default)] |
| 56 | pub struct RepoSyncNeeds { | 65 | pub struct RepoSyncNeeds { |
| 57 | /// Relay URLs listed in this repo's 30617 announcement | 66 | /// Relay URLs listed in this repo's 30617 announcement |
| 58 | pub relays: HashSet<String>, | 67 | pub relays: HashSet<String>, |
| 59 | /// Root event IDs - 1617/1618/1621 - that reference this repo | 68 | /// Root event IDs - 1617/1618/1621 - that reference this repo |
| 60 | pub root_events: HashSet<EventId>, | 69 | pub root_events: HashSet<EventId>, |
| 70 | /// Controls which filters are built: Full (L2+L3) or StateOnly (kind 30618 only) | ||
| 71 | pub sync_level: SyncLevel, | ||
| 61 | } | 72 | } |
| 62 | ``` | 73 | ``` |
| 63 | 74 | ||
| 75 | **Two sources populate `RepoSyncIndex`:** | ||
| 76 | |||
| 77 | 1. **`SelfSubscriber`** — monitors the relay's own event stream for accepted announcements (kinds 30617, 1617, 1618, 1621). Adds entries with `SyncLevel::Full`. When an announcement is promoted from purgatory to the database, the SelfSubscriber sees it and upgrades the entry to `Full`. | ||
| 78 | |||
| 79 | 2. **Purgatory announcement sync timer** (`run_purgatory_announcement_sync`, every 5 seconds) — iterates `purgatory.announcements_for_sync()` and ensures each purgatory announcement has a `SyncLevel::StateOnly` entry in `RepoSyncIndex`. This is the only registration path for purgatory announcements because they are not saved to the database and therefore never seen by the SelfSubscriber. | ||
| 80 | |||
| 64 | ### RelaySyncIndex (Confirmed State + Connection) | 81 | ### RelaySyncIndex (Confirmed State + Connection) |
| 65 | 82 | ||
| 66 | ```rust | 83 | ```rust |
| @@ -336,7 +353,23 @@ The sync system uses three background tasks that run continuously: | |||
| 336 | 353 | ||
| 337 | 1. Queue events to `PendingUpdates` | 354 | 1. Queue events to `PendingUpdates` |
| 338 | 2. Timer fires (interval, does not reset on events) | 355 | 2. Timer fires (interval, does not reset on events) |
| 339 | 3. Process batch: update RepoSyncIndex → derive targets → send AddFilters to SyncManager | 356 | 3. Process batch: update RepoSyncIndex with `SyncLevel::Full` → derive targets → send AddFilters to SyncManager |
| 357 | |||
| 358 | **Note**: The SelfSubscriber only sees announcements that have been accepted to the database (promoted from purgatory). Purgatory announcements are registered separately by the purgatory sync timer (see below). | ||
| 359 | |||
| 360 | ### 4. Purgatory Announcement Sync Timer (`run_purgatory_announcement_sync`) | ||
| 361 | |||
| 362 | **Purpose**: Register purgatory announcements in `RepoSyncIndex` so state events are synced for them | ||
| 363 | |||
| 364 | **Interval**: Every 5 seconds (200ms in test mode) | ||
| 365 | |||
| 366 | **Flow**: | ||
| 367 | |||
| 368 | 1. Iterate `purgatory.announcements_for_sync()` | ||
| 369 | 2. For each announcement not already in `RepoSyncIndex`: insert with `SyncLevel::StateOnly` | ||
| 370 | 3. When an announcement is promoted (git data arrives), the SelfSubscriber sees the newly accepted event and upgrades the entry to `SyncLevel::Full` | ||
| 371 | |||
| 372 | **Why a separate timer?** Purgatory announcements are never saved to the database, so the SelfSubscriber never sees them. The timer bridges this gap, ensuring state events are synced for repos that may still receive git data. | ||
| 340 | 373 | ||
| 341 | --- | 374 | --- |
| 342 | 375 | ||
| @@ -602,9 +635,10 @@ flowchart TB | |||
| 602 | 635 | ||
| 603 | - Self-subscriber monitors own relay for 30617, 1617, 1618, 1621 (NOT 1619 or 30618) | 636 | - Self-subscriber monitors own relay for 30617, 1617, 1618, 1621 (NOT 1619 or 30618) |
| 604 | - Batches events in `PendingUpdates` (5 second window via interval timer) | 637 | - Batches events in `PendingUpdates` (5 second window via interval timer) |
| 605 | - `process_batch()` updates RepoSyncIndex, then builds AddFilters **directly** (no compute_actions) | 638 | - `process_batch()` updates RepoSyncIndex with `SyncLevel::Full`, then builds AddFilters **directly** (no compute_actions) |
| 606 | - AddFilters sent via channel to SyncManager, which calls `handle_new_sync_filters()` | 639 | - AddFilters sent via channel to SyncManager, which calls `handle_new_sync_filters()` |
| 607 | - This path does NOT use compute_actions because it's building fresh filters from the updated index | 640 | - This path does NOT use compute_actions because it's building fresh filters from the updated index |
| 641 | - Purgatory announcements (not in DB) are registered separately by the purgatory sync timer with `SyncLevel::StateOnly` | ||
| 608 | 642 | ||
| 609 | --- | 643 | --- |
| 610 | 644 | ||
| @@ -687,16 +721,23 @@ fn compute_actions( | |||
| 687 | - **Tags**: lowercase `a`, uppercase `A`, and `q` tags for comprehensive coverage | 721 | - **Tags**: lowercase `a`, uppercase `A`, and `q` tags for comprehensive coverage |
| 688 | - **Batching**: Per 100 repo refs | 722 | - **Batching**: Per 100 repo refs |
| 689 | - **Function**: `build_repo_tag_filters(repos, since)` | 723 | - **Function**: `build_repo_tag_filters(repos, since)` |
| 724 | - **Only for `SyncLevel::Full` repos** — purgatory announcements (`StateOnly`) skip this layer | ||
| 690 | 725 | ||
| 691 | ### Layer 3: Events Tagging Our Root Events | 726 | ### Layer 3: Events Tagging Our Root Events |
| 692 | 727 | ||
| 693 | - **Tags**: lowercase `e`, uppercase `E`, and `q` tags for comprehensive coverage | 728 | - **Tags**: lowercase `e`, uppercase `E`, and `q` tags for comprehensive coverage |
| 694 | - **Batching**: Per 100 event IDs | 729 | - **Batching**: Per 100 event IDs |
| 695 | - **Function**: `build_root_event_tag_filters(root_events, since)` | 730 | - **Function**: `build_root_event_tag_filters(root_events, since)` |
| 731 | - **Only for `SyncLevel::Full` repos** — purgatory announcements (`StateOnly`) skip this layer | ||
| 732 | |||
| 733 | ### Combined Layer 2+3 (SyncLevel-Aware) | ||
| 734 | |||
| 735 | The `build_sync_level_aware_filters()` function combines both layers, partitioning repos by `SyncLevel`: | ||
| 696 | 736 | ||
| 697 | ### Combined Layer 2+3 | 737 | - **`Full` repos**: state event filters + repo-tag filters + root-event-tag filters |
| 738 | - **`StateOnly` repos**: state event filters only (kind 30618 with `#d` tags) | ||
| 698 | 739 | ||
| 699 | The `build_layer2_and_layer3_filters()` function combines both layers. Used by: | 740 | Used by: |
| 700 | 741 | ||
| 701 | - `recompute_new_sync_filters_for_relay` for new item subscriptions | 742 | - `recompute_new_sync_filters_for_relay` for new item subscriptions |
| 702 | - `reconstruct_filters` for rebuilding from confirmed state | 743 | - `reconstruct_filters` for rebuilding from confirmed state |
| @@ -871,9 +912,9 @@ flowchart TB | |||
| 871 | 912 | ||
| 872 | ``` | 913 | ``` |
| 873 | src/sync/ | 914 | src/sync/ |
| 874 | ├── mod.rs # SyncManager, main loop, data structures | 915 | ├── mod.rs # SyncManager, main loop, data structures, SyncLevel, run_purgatory_announcement_sync |
| 875 | ├── algorithms.rs # derive_relay_targets(), compute_actions() | 916 | ├── algorithms.rs # derive_relay_targets(), compute_actions() |
| 876 | ├── filters.rs # build_announcement_filter(), build_layer2_and_layer3_filters() | 917 | ├── filters.rs # build_announcement_filter(), build_sync_level_aware_filters() |
| 877 | ├── health.rs # RelayHealthTracker with exponential backoff | 918 | ├── health.rs # RelayHealthTracker with exponential backoff |
| 878 | ├── relay_connection.rs # RelayConnection, RelayEvent handling | 919 | ├── relay_connection.rs # RelayConnection, RelayEvent handling |
| 879 | ├── self_subscriber.rs # SelfSubscriber with batching | 920 | ├── self_subscriber.rs # SelfSubscriber with batching |