upleb.uk

Public git repos — served from a NIP-34 GRASP relay at git.upleb.uk

summaryrefslogtreecommitdiff
diff options
context:
space:
mode:
authorDanConwayDev <DanConwayDev@protonmail.com>2025-12-19 16:02:04 +0000
committerDanConwayDev <DanConwayDev@protonmail.com>2025-12-19 16:02:04 +0000
commit1adbd93e5bb8e14403ba64a76d5dc93209227514 (patch)
treecf21d9ec7958ac1918143b72649c53eb14804283
parent02a90c109d4d08c6a54184f821c100f4eba92545 (diff)
docs: cleanup
Diffstat
-rw-r--r--docs/explanation/state-structure-redesign-proposal.md373
-rw-r--r--work/phase1-baseline.md159
2 files changed, 0 insertions, 532 deletions
diff --git a/docs/explanation/state-structure-redesign-proposal.md b/docs/explanation/state-structure-redesign-proposal.md
deleted file mode 100644
index 0a27cf4..0000000
--- a/docs/explanation/state-structure-redesign-proposal.md
+++ /dev/null
@@ -1,373 +0,0 @@
1# State Structure Redesign Proposal v2
2
3## The Core Problem
4
5We need to transform:
6- **Repo Announcements** (30617) that list relays
7- **Root Events** (1617/1618/1619/1621) that tag repos
8
9Into:
10- **Per-relay subscriptions**: which repos and root events to sync from each relay
11
12And generate **RelayActions** when this mapping changes.
13
14---
15
16## Proposed Data Model
17
18### 1. RepoIndex (Primary Source of Truth)
19
20```rust
21/// Everything we know about repos we're tracking
22/// Key: repo addressable ref ("30617:pubkey:identifier")
23pub type RepoIndex = Arc<RwLock<HashMap<String, RepoInfo>>>;
24
25#[derive(Debug, Clone, Default)]
26pub struct RepoInfo {
27 /// Relay URLs listed in the repo's announcement
28 pub relays: HashSet<String>,
29 /// Root event IDs that reference this repo
30 pub root_events: HashSet<EventId>,
31}
32```
33
34**Updated by:** Database init, batch processing of new announcements/root events
35
36### 2. RelayIndex (Applied State)
37
38```rust
39/// What we've told each relay to sync
40/// Key: relay URL
41pub type RelayIndex = Arc<RwLock<HashMap<String, SyncTarget>>>;
42
43#[derive(Debug, Clone, Default, PartialEq)]
44pub struct SyncTarget {
45 /// Repos we're syncing for this relay
46 pub repos: HashSet<String>,
47 /// Root events we're tracking
48 pub root_events: HashSet<EventId>,
49}
50```
51
52**Updated by:** SyncManager after RelayActions are applied
53
54---
55
56## The Transformation
57
58```mermaid
59flowchart LR
60 subgraph Input
61 RA[Repo Announcements]
62 RE[Root Events]
63 end
64
65 subgraph RepoIndex
66 R1[repo_a: relays=X,Y events=1,2]
67 R2[repo_b: relays=Y,Z events=3]
68 end
69
70 subgraph Derived Target
71 T1[relay_X: repos=a events=1,2]
72 T2[relay_Y: repos=a,b events=1,2,3]
73 T3[relay_Z: repos=b events=3]
74 end
75
76 subgraph RelayIndex Applied
77 A1[relay_X: repos=a events=1,2]
78 A2[relay_Y: repos=a events=1,2]
79 end
80
81 RA --> R1
82 RA --> R2
83 RE --> R1
84 RE --> R2
85
86 R1 --> T1
87 R1 --> T2
88 R2 --> T2
89 R2 --> T3
90```
91
92The **diff** between Derived Target and RelayIndex produces RelayActions:
93- relay_Y needs AddFilters for repo_b and event 3
94- relay_Z needs SpawnRelay
95
96---
97
98## Algorithm: derive_target_from_repo_index
99
100```rust
101/// Derive what we SHOULD be syncing from the repo data
102fn derive_relay_targets(repo_index: &HashMap<String, RepoInfo>) -> HashMap<String, SyncTarget> {
103 let mut targets: HashMap<String, SyncTarget> = HashMap::new();
104
105 for (repo_ref, info) in repo_index {
106 // For each relay that lists this repo
107 for relay_url in &info.relays {
108 let target = targets.entry(relay_url.clone()).or_default();
109 target.repos.insert(repo_ref.clone());
110 target.root_events.extend(info.root_events.iter().cloned());
111 }
112 }
113
114 targets
115}
116```
117
118---
119
120## Algorithm: process_batch
121
122```rust
123async fn process_batch(&self, pending: &mut PendingUpdates) {
124 // ============================================
125 // STEP 1: Update RepoIndex from batch
126 // ============================================
127
128 let mut repo_index = self.repo_index.write().await;
129
130 // 1a. Process root events - add to repo's root_events set
131 for event in pending.root_events.drain(..) {
132 for repo_ref in extract_repo_refs(&event) {
133 repo_index.entry(repo_ref)
134 .or_default()
135 .root_events
136 .insert(event.id);
137 }
138 }
139
140 // 1b. Process announcements - update repo's relay set
141 for event in pending.announcements.drain(..) {
142 if !lists_our_service(&event) {
143 continue;
144 }
145 let repo_ref = build_repo_ref(&event);
146 let relay_urls: HashSet<String> = extract_relay_urls(&event)
147 .into_iter()
148 .filter(|url| !is_own_relay(url))
149 .collect();
150
151 // Replace relay set (handles updates that change relays)
152 repo_index.entry(repo_ref)
153 .or_default()
154 .relays = relay_urls;
155 }
156
157 // ============================================
158 // STEP 2: Derive target state from RepoIndex
159 // ============================================
160
161 let target = derive_relay_targets(&repo_index);
162 drop(repo_index); // Release write lock
163
164 // ============================================
165 // STEP 3: Diff target vs applied (RelayIndex)
166 // ============================================
167
168 let applied = self.relay_index.read().await;
169 let actions = compute_relay_actions(&target, &applied);
170 drop(applied); // Release read lock
171
172 // ============================================
173 // STEP 4: Send actions & update RelayIndex
174 // ============================================
175
176 for action in actions {
177 match &action {
178 RelayAction::SpawnRelay { relay_url, repos_and_root_events } => {
179 // Update RelayIndex with new relay
180 let mut applied = self.relay_index.write().await;
181 applied.insert(relay_url.clone(), SyncTarget {
182 repos: repos_and_root_events.keys().cloned().collect(),
183 root_events: repos_and_root_events.values()
184 .flat_map(|e| e.iter().cloned())
185 .collect(),
186 });
187 }
188 RelayAction::AddFilters { relay_url, repos_and_new_root_event } => {
189 // Update RelayIndex with additions
190 let mut applied = self.relay_index.write().await;
191 if let Some(target) = applied.get_mut(relay_url) {
192 for (repo, events) in repos_and_new_root_event {
193 target.repos.insert(repo.clone());
194 target.root_events.extend(events.iter().cloned());
195 }
196 }
197 }
198 }
199
200 // Send action to SyncManager
201 let _ = self.action_tx.send(action).await;
202 }
203}
204```
205
206---
207
208## Algorithm: compute_relay_actions
209
210```rust
211fn compute_relay_actions(
212 target: &HashMap<String, SyncTarget>,
213 applied: &HashMap<String, SyncTarget>,
214) -> Vec<RelayAction> {
215 let mut actions = Vec::new();
216
217 for (relay_url, target_state) in target {
218 match applied.get(relay_url) {
219 None => {
220 // New relay - spawn it
221 let mut repos_and_events = HashMap::new();
222 for repo in &target_state.repos {
223 // Get events for this specific repo
224 let events = target_state.root_events.clone(); // simplified
225 repos_and_events.insert(repo.clone(), events);
226 }
227 actions.push(RelayAction::SpawnRelay {
228 relay_url: relay_url.clone(),
229 repos_and_root_events: repos_and_events,
230 });
231 }
232 Some(applied_state) => {
233 // Existing relay - check for new repos/events
234 let new_repos: HashSet<_> = target_state.repos
235 .difference(&applied_state.repos)
236 .cloned()
237 .collect();
238 let new_events: HashSet<_> = target_state.root_events
239 .difference(&applied_state.root_events)
240 .cloned()
241 .collect();
242
243 if !new_repos.is_empty() || !new_events.is_empty() {
244 let mut repos_and_events = HashMap::new();
245 for repo in &new_repos {
246 repos_and_events.insert(repo.clone(), new_events.clone());
247 }
248 // Also handle new events for existing repos
249 if !new_events.is_empty() && new_repos.is_empty() {
250 for repo in &applied_state.repos {
251 repos_and_events.insert(repo.clone(), new_events.clone());
252 }
253 }
254
255 actions.push(RelayAction::AddFilters {
256 relay_url: relay_url.clone(),
257 repos_and_new_root_event: repos_and_events,
258 });
259 }
260 }
261 }
262 }
263
264 // Future: detect relay removal (in applied but not in target)
265
266 actions
267}
268```
269
270---
271
272## Handling Announcement Updates
273
274When an announcement is **updated** and changes its relay list:
275
276```mermaid
277flowchart TD
278 A[repo_a announcement updated] --> B[Old: relays X,Y]
279 B --> C[New: relays Y,Z]
280 C --> D[RepoIndex updated: repo_a.relays = Y,Z]
281 D --> E[derive_relay_targets]
282 E --> F[Target: X=empty, Y=repo_a, Z=repo_a]
283 F --> G[Diff with Applied: X=repo_a, Y=repo_a]
284 G --> H1[X: repo_a removed - future RemoveFilters]
285 G --> H2[Z: new relay - SpawnRelay]
286```
287
288The current RelayAction types only support growth (SpawnRelay, AddFilters). Removal would need a new `RemoveFilters` action type - this is a future enhancement.
289
290---
291
292## Name Mappings
293
294| Current | Proposed | Semantics |
295|---------|----------|-----------|
296| `FollowingRepoRootEvents` | `RepoIndex` | Per-repo: relays + root events |
297| `SyncRelays` | `RelayIndex` | Per-relay: what we're syncing (applied state) |
298| - | `SyncTarget` | Struct for repos + events |
299| - | `RepoInfo` | Struct for relay set + event set |
300
301---
302
303## Data Flow Summary
304
305```mermaid
306flowchart TB
307 subgraph Batch Input
308 RA[30617 Announcements]
309 RE[Root Events 1617-1621]
310 end
311
312 subgraph Step 1: Update Source
313 RI[RepoIndex]
314 end
315
316 subgraph Step 2: Derive Target
317 DT[derive_relay_targets]
318 TGT[Target HashMap]
319 end
320
321 subgraph Step 3: Diff
322 RLI[RelayIndex - Applied]
323 DIFF[compute_relay_actions]
324 end
325
326 subgraph Step 4: Apply
327 ACT[RelayActions]
328 SM[SyncManager]
329 end
330
331 RA --> RI
332 RE --> RI
333 RI --> DT
334 DT --> TGT
335 TGT --> DIFF
336 RLI --> DIFF
337 DIFF --> ACT
338 ACT --> SM
339 ACT --> |update| RLI
340```
341
342---
343
344## Files to Modify
345
346| File | Changes |
347|------|---------|
348| [`src/sync/mod.rs`](src/sync/mod.rs) | Replace type aliases with RepoIndex/RelayIndex + structs |
349| [`src/sync/self_subscriber.rs`](src/sync/self_subscriber.rs) | Rewrite process_batch with new algorithm |
350
351---
352
353## Questions for Approval
354
3551. **Naming**: Are `RepoIndex`/`RelayIndex` and `RepoInfo`/`SyncTarget` clear enough?
356
3572. **When to update RelayIndex**: Should we:
358 - (a) Update immediately when generating action (optimistic) ← proposed above
359 - (b) Update only after SyncManager confirms action succeeded
360
3613. **Bootstrap relay**: Keep special-casing it in RelayIndex (always present)?
362
3634. **Future work**: Add `RemoveFilters` action for relay removal, or defer?
364
365---
366
367## Benefits
368
3691. **Logical flow**: Source → Derived → Diff → Actions
3702. **Single source of truth**: RepoIndex is the authoritative data
3713. **Clear transformation**: `derive_relay_targets()` is a pure function
3724. **Handles updates**: Replacing `repo.relays` naturally handles announcement changes
3735. **Testable**: Each step can be unit tested independently \ No newline at end of file
diff --git a/work/phase1-baseline.md b/work/phase1-baseline.md
deleted file mode 100644
index 8bd3902..0000000
--- a/work/phase1-baseline.md
+++ /dev/null
@@ -1,159 +0,0 @@
1# Phase 1: Sync Test Baseline
2
3**Timestamp:** 2025-12-18T16:50:07Z (UTC)
4**Git Commit:** (pre-refactoring baseline)
5
6## Test Execution Command
7```bash
8cargo test --test sync
9```
10
11## Summary Statistics
12
13- **Total Tests:** 40
14- **Passed:** 38 (95%)
15- **Failed:** 2 (5%)
16- **Ignored:** 0
17- **Filtered Out:** 0
18- **Execution Time:** 8.05s
19
20## Passing Tests (38)
21
22### Common Module Tests (7)
23- `common::relay::tests::test_find_free_port`
24- `common::sync_helpers::tests::test_parse_empty_metrics`
25- `common::sync_helpers::tests::test_parse_counter_with_labels`
26- `common::sync_helpers::tests::test_parse_gauge_without_labels`
27- `common::sync_helpers::tests::test_parse_metric_with_relay_url_label`
28- `common::sync_helpers::tests::test_repo_coord_format`
29- `common::sync_helpers::tests::test_build_layer3_comment_with_uppercase_e`
30
31### Sync Helper Builder Tests (6)
32- `common::sync_helpers::tests::test_build_layer3_comment_kind_1`
33- `common::sync_helpers::tests::test_build_layer3_quote_with_q`
34- `common::sync_helpers::tests::test_build_layer3_comment_kind_1111`
35- `common::sync_helpers::tests::test_build_layer2_issue_event`
36- `common::sync_helpers::tests::test_build_layer3_reply_with_e_tag`
37- `common::sync_helpers::tests::test_build_layer2_issue_with_uppercase_a`
38- `common::sync_helpers::tests::test_build_layer2_issue_with_q_tag`
39
40### Metrics Tests (6 passing)
41- `sync::metrics::test_metric_values_are_numeric`
42- `sync::metrics::test_concurrent_metrics_requests`
43- `sync::metrics::test_metrics_availability_during_sync`
44- `sync::metrics::test_connection_failure_increments_counter`
45- `sync::metrics::test_prometheus_format_valid`
46- `sync::metrics::test_relay_connected_status`
47- `sync::metrics::test_health_state_degrades_on_failure`
48- `sync::metrics::test_startup_sync_event_count`
49
50### Live Sync Tests (3)
51- `sync::live_sync::test_live_sync_layer2_events`
52- `sync::live_sync::test_live_sync_layer3_events`
53- `sync::live_sync::test_live_sync_event_ordering`
54
55### Bootstrap Tests (3)
56- `sync::bootstrap::test_announcement_not_listing_relay_is_not_synced`
57- `sync::bootstrap::test_history_sync_without_negentropy`
58- `sync::bootstrap::test_bootstrap_syncs_existing_layer2_events`
59- `sync::bootstrap::test_relay_replays_events_after_restart`
60
61### Discovery Tests (3)
62- `sync::discovery::test_layer2_discovery_with_chain`
63- `sync::discovery::test_discovers_layer3_via_layer2`
64- `sync::discovery::test_recursive_relay_discovery_syncs_announcement`
65
66### Tag Variations Tests (6)
67- `sync::tag_variations::test_layer2_sync_with_lowercase_a_tag`
68- `sync::tag_variations::test_layer2_sync_with_q_tag`
69- `sync::tag_variations::test_layer2_sync_with_uppercase_a_tag`
70- `sync::tag_variations::test_layer3_sync_with_lowercase_e_tag`
71- `sync::tag_variations::test_layer3_sync_with_q_tag`
72- `sync::tag_variations::test_layer3_sync_with_uppercase_e_tag`
73
74## Failing Tests (2)
75
76### 1. sync::metrics::test_live_sync_event_count
77
78**Location:** `tests/sync/metrics.rs:444`
79
80**Error Type:** Assertion failure
81
82**Details:**
83```
84assertion `left == right` failed: Should have 2 live events
85 left: None
86 right: Some(2)
87```
88
89**Root Cause:** Live event counting metric is not being populated. The metric parser is returning `None` when it should find a count of 2 live synced events.
90
91**Output Sample:**
92```
93Live events synced: None
94```
95
96**Impact:** This suggests that the `sync_events_total{sync_type="live"}` metric either:
97- Is not being incremented correctly during live sync
98- Is using a different metric name/label than expected
99- Is not being exposed in the metrics endpoint
100
101---
102
103### 2. sync::metrics::test_multi_source_aggregate_counts
104
105**Location:** `tests/sync/metrics.rs:603`
106
107**Error Type:** Assertion failure
108
109**Details:**
110```
111assertion `left == right` failed: Should have 0 connected
112 left: Some(1)
113 right: Some(0)
114```
115
116**Root Cause:** After stopping a relay connection, the `sync_relays_connected_total` metric is not being decremented. The test expects 0 connected relays after calling stop, but the metric still shows 1.
117
118**Output Sample:**
119```
120Tracked total: Some(1)
121Connected total: Some(1)
122After stop - Tracked total: Some(1)
123After stop - Connected total: Some(1)
124```
125
126**Impact:** This indicates that relay disconnection is not properly updating the connection count metric. This could be:
127- A lifecycle issue where the metric update happens asynchronously after the test assertion
128- A bug where the disconnect handler doesn't decrement the counter
129- A race condition in the test timing
130
131---
132
133## Analysis
134
135### Test Health
136The sync test suite is in relatively good shape with a 95% pass rate. The failures are both isolated to the metrics module and appear to be either timing/synchronization issues or metric collection bugs rather than fundamental sync logic problems.
137
138### Pre-existing Issues
139Both failing tests appear to be pre-existing issues unrelated to the planned refactoring work. They should be tracked separately and not conflated with any issues introduced during the refactor.
140
141### Refactoring Risk Assessment
142- **Low Risk Areas:** Bootstrap, discovery, live_sync, tag_variations modules are all passing
143- **Medium Risk Area:** Metrics tests have 2 failures, but they're specific to metric collection, not sync functionality
144- **Safe to Refactor:** The core sync logic tests are passing, so structural refactoring of test helpers and organization should not affect test outcomes
145
146## Next Steps
147
148This baseline will be used to:
1491. Verify that refactoring doesn't introduce new failures
1502. Distinguish pre-existing failures from regressions
1513. Track if the refactoring inadvertently fixes the existing failures
1524. Ensure that after refactoring, we still have 38 passing tests (or more if we fix the failing ones)
153
154## Notes
155
156- Both failures are in `tests/sync/metrics.rs`
157- The failures appear to be metric collection/timing issues rather than sync logic bugs
158- All functional sync tests (bootstrap, discovery, live_sync, tag_variations) are passing
159- The refactoring should not affect these test results unless we accidentally change metric collection timing \ No newline at end of file
generated by cgit v1.2.3 (git 2.39.1) at 2026-06-05 03:26:50 +0000