upleb.uk

Public git repos — served from a NIP-34 GRASP relay at git.upleb.uk

summaryrefslogtreecommitdiff
path: root/docs/explanation
diff options
context:
space:
mode:
authorDanConwayDev <DanConwayDev@protonmail.com>2025-12-10 13:08:41 +0000
committerDanConwayDev <DanConwayDev@protonmail.com>2025-12-10 13:08:41 +0000
commit67f7ebdaeff7b17af077c6ae7d2ecdf786ddf2ef (patch)
tree7f8952c9648b8563ee7e345e761013944acb0d01 /docs/explanation
parent8cfe8546e5ed1118adae6bfa041611e94d15c6dd (diff)
docs: update sync docs post implementation
Diffstat (limited to 'docs/explanation')
-rw-r--r--docs/explanation/architecture.md60
-rw-r--r--docs/explanation/grasp-02-proactive-sync-v4.md97
2 files changed, 151 insertions, 6 deletions
diff --git a/docs/explanation/architecture.md b/docs/explanation/architecture.md
index 3fb895c..efa3423 100644
--- a/docs/explanation/architecture.md
+++ b/docs/explanation/architecture.md
@@ -11,6 +11,7 @@
11After examining both the reference implementation and HTTP server options, we have two options: 11After examining both the reference implementation and HTTP server options, we have two options:
12 12
13#### Option 1: Hook-Based (Reference Implementation Approach) 13#### Option 1: Hook-Based (Reference Implementation Approach)
14
14- Use standard Git HTTP backend 15- Use standard Git HTTP backend
15- Create pre-receive and post-receive hooks 16- Create pre-receive and post-receive hooks
16- Hooks query the Nostr relay and validate pushes 17- Hooks query the Nostr relay and validate pushes
@@ -18,6 +19,7 @@ After examining both the reference implementation and HTTP server options, we ha
18- **Cons**: Requires hook management, harder to test, less Rust-native 19- **Cons**: Requires hook management, harder to test, less Rust-native
19 20
20#### Option 2: Inline Authorization (Recommended) 21#### Option 2: Inline Authorization (Recommended)
22
21- Intercept Git receive-pack requests in the HTTP handler 23- Intercept Git receive-pack requests in the HTTP handler
22- Validate against Nostr state before spawning Git process 24- Validate against Nostr state before spawning Git process
23- Only forward valid pushes to Git 25- Only forward valid pushes to Git
@@ -30,13 +32,15 @@ After examining both the reference implementation and HTTP server options, we ha
30 32
311. **Full control over HTTP layer**: Using Hyper directly gives us complete control over request handling, WebSocket upgrades, and CORS headers. 331. **Full control over HTTP layer**: Using Hyper directly gives us complete control over request handling, WebSocket upgrades, and CORS headers.
32 34
332. **Better Developer Experience**: 352. **Better Developer Experience**:
36
34 - Validation errors can be returned as proper HTTP responses 37 - Validation errors can be returned as proper HTTP responses
35 - No need to parse hook stderr output 38 - No need to parse hook stderr output
36 - Shared state between Git and Nostr components 39 - Shared state between Git and Nostr components
37 - Pure Rust testing without shell scripts 40 - Pure Rust testing without shell scripts
38 41
393. **Simpler Deployment**: 423. **Simpler Deployment**:
43
40 - Single binary 44 - Single binary
41 - No hook symlinks or permissions to manage 45 - No hook symlinks or permissions to manage
42 - No multi-process coordination 46 - No multi-process coordination
@@ -93,6 +97,7 @@ After examining both the reference implementation and HTTP server options, we ha
93### 1. Main Server ([`src/main.rs`](src/main.rs)) 97### 1. Main Server ([`src/main.rs`](src/main.rs))
94 98
95**Responsibilities:** 99**Responsibilities:**
100
96- Initialize configuration from environment (clap + dotenvy) 101- Initialize configuration from environment (clap + dotenvy)
97- Set up Hyper HTTP server with request routing 102- Set up Hyper HTTP server with request routing
98- Initialize Nostr relay builder with custom [`Nip34WritePolicy`](src/nostr/builder.rs:51) 103- Initialize Nostr relay builder with custom [`Nip34WritePolicy`](src/nostr/builder.rs:51)
@@ -101,6 +106,7 @@ After examining both the reference implementation and HTTP server options, we ha
101- Handle graceful shutdown 106- Handle graceful shutdown
102 107
103**Key Dependencies:** 108**Key Dependencies:**
109
104```rust 110```rust
105hyper = "1" 111hyper = "1"
106tokio = { version = "1", features = ["full"] } 112tokio = { version = "1", features = ["full"] }
@@ -112,6 +118,7 @@ nostr-lmdb = "0.43"
112### 2. HTTP Module ([`src/http/mod.rs`](src/http/mod.rs)) 118### 2. HTTP Module ([`src/http/mod.rs`](src/http/mod.rs))
113 119
114**Responsibilities:** 120**Responsibilities:**
121
115- Route HTTP requests to appropriate handlers 122- Route HTTP requests to appropriate handlers
116- WebSocket upgrade for Nostr relay at `/` 123- WebSocket upgrade for Nostr relay at `/`
117- Git Smart HTTP endpoints at `/<npub>/<identifier>.git/*` 124- Git Smart HTTP endpoints at `/<npub>/<identifier>.git/*`
@@ -228,7 +235,7 @@ Provides structures for parsing NIP-34 events:
228/// Parsed repository announcement (Kind 30617) 235/// Parsed repository announcement (Kind 30617)
229pub struct RepositoryAnnouncement { ... } 236pub struct RepositoryAnnouncement { ... }
230 237
231/// Parsed repository state (Kind 30618) 238/// Parsed repository state (Kind 30618)
232pub struct RepositoryState { ... } 239pub struct RepositoryState { ... }
233``` 240```
234 241
@@ -332,10 +339,12 @@ See [test-strategy.md](../reference/test-strategy.md) for comprehensive testing
332### Quick Overview 339### Quick Overview
333 340
334**Integration Tests** ([`tests/`](tests/)): 341**Integration Tests** ([`tests/`](tests/)):
342
335- Use [`TestRelay`](tests/common/relay.rs:14) fixture for automatic relay lifecycle 343- Use [`TestRelay`](tests/common/relay.rs:14) fixture for automatic relay lifecycle
336- Each test file in [`tests/`](tests/) covers a GRASP-01 requirement 344- Each test file in [`tests/`](tests/) covers a GRASP-01 requirement
337 345
338**Audit Tests** ([`grasp-audit/`](grasp-audit/)): 346**Audit Tests** ([`grasp-audit/`](grasp-audit/)):
347
339- Reusable compliance testing for any GRASP implementation 348- Reusable compliance testing for any GRASP implementation
340- Spec-mirrored structure in [`grasp-audit/src/specs/grasp01/`](grasp-audit/src/specs/grasp01/) 349- Spec-mirrored structure in [`grasp-audit/src/specs/grasp01/`](grasp-audit/src/specs/grasp01/)
341 350
@@ -367,11 +376,52 @@ async fn test_nip01_websocket_connection() {
367- Maintainer sets computed once per event validation 376- Maintainer sets computed once per event validation
368- State lookups use database indexes 377- State lookups use database indexes
369 378
379## Proactive Sync (GRASP-02)
380
381The ngit-grasp relay implements **Proactive Sync of Nostr Eevents**, which synchronizes repository data from external relays listed in 30617 repository announcements. This enables the relay to maintain complete repository graphs even when events are published to other listed relays.
382
383**Key Features:**
384
385- **Self-subscription** discovery - monitors own relay for announcements and root events to follow
386- **Three-way diff** (`compute_actions`) determines new subscriptions needed
387- **Smart reconnection** - uses `since` filter for quick reconnects (<15 min), fresh sync otherwise
388- **Health tracking** with exponential backoff for failing relays
389- **Daily sync** with random 23-25h timer to detect state drift
390- **Filter consolidation** when count exceeds 70 to prevent subscription explosion
391
392**Architecture:**
393
394```
395┌─────────────────────────────────────────────────────────────┐
396│ SyncManager │
397├─────────────────────────────────────────────────────────────┤
398│ │
399│ ┌──────────────────┐ ┌──────────────────┐ │
400│ │ SelfSubscriber │──actions──▶ │ Main Event Loop │ │
401│ │ (own relay) │ │ (Arc<Mutex>) │ │
402│ └──────────────────┘ └────────┬─────────┘ │
403│ │ │
404│ ┌──────────────────┐ ┌────────▼─────────┐ │
405│ │ Daily Timer │──────────────▶ RelayConnection │ │
406│ │ (23-25h random) │ │ per external │ │
407│ └──────────────────┘ │ relay │ │
408│ └──────────────────┘ │
409│ ┌──────────────────┐ │
410│ │ Health Tracker │ Exponential backoff, dead detection │
411│ │ (DashMap) │ │
412│ └──────────────────┘ │
413└─────────────────────────────────────────────────────────────┘
414```
415
416**Source Code:** [`src/sync/`](src/sync/)
417
418For full design details, see [grasp-02-proactive-sync-v4.md](grasp-02-proactive-sync-v4.md).
419
370## Future Extensions 420## Future Extensions
371 421
372### GRASP-02: Proactive Sync 422### GRASP-02: Proactive Sync
373 423
374See [grasp-02-proactive-sync.md](grasp-02-proactive-sync.md) for detailed design. 424GRASP-02 is only partially implemented. still outstanding is the proactive sync of git data for 1. state event and 2. PRs / PR Update refs.
375 425
376### GRASP-05: Archive 426### GRASP-05: Archive
377 427
@@ -437,6 +487,6 @@ The key insight is that we don't need to rely on Git's hook mechanism when we ha
437## Related Documentation 487## Related Documentation
438 488
439- [Inline Authorization Explanation](inline-authorization.md) - Why we chose this approach 489- [Inline Authorization Explanation](inline-authorization.md) - Why we chose this approach
440- [GRASP-02 Proactive Sync](grasp-02-proactive-sync.md) - Future work design 490- [GRASP-02 Proactive Sync v4 Design](grasp-02-proactive-sync-v4.md) - Current production sync implementation
441- [Test Strategy](../reference/test-strategy.md) - Comprehensive testing documentation 491- [Test Strategy](../reference/test-strategy.md) - Comprehensive testing documentation
442- [GRASP-01 Implementation Learnings](../learnings/grasp-01-implementation.md) - Patterns and lessons learned \ No newline at end of file 492- [GRASP-01 Implementation Learnings](../learnings/grasp-01-implementation.md) - Patterns and lessons learned
diff --git a/docs/explanation/grasp-02-proactive-sync-v4.md b/docs/explanation/grasp-02-proactive-sync-v4.md
index aba88a5..5ac92cd 100644
--- a/docs/explanation/grasp-02-proactive-sync-v4.md
+++ b/docs/explanation/grasp-02-proactive-sync-v4.md
@@ -1226,4 +1226,99 @@ impl SelfSubscriber {
1226 } 1226 }
1227 } 1227 }
1228} 1228}
1229``` \ No newline at end of file 1229```
1230
1231---
1232
1233## Implementation Notes
1234
1235This section documents the actual implementation details as of December 2024 (Phases 1-10 complete).
1236
1237### Architectural Decisions During Implementation
1238
1239**Phase 7 Refactoring**: The `SyncManager::run()` method required refactoring to use `Arc<Mutex<SyncManager>>` for shared access. The daily timer and disconnect checker tasks need to access the manager, so `self` is wrapped after initial setup:
1240
1241```rust
1242// 7. Wrap self in Arc<Mutex> for sharing with timer task
1243let sync_manager = Arc::new(Mutex::new(self));
1244```
1245
1246This allows background tasks (daily timer, disconnect checker) to acquire the lock when needed while the main event loop handles actions from the self-subscriber.
1247
1248**Health Module**: The health tracking module was adapted from the v3 implementation at `work/sync-v3/health.rs`. The implementation uses:
1249- `DashMap` for thread-safe concurrent access without external locking
1250- Three states: `Healthy`, `Degraded`, `Dead`
1251- Exponential backoff: `base * 2^(failures-1)`, capped at max_backoff
1252- Dead threshold: 24 hours of continuous failures
1253- Dead relay retry: Once per 24 hours
1254
1255### Implementation Constants
1256
1257| Constant | Value | Purpose |
1258|----------|-------|---------|
1259| `CONSOLIDATION_THRESHOLD` | 70 filters | Maximum filters before triggering consolidation |
1260| `CONSOLIDATION_WAIT_TIMEOUT_SECS` | 30 seconds | Timeout for pending batches during consolidation |
1261| `QUICK_RECONNECT_WINDOW_SECS` | 15 minutes | Window for quick reconnect vs fresh sync |
1262| `DISCONNECT_CHECK_INTERVAL_SECS` | 60 seconds | Interval for checking empty relays to disconnect |
1263| `DEAD_THRESHOLD_HOURS` | 24 hours | Time before relay marked as dead |
1264| `BASE_BACKOFF_SECS` | 5 seconds | Base duration for exponential backoff |
1265
1266### Daily Timer Randomization
1267
1268The daily timer uses randomization between 23-25 hours to prevent thundering herd effects when multiple ngit-grasp instances are running:
1269
1270```rust
1271let hours = 23.0 + rand::thread_rng().gen::<f64>() * 2.0;
1272```
1273
1274### Bootstrap Relay Protection
1275
1276Bootstrap relays are never disconnected by the cleanup system. The `check_disconnects()` method explicitly filters them out:
1277
1278```rust
1279.filter(|(_, state)| {
1280 !state.is_bootstrap &&
1281 state.repos.is_empty() &&
1282 state.root_events.is_empty()
1283})
1284```
1285
1286### Graceful Shutdown
1287
1288Shutdown uses a tokio broadcast channel for coordinated termination:
1289
1290```rust
1291let (shutdown_tx, _shutdown_rx) = broadcast::channel(1);
1292```
1293
1294Each background task (self-subscriber, daily timer, disconnect checker) receives its own `broadcast::Receiver` subscription and monitors for the shutdown signal in its main loop.
1295
1296### Actual Module Structure
1297
1298The implemented module structure differs from the original spec:
1299
1300```
1301src/sync/
1302├── mod.rs # SyncManager, main loop, index types, metrics
1303├── algorithms.rs # derive_relay_targets, compute_actions, AddFilters
1304├── filters.rs # build_announcement_filter, build_layer2_and_layer3_filters
1305├── health.rs # RelayHealthTracker, HealthState, exponential backoff
1306├── relay_connection.rs # RelayConnection, RelayEvent, WebSocket handling
1307└── self_subscriber.rs # SelfSubscriber, RelayAction, batching logic
1308```
1309
1310Key differences from spec:
1311- No separate `state.rs` - types are defined in `mod.rs`
1312- No separate `actions.rs` - moved to `algorithms.rs`
1313- No separate `consolidation.rs` - consolidation logic in `mod.rs`
1314- No separate `metrics.rs` - `SyncMetrics` defined in `mod.rs`
1315
1316### Deviations from Original v4 Spec
1317
13181. **RelayState lacks `connection` field**: The spec showed `connection: Option<RelayConnection>` in `RelayState`, but the implementation stores connections in a separate `HashMap<String, RelayConnection>` in `SyncManager`.
1319
13202. **SelfSubscriber simplified**: The actual implementation uses `RelayAction` enum (SpawnRelay/AddFilters) rather than directly using `AddFilters` struct.
1321
13223. **Consolidation wait_pending_complete**: The spec described a `wait_pending_complete()` method, but the implementation uses a simpler timeout-based approach checking pending batches.
1323
13244. **Timestamp API**: Uses `Timestamp::now().as_secs()` instead of `.as_u64()` due to nostr-sdk 0.43 API. \ No newline at end of file