diff options
| author | DanConwayDev <DanConwayDev@protonmail.com> | 2026-02-25 15:14:46 +0000 |
|---|---|---|
| committer | DanConwayDev <DanConwayDev@protonmail.com> | 2026-02-25 15:14:46 +0000 |
| commit | 8cd232727ae31613abba7a3d0485a1cb94fda2f3 (patch) | |
| tree | 7b761929d919867fee3ffc68df436c250b9c5336 | |
| parent | 5ad9d9093fcbe7037e5474a9d8fa20a0b64fb79a (diff) | |
docs: remove comparison doc and update architecture to reflect announcement purgatory
| -rw-r--r-- | docs/explanation/architecture.md | 131 | ||||
| -rw-r--r-- | docs/explanation/comparison.md | 379 |
2 files changed, 62 insertions, 448 deletions
diff --git a/docs/explanation/architecture.md b/docs/explanation/architecture.md index e0a57e5..09737df 100644 --- a/docs/explanation/architecture.md +++ b/docs/explanation/architecture.md | |||
| @@ -2,53 +2,7 @@ | |||
| 2 | 2 | ||
| 3 | ## Executive Summary | 3 | ## Executive Summary |
| 4 | 4 | ||
| 5 | `ngit-grasp` implements the GRASP protocol in Rust with **inline authorization** rather than Git hooks. The key architectural insight is that we can intercept and validate Git push operations at the HTTP handler level before reaching the Git repository, eliminating the need for pre-receive hooks. | 5 | `ngit-grasp` implements the GRASP protocol in Rust with **inline authorization** rather than Git hooks. Git push operations are intercepted and validated at the HTTP handler level before reaching the Git repository, eliminating the need for pre-receive hooks. |
| 6 | |||
| 7 | ## Architectural Decision: Inline vs. Hook-Based Authorization | ||
| 8 | |||
| 9 | ### Investigation Summary | ||
| 10 | |||
| 11 | After examining both the reference implementation and HTTP server options, we have two options: | ||
| 12 | |||
| 13 | #### Option 1: Hook-Based (Reference Implementation Approach) | ||
| 14 | |||
| 15 | - Use standard Git HTTP backend | ||
| 16 | - Create pre-receive and post-receive hooks | ||
| 17 | - Hooks query the Nostr relay and validate pushes | ||
| 18 | - **Pros**: Follows reference implementation closely | ||
| 19 | - **Cons**: Requires hook management, harder to test, less Rust-native | ||
| 20 | |||
| 21 | #### Option 2: Inline Authorization (Recommended) | ||
| 22 | |||
| 23 | - Intercept Git receive-pack requests in the HTTP handler | ||
| 24 | - Validate against Nostr state before spawning Git process | ||
| 25 | - Only forward valid pushes to Git | ||
| 26 | - **Pros**: Better error handling, easier testing, pure Rust, simpler deployment | ||
| 27 | - **Cons**: Requires custom Git protocol handling | ||
| 28 | |||
| 29 | ### Decision: Inline Authorization (Option 2) | ||
| 30 | |||
| 31 | **Rationale:** | ||
| 32 | |||
| 33 | 1. **Full control over HTTP layer**: Using Hyper directly gives us complete control over request handling, WebSocket upgrades, and CORS headers. | ||
| 34 | |||
| 35 | 2. **Better Developer Experience**: | ||
| 36 | |||
| 37 | - Validation errors can be returned as proper HTTP responses | ||
| 38 | - No need to parse hook stderr output | ||
| 39 | - Shared state between Git and Nostr components | ||
| 40 | - Pure Rust testing without shell scripts | ||
| 41 | |||
| 42 | 3. **Simpler Deployment**: | ||
| 43 | |||
| 44 | - Single binary | ||
| 45 | - No hook symlinks or permissions to manage | ||
| 46 | - No multi-process coordination | ||
| 47 | |||
| 48 | 4. **Performance**: | ||
| 49 | - Can parse incoming pack data once | ||
| 50 | - Avoid process spawn overhead for invalid pushes | ||
| 51 | - Better async integration | ||
| 52 | 6 | ||
| 53 | ## System Architecture | 7 | ## System Architecture |
| 54 | 8 | ||
| @@ -257,57 +211,73 @@ State events undergo authorization checks at multiple points: | |||
| 257 | 211 | ||
| 258 | ### 5. Purgatory System ([`src/purgatory/`](../../src/purgatory/)) | 212 | ### 5. Purgatory System ([`src/purgatory/`](../../src/purgatory/)) |
| 259 | 213 | ||
| 260 | The purgatory system solves the "which arrives first?" problem where either nostr events or git pushes can arrive in any order. It provides an in-memory holding area for events and git data awaiting their counterparts. | 214 | The purgatory system solves two related problems: |
| 215 | |||
| 216 | 1. **"Which arrives first?"** — Either nostr events or git pushes can arrive in any order. Purgatory holds events awaiting their git data counterparts. | ||
| 217 | 2. **Misleading empty repository announcements** — New announcements are held in purgatory until git data arrives, ensuring clients are never served announcements for repos with no content. | ||
| 261 | 218 | ||
| 262 | **Design Document**: See [`purgatory-design.md`](purgatory-design.md) for complete design specifications. | 219 | **Design Document**: See [`purgatory-design.md`](purgatory-design.md) for complete design specifications. |
| 263 | 220 | ||
| 264 | #### Architecture | 221 | #### Architecture |
| 265 | 222 | ||
| 266 | ```rust | 223 | ```rust |
| 267 | /// Main purgatory structure with two separate stores | 224 | /// Main purgatory structure with separate stores per event type |
| 268 | pub struct Purgatory { | 225 | pub struct Purgatory { |
| 226 | /// Announcement events (kind 30617) indexed by (owner, identifier) | ||
| 227 | /// Held until git data proves content exists | ||
| 228 | announcement_purgatory: DashMap<(PublicKey, String), AnnouncementPurgatoryEntry>, | ||
| 229 | |||
| 269 | /// State events (kind 30618) indexed by repository identifier | 230 | /// State events (kind 30618) indexed by repository identifier |
| 270 | state_events: Arc<DashMap<String, Vec<StatePurgatoryEntry>>>, | 231 | state_events: DashMap<String, Vec<StatePurgatoryEntry>>, |
| 271 | 232 | ||
| 272 | /// PR events (kind 1617/1618) or placeholders indexed by event ID | 233 | /// PR events (kind 1617/1618) or placeholders indexed by event ID |
| 273 | pr_events: Arc<DashMap<String, PrPurgatoryEntry>>, | 234 | pr_events: DashMap<String, PrPurgatoryEntry>, |
| 274 | } | 235 | } |
| 275 | ``` | 236 | ``` |
| 276 | 237 | ||
| 277 | **Key Design Principles:** | 238 | **Key Design Principles:** |
| 278 | 239 | ||
| 279 | 1. **Separate Storage**: State events and PR events use different indexing strategies | 240 | 1. **Separate Storage**: Each event type uses a different indexing strategy |
| 241 | - Announcements: Indexed by `(pubkey, identifier)` (unique per owner) | ||
| 280 | - State events: Indexed by `identifier` (multiple events can wait for same repo) | 242 | - State events: Indexed by `identifier` (multiple events can wait for same repo) |
| 281 | - PR events: Indexed by `event_id` (one-to-one mapping) | 243 | - PR events: Indexed by `event_id` (one-to-one mapping) |
| 282 | 244 | ||
| 283 | 2. **Late Binding**: State event refs are extracted at git push time, not event arrival | 245 | 2. **Announcement Purgatory**: New announcements are held until git data arrives |
| 246 | - Bare repo created immediately so pushes can succeed | ||
| 247 | - Announcement promoted to database only when git data proves content exists | ||
| 248 | - Two-phase soft expiry: bare repo deleted at 30 min, event retained 24h for revival | ||
| 249 | |||
| 250 | 3. **Late Binding**: State event refs are extracted at git push time, not event arrival | ||
| 284 | - Enables flexible matching when pushes arrive out-of-order | 251 | - Enables flexible matching when pushes arrive out-of-order |
| 285 | - Helper functions in [`helpers.rs`](../../src/purgatory/helpers.rs) handle ref extraction | 252 | - Helper functions in [`helpers.rs`](../../src/purgatory/helpers.rs) handle ref extraction |
| 286 | 253 | ||
| 287 | 3. **Bidirectional Waiting**: Either side can arrive first | 254 | 4. **Bidirectional Waiting**: Either side can arrive first |
| 288 | - **Event-first**: Event waits for git push | 255 | - **Event-first**: Event waits for git push |
| 289 | - **Git-first**: Placeholder created, waits for event | 256 | - **Git-first**: Placeholder created, waits for event |
| 290 | 257 | ||
| 291 | 4. **Automatic Expiry**: 30-minute default expiry, extensible during processing | 258 | 5. **Automatic Expiry**: 30-minute default expiry, extensible during processing |
| 292 | - Background cleanup task runs every 60 seconds | 259 | - Background cleanup task runs every 60 seconds |
| 293 | - Removes expired entries from both stores | 260 | - Removes expired entries from all stores |
| 294 | 261 | ||
| 295 | #### Data Types | 262 | #### Data Types |
| 296 | 263 | ||
| 297 | See [`types.rs`](../../src/purgatory/types.rs) for complete definitions: | 264 | See [`types.rs`](../../src/purgatory/types.rs) for complete definitions: |
| 298 | 265 | ||
| 299 | - **[`RefPair`](../../src/purgatory/types.rs:16)**: Ref name + object SHA pair | 266 | - **[`RefPair`](../../src/purgatory/types.rs:16)**: Ref name + object SHA pair |
| 267 | - **[`AnnouncementPurgatoryEntry`](../../src/purgatory/types.rs)**: Announcement with bare repo path, relays, and expiry | ||
| 300 | - **[`StatePurgatoryEntry`](../../src/purgatory/types.rs:29)**: State event with metadata | 268 | - **[`StatePurgatoryEntry`](../../src/purgatory/types.rs:29)**: State event with metadata |
| 301 | - **[`PrPurgatoryEntry`](../../src/purgatory/types.rs:52)**: PR event or placeholder with metadata | 269 | - **[`PrPurgatoryEntry`](../../src/purgatory/types.rs:52)**: PR event or placeholder with metadata |
| 302 | 270 | ||
| 303 | #### Integration Points | 271 | #### Integration Points |
| 304 | 272 | ||
| 305 | **Write Policy** ([`src/nostr/policy/`](../../src/nostr/policy/)): | 273 | **Write Policy** ([`src/nostr/policy/`](../../src/nostr/policy/)): |
| 306 | - State policy checks git data existence before adding to purgatory | 274 | - Announcement policy routes new announcements to purgatory; replacements accepted immediately |
| 275 | - State policy checks git data existence before adding to purgatory; checks purgatory announcements for authorization | ||
| 307 | - PR policy checks for placeholders before adding to purgatory | 276 | - PR policy checks for placeholders before adding to purgatory |
| 308 | - Events return "purgatory: will not be served until git data arrives" message | 277 | - Events return "purgatory: will not be served until git data arrives" message |
| 309 | 278 | ||
| 310 | **Git Handlers** ([`src/git/handlers.rs`](../../src/git/handlers.rs)): | 279 | **Git Handlers** ([`src/git/handlers.rs`](../../src/git/handlers.rs)): |
| 280 | - On git push: Promote announcement from purgatory to database if present | ||
| 311 | - On git push: Check purgatory for matching state events | 281 | - On git push: Check purgatory for matching state events |
| 312 | - On refs/nostr/* push: Check purgatory for PR events or create placeholders | 282 | - On refs/nostr/* push: Check purgatory for PR events or create placeholders |
| 313 | - Release events from purgatory when git data arrives | 283 | - Release events from purgatory when git data arrives |
| @@ -392,11 +362,25 @@ Configuration is loaded via **clap CLI > environment variables > .env > defaults | |||
| 392 | └─ Accept or reject | 362 | └─ Accept or reject |
| 393 | ↓ | 363 | ↓ |
| 394 | 4. If ACCEPTED: | 364 | 4. If ACCEPTED: |
| 395 | ├─ Event saved to database | 365 | ├─ Is there an active announcement for (pubkey, identifier) in DB? |
| 396 | └─ ensure_bare_repository() called | 366 | │ ├─ YES → Accept immediately (replacement, repo already proven) |
| 367 | │ └─ NO → Route to announcement purgatory | ||
| 397 | ↓ | 368 | ↓ |
| 398 | 5. Bare Git repository created at | 369 | 5. Announcement Purgatory path: |
| 399 | <git_data_path>/<npub>/<identifier>.git | 370 | ├─ Bare Git repository created immediately at |
| 371 | │ <git_data_path>/<npub>/<identifier>.git | ||
| 372 | ├─ Announcement held in purgatory (not served to clients) | ||
| 373 | └─ Awaiting git data to prove content exists | ||
| 374 | ↓ | ||
| 375 | 6. When git data arrives (push or background sync): | ||
| 376 | ├─ Announcement promoted from purgatory to database | ||
| 377 | ├─ Event now served to clients | ||
| 378 | └─ SyncManager upgrades to Full sync level | ||
| 379 | ↓ | ||
| 380 | 7. If no git data within 30 minutes: | ||
| 381 | ├─ Bare repo deleted (soft expiry) | ||
| 382 | ├─ Event retained 24h for potential revival | ||
| 383 | └─ Eventually discarded if no git data arrives | ||
| 400 | ``` | 384 | ``` |
| 401 | 385 | ||
| 402 | ### State Event Flow | 386 | ### State Event Flow |
| @@ -407,14 +391,25 @@ Configuration is loaded via **clap CLI > environment variables > .env > defaults | |||
| 407 | 2. Nostr relay receives event | 391 | 2. Nostr relay receives event |
| 408 | ↓ | 392 | ↓ |
| 409 | 3. Nip34WritePolicy::admit_event() | 393 | 3. Nip34WritePolicy::admit_event() |
| 410 | ├─ Check author is in maintainer set | 394 | ├─ Check author is in maintainer set (DB + purgatory announcements) |
| 411 | ├─ Validate state structure | 395 | ├─ Validate state structure |
| 412 | └─ Accept or reject | 396 | └─ Accept or reject |
| 413 | ↓ | 397 | ↓ |
| 414 | 4. If ACCEPTED and is latest state: | 398 | 4. If ACCEPTED: |
| 415 | ├─ Align repository refs to match state | 399 | ├─ Does git data already exist for this state? |
| 416 | ├─ Create/update/delete refs as needed | 400 | │ ├─ YES → Save to database immediately |
| 417 | └─ Set HEAD if commit available | 401 | │ └─ NO → Add to state purgatory |
| 402 | ↓ | ||
| 403 | 5. State Purgatory path: | ||
| 404 | ├─ Event held in purgatory (not served to clients) | ||
| 405 | ├─ Enqueued for background git data sync (3 min delay) | ||
| 406 | └─ Awaiting git push or background sync | ||
| 407 | ↓ | ||
| 408 | 6. When git push arrives: | ||
| 409 | ├─ Authorization checks both database AND purgatory | ||
| 410 | ├─ If authorized via purgatory state: push proceeds | ||
| 411 | ├─ After successful push: state event saved to database | ||
| 412 | └─ Removed from purgatory | ||
| 418 | ``` | 413 | ``` |
| 419 | 414 | ||
| 420 | ## Testing Strategy | 415 | ## Testing Strategy |
| @@ -613,9 +608,7 @@ WantedBy=multi-user.target | |||
| 613 | 608 | ||
| 614 | ## Conclusion | 609 | ## Conclusion |
| 615 | 610 | ||
| 616 | The inline authorization approach provides a cleaner, more maintainable architecture than hook-based authorization while maintaining full GRASP-01 compliance. Using Hyper for the HTTP layer gives us complete control over request handling, WebSocket upgrades, and CORS headers. | 611 | ngit-grasp uses inline authorization at the HTTP handler level, giving full control over request handling, WebSocket upgrades, and CORS headers while maintaining full GRASP-01 compliance. The purgatory system ensures that only repositories with actual git content are served to clients, and that events and git data are always consistent when released to the database. |
| 617 | |||
| 618 | The key insight is that we don't need to rely on Git's hook mechanism when we have full control over the HTTP layer that Git operates through. By intercepting at the HTTP handler level, we gain better error handling, easier testing, and tighter integration between the Git and Nostr components. | ||
| 619 | 612 | ||
| 620 | ## Related Documentation | 613 | ## Related Documentation |
| 621 | 614 | ||
diff --git a/docs/explanation/comparison.md b/docs/explanation/comparison.md deleted file mode 100644 index 315f091..0000000 --- a/docs/explanation/comparison.md +++ /dev/null | |||
| @@ -1,379 +0,0 @@ | |||
| 1 | # ngit-grasp vs ngit-relay Comparison | ||
| 2 | |||
| 3 | This document compares ngit-grasp (this project) with ngit-relay (the reference implementation) based on their actual implementations. | ||
| 4 | |||
| 5 | ## High-Level Overview | ||
| 6 | |||
| 7 | | Aspect | ngit-relay (Reference) | ngit-grasp (This Project) | | ||
| 8 | |--------|------------------------|---------------------------| | ||
| 9 | | **Language** | Go | Rust | | ||
| 10 | | **Architecture** | Multi-process (nginx + fcgiwrap + khatru + sync daemon) | Single integrated process | | ||
| 11 | | **Git Protocol** | git-http-backend (C via fcgiwrap) | HTTP layer in Rust + git subprocess | | ||
| 12 | | **Authorization** | Pre-receive Git hook | Inline HTTP handler validation | | ||
| 13 | | **Nostr Relay** | Khatru (Go library) | nostr-relay-builder (Rust library) | | ||
| 14 | | **Event Store** | Badger (Go KV database) | LMDB (Rust) | | ||
| 15 | | **Proactive Sync** | Git-only (polls DB + fetches from git servers) | Nostr event sync + git sync (event-driven) | | ||
| 16 | | **Process Management** | supervisord (4 processes) | Single tokio runtime | | ||
| 17 | | **Packaging** | Docker with supervisord | Single static binary or Docker | | ||
| 18 | | **Configuration** | Environment variables | Environment variables + CLI flags | | ||
| 19 | | **Total Code** | ~1,866 lines of Go | ~25,000 lines of Rust | | ||
| 20 | |||
| 21 | ## Architecture Comparison | ||
| 22 | |||
| 23 | ### ngit-relay (Multi-Process) | ||
| 24 | |||
| 25 | ``` | ||
| 26 | ┌──────────────── Docker Container ────────────────┐ | ||
| 27 | │ │ | ||
| 28 | │ ┌─────────────────────────────────────────────┐ │ | ||
| 29 | │ │ supervisord │ │ | ||
| 30 | │ │ - fcgiwrap (git-http-backend wrapper) │ │ | ||
| 31 | │ │ - nginx (HTTP + reverse proxy) │ │ | ||
| 32 | │ │ - ngit-relay-khatru (Nostr relay) │ │ | ||
| 33 | │ │ - ngit-relay-proactive-sync (sync daemon) │ │ | ||
| 34 | │ └─────────────────────────────────────────────┘ │ | ||
| 35 | │ │ | ||
| 36 | │ ┌──────────┐ ┌────────────────────┐ │ | ||
| 37 | │ │ nginx │────────▶│ git-http-backend │ │ | ||
| 38 | │ │ :80 │ │ (C binary via CGI) │ │ | ||
| 39 | │ └──────┬───┘ └──────────┬─────────┘ │ | ||
| 40 | │ │ │ │ | ||
| 41 | │ │ ▼ │ | ||
| 42 | │ │ ┌──────────────────┐ │ | ||
| 43 | │ │ │ Git Repos │ │ | ||
| 44 | │ │ │ + pre-receive │ │ | ||
| 45 | │ │ │ hook (Go) │ │ | ||
| 46 | │ │ └────────┬─────────┘ │ | ||
| 47 | │ │ │ WebSocket │ | ||
| 48 | │ │ │ query │ | ||
| 49 | │ │ ▼ │ | ||
| 50 | │ │ ┌──────────────────┐ │ | ||
| 51 | │ └──────────────▶│ Khatru Relay │ │ | ||
| 52 | │ │ :3334 │ │ | ||
| 53 | │ │ (Badger DB) │ │ | ||
| 54 | │ └──────────────────┘ │ | ||
| 55 | │ │ | ||
| 56 | │ Separate sync daemon polls relay DB │ | ||
| 57 | │ and fetches from remote git servers │ | ||
| 58 | │ │ | ||
| 59 | └───────────────────────────────────────────────────┘ | ||
| 60 | ``` | ||
| 61 | |||
| 62 | ### ngit-grasp (Single Process) | ||
| 63 | |||
| 64 | ``` | ||
| 65 | ┌────────────── ngit-grasp (Single Binary) ─────────────┐ | ||
| 66 | │ │ | ||
| 67 | │ ┌──────────────────────────────────────────────────┐ │ | ||
| 68 | │ │ hyper HTTP Server (:7334) │ │ | ||
| 69 | │ │ - WebSocket upgrade for Nostr relay │ │ | ||
| 70 | │ │ - Git Smart HTTP handlers │ │ | ||
| 71 | │ │ - Landing page + metrics endpoint │ │ | ||
| 72 | │ └───────┬──────────────────────┬───────────────────┘ │ | ||
| 73 | │ │ │ │ | ||
| 74 | │ ▼ ▼ │ | ||
| 75 | │ ┌──────────────┐ ┌────────────────────┐ │ | ||
| 76 | │ │ Git Handlers │ │ Nostr Relay │ │ | ||
| 77 | │ │ (HTTP layer) │ │ (nostr-relay- │ │ | ||
| 78 | │ │ │ │ builder library) │ │ | ||
| 79 | │ │ - info/refs │ │ - NIP-34 Policy │ │ | ||
| 80 | │ │ - upload-pk │◀─────┤ (inline query) │ │ | ||
| 81 | │ │ - receive-pk │ auth │ - LMDB/Memory │ │ | ||
| 82 | │ │ + inline │ check│ - WebSocket │ │ | ||
| 83 | │ │ validation │ │ - NIP-11 endpoint │ │ | ||
| 84 | │ └──────┬───────┘ └──────────┬─────────┘ │ | ||
| 85 | │ │ │ │ | ||
| 86 | │ ▼ ▼ │ | ||
| 87 | │ ┌──────────────┐ ┌────────────────────┐ │ | ||
| 88 | │ │ git binary │ │ Purgatory │ │ | ||
| 89 | │ │ upload-pack │ │ (in-memory queue) │ │ | ||
| 90 | │ │ receive-pk │ │ + sync loop │ │ | ||
| 91 | │ └──────────────┘ └────────────────────┘ │ | ||
| 92 | │ │ | ||
| 93 | │ ┌──────────────────────────────────────────────────┐ │ | ||
| 94 | │ │ SyncManager (tokio background task) │ │ | ||
| 95 | │ │ - Multi-relay Nostr event sync (GRASP-02) │ │ | ||
| 96 | │ │ - Negentropy + REQ/EOSE support │ │ | ||
| 97 | │ │ - Health tracking & exponential backoff │ │ | ||
| 98 | │ │ - Git fetch from remote servers (via purgatory) │ │ | ||
| 99 | │ └──────────────────────────────────────────────────┘ │ | ||
| 100 | │ │ | ||
| 101 | │ ┌──────────────────────────────────────────────────┐ │ | ||
| 102 | │ │ Shared State (Arc<T>) │ │ | ||
| 103 | │ │ - Database (LMDB/Memory) │ │ | ||
| 104 | │ │ - Purgatory (DashMap - concurrent queue) │ │ | ||
| 105 | │ │ - Metrics (Prometheus) │ │ | ||
| 106 | │ └──────────────────────────────────────────────────┘ │ | ||
| 107 | │ │ | ||
| 108 | └────────────────────────────────────────────────────────┘ | ||
| 109 | ``` | ||
| 110 | |||
| 111 | ## Feature Comparison | ||
| 112 | |||
| 113 | ### Key Architectural Difference: Nostr Event Sync | ||
| 114 | |||
| 115 | **The biggest difference between the two implementations is how they handle Nostr events:** | ||
| 116 | |||
| 117 | | Aspect | ngit-relay | ngit-grasp | | ||
| 118 | |--------|-----------|-----------| | ||
| 119 | | **Event Arrival** | Relies on clients to push events directly | Proactively syncs events from other relays | | ||
| 120 | | **Discovery** | None - only stores what clients send | Discovers events from relay network | | ||
| 121 | | **Coordination** | Events and git data handled separately | Purgatory coordinates events + git data | | ||
| 122 | | **Completeness** | May miss events if clients don't push to this relay | Actively fetches missing events from network | | ||
| 123 | | **Implementation** | No event sync code (~0 lines) | Full multi-relay sync system (~5,000 lines) | | ||
| 124 | |||
| 125 | **Example scenario:** | ||
| 126 | - User creates PR on relay A, pushes git data to server B | ||
| 127 | - **ngit-relay**: Only knows about events/data pushed directly to it | ||
| 128 | - **ngit-grasp**: Discovers PR event from relay A, fetches git data from server B | ||
| 129 | |||
| 130 | This is why ngit-grasp has ~13x more code - the majority is implementing GRASP-02 proactive event sync. | ||
| 131 | |||
| 132 | ### Git Protocol Implementation | ||
| 133 | |||
| 134 | | Feature | ngit-relay | ngit-grasp | | ||
| 135 | |---------|-----------|-----------| | ||
| 136 | | **HTTP Server** | nginx | hyper (Rust) | | ||
| 137 | | **Git Backend** | git-http-backend (C) via fcgiwrap | HTTP protocol layer (Rust) + git binary | | ||
| 138 | | **Process Model** | FastCGI spawns git-http-backend | HTTP handler spawns git subprocess | | ||
| 139 | | **Upload Pack** | C binary passthrough | Rust parses HTTP → spawns `git upload-pack` | | ||
| 140 | | **Receive Pack** | C binary → pre-receive hook | Rust validates → spawns `git receive-pack` | | ||
| 141 | | **Authorization** | Go hook queries relay via WebSocket | In-process function call before git spawn | | ||
| 142 | | **Error Reporting** | Hook stderr → git client | HTTP response body (before git runs) | | ||
| 143 | | **CORS** | nginx config | hyper middleware | | ||
| 144 | | **Lines of Code** | ~0 (uses C binary) + hook ~135 | ~1,000+ (HTTP protocol layer) | | ||
| 145 | |||
| 146 | ### Authorization Logic | ||
| 147 | |||
| 148 | | Feature | ngit-relay | ngit-grasp | | ||
| 149 | |---------|-----------|-----------| | ||
| 150 | | **Location** | pre-receive hook (separate Go binary) | Inline HTTP handler (Rust) | | ||
| 151 | | **Trigger** | Git invokes hook during push | HTTP handler before spawning git | | ||
| 152 | | **State Query** | WebSocket to localhost:3334 | Direct database query (in-process) | | ||
| 153 | | **Latency** | +50-100ms (hook spawn + WS query) | +10-20ms (function call) | | ||
| 154 | | **Error Channel** | stderr → git client | HTTP 403 response | | ||
| 155 | | **Ref Parsing** | Read from stdin (hook protocol) | Parse from HTTP request body | | ||
| 156 | | **Maintainer Resolution** | Recursive Go function | Recursive Rust function (similar) | | ||
| 157 | | **State Caching** | None (queries relay per push) | Purgatory tracks pending events | | ||
| 158 | |||
| 159 | ### Nostr Relay | ||
| 160 | |||
| 161 | | Feature | ngit-relay | ngit-grasp | | ||
| 162 | |---------|-----------|-----------| | ||
| 163 | | **Implementation** | Khatru (Go library) | nostr-relay-builder (Rust library) | | ||
| 164 | | **Database** | Badger (Go KV store) | LMDB (Rust) | | ||
| 165 | | **Process** | Separate process on :3334 | Integrated (same binary) | | ||
| 166 | | **Policies** | Go functions in `policies.go` | Rust traits (modular sub-policies) | | ||
| 167 | | **Event Validation** | Single function with branches | 4 separate policy modules | | ||
| 168 | | **WebSocket** | Khatru built-in | nostr-relay-builder + hyper | | ||
| 169 | | **NIP-11** | Manual JSON in code | Built-in support from library | | ||
| 170 | | **Connection** | Separate from HTTP | Shared hyper server | | ||
| 171 | | **Lines of Code** | ~186 (policies.go) + Khatru library | ~3,000+ (policy modules) + nostr-relay-builder library | | ||
| 172 | |||
| 173 | ### Proactive Sync | ||
| 174 | |||
| 175 | | Feature | ngit-relay | ngit-grasp | | ||
| 176 | |---------|-----------|-----------| | ||
| 177 | | **Architecture** | Separate daemon (`ngit-relay-proactive-sync`) | Integrated SyncManager (tokio task) | | ||
| 178 | | **Nostr Event Sync** | ❌ None (relies on client pushes) | ✅ Multi-relay sync with negentropy/REQ | | ||
| 179 | | **Git Data Sync** | ✅ Polls local DB + fetches from git servers | ✅ Event-driven via purgatory queue | | ||
| 180 | | **Sync Trigger** | Timer (every 15 minutes) | Immediate on event arrival + timer for retries | | ||
| 181 | | **Relay Discovery** | N/A (no event sync) | Dynamic from 30617 announcement events | | ||
| 182 | | **Protocol** | Git fetch only | Nostr WebSocket + git fetch | | ||
| 183 | | **Concurrency** | Goroutines (per-repo iteration) | Tokio async tasks (per-relay connections) | | ||
| 184 | | **Health Tracking** | Basic retry on git fetch failures | RelayHealthTracker with exponential backoff | | ||
| 185 | | **Connection Management** | N/A (no Nostr connections) | Persistent connections with reconnect | | ||
| 186 | | **Coordination** | Separate process | Purgatory + SyncManager coordination | | ||
| 187 | | **Lines of Code** | ~112 (main.go) + ~305 (git sync) | ~5,000+ (Nostr sync + git sync + coordination) | | ||
| 188 | |||
| 189 | ### Repository Management | ||
| 190 | |||
| 191 | | Feature | ngit-relay | ngit-grasp | | ||
| 192 | |---------|-----------|-----------| | ||
| 193 | | **Creation** | Event hook → shell commands | Event hook → tokio::process | | ||
| 194 | | **Trigger** | `EventReceiveHook()` in Go | `handle_announcement()` in Rust | | ||
| 195 | | **Configuration** | `git config` via shell | `git config` via tokio::process | | ||
| 196 | | **Hook Installation** | Symlinks to pre-receive/post-receive | Not needed (inline auth) | | ||
| 197 | | **Permissions** | `chown nginx:nginx` | tokio::fs permissions | | ||
| 198 | | **Path Structure** | `<npub>/<id>.git` | `<npub>/<id>.git` (same) | | ||
| 199 | |||
| 200 | ### Event Coordination (Purgatory) | ||
| 201 | |||
| 202 | | Feature | ngit-relay | ngit-grasp | | ||
| 203 | |---------|-----------|-----------| | ||
| 204 | | **Implementation** | None | Dedicated Purgatory system | | ||
| 205 | | **Purpose** | N/A | Solves "which arrives first?" problem | | ||
| 206 | | **Storage** | N/A | In-memory DashMap (thread-safe) | | ||
| 207 | | **Expiry** | N/A | 30 minutes default TTL | | ||
| 208 | | **State Events** | Accepted (git sync happens later via timer) | Queued until git data arrives | | ||
| 209 | | **PR Events** | Accepted (references may be missing) | Queued with placeholder refs | | ||
| 210 | | **Sync Queue** | Timer-based (polls all repos) | Event-driven (only syncs needed repos) | | ||
| 211 | | **Cleanup** | N/A | Background task (60s interval) | | ||
| 212 | | **Lines of Code** | 0 | ~2,000+ | | ||
| 213 | |||
| 214 | **Impact**: ngit-relay accepts all events and relies on periodic sync to eventually fetch git data. ngit-grasp holds events in purgatory and triggers targeted syncs, providing faster convergence and better coordination between Nostr events and git data. | ||
| 215 | |||
| 216 | ### Deployment & Operations | ||
| 217 | |||
| 218 | | Feature | ngit-relay | ngit-grasp | | ||
| 219 | |---------|-----------|-----------| | ||
| 220 | | **Dependencies** | nginx, git, fcgiwrap, supervisord, Go runtime | git, Rust binary (statically linked) | | ||
| 221 | | **Process Count** | 4 (supervisord + nginx + khatru + sync) | 1 (single tokio runtime) | | ||
| 222 | | **Configuration** | `.env` file | `.env` + CLI flags (clap) | | ||
| 223 | | **Docker Image Size** | ~500MB (Alpine + tools + Go runtime) | ~100MB (Debian slim + git + binary) | | ||
| 224 | | **Startup Time** | ~2-5 seconds (multiple processes) | ~0.5 seconds (single process) | | ||
| 225 | | **Memory (Idle)** | ~150-200MB (4 processes + Go GC) | ~50-100MB (single process, no GC) | | ||
| 226 | | **Logs** | supervisord → stdout (4 streams) | tracing → stdout (unified) | | ||
| 227 | | **Monitoring** | None built-in | Prometheus metrics endpoint | | ||
| 228 | | **Binary Distribution** | Docker only | Native binary + Docker | | ||
| 229 | |||
| 230 | ### Development Experience | ||
| 231 | |||
| 232 | | Feature | ngit-relay | ngit-grasp | | ||
| 233 | |---------|-----------|-----------| | ||
| 234 | | **Build Time** | Fast (~5s incremental, Go) | Slow first build (~5min), fast incremental | | ||
| 235 | | **Type Safety** | Good (Go interfaces) | Excellent (Rust traits + ownership) | | ||
| 236 | | **Testing** | Go tests + shell scripts | Rust unit + integration tests | | ||
| 237 | | **Test Relay** | Manual Docker setup | `TestRelay` fixture (auto-start binary) | | ||
| 238 | | **Debugging** | Multi-process (harder) | Single process (easier) | | ||
| 239 | | **IDE Support** | Good (gopls) | Excellent (rust-analyzer) | | ||
| 240 | | **Async Model** | Goroutines (simple) | Tokio (more complex) | | ||
| 241 | | **Error Handling** | `error` interface + if checks | Result<T, E> + `?` operator | | ||
| 242 | | **Dependencies** | Go modules | Cargo crates (larger ecosystem) | | ||
| 243 | |||
| 244 | ### Code Complexity | ||
| 245 | |||
| 246 | | Component | ngit-relay | ngit-grasp | Notes | | ||
| 247 | |-----------|-----------|-----------|-------| | ||
| 248 | | Main server | 129 | 196 | ngit-relay uses supervisord | | ||
| 249 | | Git HTTP protocol | 0 (C binary via fcgiwrap) | ~1,000 | ngit-grasp implements HTTP layer | | ||
| 250 | | Auth logic (hooks) | 135 + 52 | 0 | ngit-grasp inline, no hooks | | ||
| 251 | | Auth logic (inline) | 0 | ~800 | ngit-grasp authorization module | | ||
| 252 | | Nostr relay policies | 186 | ~3,000 | Both use libraries (Khatru vs nostr-relay-builder) | | ||
| 253 | | Git-only proactive sync | 112 + 305 | 0 | ngit-relay git sync only | | ||
| 254 | | Nostr event proactive sync | 0 | ~5,000 | ngit-grasp adds full event sync (GRASP-02 v4) | | ||
| 255 | | Purgatory coordination | 0 | ~2,000 | ngit-grasp event/git coordination | | ||
| 256 | | Shared utils | 241 + 132 | ~4,000 | ngit-grasp more comprehensive | | ||
| 257 | | Config | ~50 | ~400 | ngit-grasp CLI + validation | | ||
| 258 | | Metrics | 0 | ~1,500 | ngit-grasp Prometheus | | ||
| 259 | | **Total** | **~1,866** | **~25,000** | ngit-grasp 13x more code | | ||
| 260 | |||
| 261 | **Why the difference?** | ||
| 262 | - **Nostr event sync**: ngit-relay has NONE, ngit-grasp implements full multi-relay event sync (~5,000 lines) | ||
| 263 | - **Git HTTP protocol**: ngit-relay uses C binary, ngit-grasp implements HTTP layer (~1,000 lines) | ||
| 264 | - **Purgatory coordination**: ngit-grasp adds event/git coordination system (~2,000 lines) | ||
| 265 | - **Metrics & observability**: ngit-grasp includes comprehensive monitoring (~1,500 lines) | ||
| 266 | - Both use relay libraries (Khatru vs nostr-relay-builder), but ngit-grasp has more modular policies | ||
| 267 | |||
| 268 | ### Performance Characteristics (Estimated) | ||
| 269 | |||
| 270 | | Metric | ngit-relay | ngit-grasp | Notes | | ||
| 271 | |--------|-----------|-----------|-------| | ||
| 272 | | **Startup** | ~2-5s | ~0.5s | Single process vs multi-process | | ||
| 273 | | **Memory (Idle)** | ~150MB | ~75MB | No GC, single process | | ||
| 274 | | **Memory (Active)** | ~200MB+ | ~100-150MB | Depends on event volume | | ||
| 275 | | **CPU (Idle)** | ~1-2% | ~0.5% | Fewer processes | | ||
| 276 | | **Push Latency** | +50-100ms | +10-20ms | No hook spawn overhead | | ||
| 277 | | **Clone Latency** | ~same | ~same | Both passthrough to git | | ||
| 278 | | **Concurrent Pushes** | Good (goroutines) | Excellent (tokio async) | | ||
| 279 | | **Event Ingestion** | Good (Badger) | Excellent (LMDB zero-copy) | | ||
| 280 | | **Sync Throughput** | Moderate (polling) | High (negentropy + async) | | ||
| 281 | |||
| 282 | *These are estimates based on architecture. Actual performance depends on workload.* | ||
| 283 | |||
| 284 | ## Migration Path | ||
| 285 | |||
| 286 | For users of ngit-relay, migration to ngit-grasp involves: | ||
| 287 | |||
| 288 | ### Data Migration | ||
| 289 | |||
| 290 | 1. **Events**: Export from Badger → Import to LMDB | ||
| 291 | - No direct migration tool yet (would need to be built) | ||
| 292 | - Alternative: Use proactive sync to re-fetch from other relays | ||
| 293 | 2. **Git Repositories**: Direct copy (same structure) | ||
| 294 | ```bash | ||
| 295 | cp -r /srv/ngit-relay/repos/* /path/to/ngit-grasp/data/git/ | ||
| 296 | ``` | ||
| 297 | 3. **Configuration**: Translate environment variables | ||
| 298 | - Most variables are compatible (`NGIT_DOMAIN`, etc.) | ||
| 299 | - Remove nginx/supervisord-specific configs | ||
| 300 | |||
| 301 | ### Compatibility | ||
| 302 | |||
| 303 | - **Git Data**: 100% compatible (same repository structure) | ||
| 304 | - **Nostr Events**: 100% compatible (standard NIP-34) | ||
| 305 | - **HTTP URLs**: Compatible (same path structure) | ||
| 306 | - **Git Hooks**: ngit-grasp doesn't use hooks (inline auth instead) | ||
| 307 | |||
| 308 | ### Downtime | ||
| 309 | |||
| 310 | - Option 1: Run both in parallel (different domains), gradually migrate | ||
| 311 | - Option 2: Short downtime for data copy + config update | ||
| 312 | |||
| 313 | ## When to Choose Each | ||
| 314 | |||
| 315 | ### Choose ngit-relay (Reference) if: | ||
| 316 | |||
| 317 | - ✅ You need proven, production-tested code | ||
| 318 | - ✅ You're already familiar with Go ecosystem | ||
| 319 | - ✅ You prefer simple, minimal codebases (~1,866 lines) | ||
| 320 | - ✅ You trust battle-tested C binaries (git-http-backend) | ||
| 321 | - ✅ You want to stay close to the reference implementation | ||
| 322 | - ✅ You need to deploy immediately without complexity | ||
| 323 | - ✅ Your users will push events directly to your relay (no sync needed) | ||
| 324 | - ✅ You only need git data sync, not Nostr event sync | ||
| 325 | |||
| 326 | ### Choose ngit-grasp (This Project) if: | ||
| 327 | |||
| 328 | - ✅ **You need Nostr event sync from other relays** (the main differentiator) | ||
| 329 | - ✅ You want better performance and lower resource usage | ||
| 330 | - ✅ You prefer Rust's type safety and memory safety | ||
| 331 | - ✅ You want simpler deployment (single binary, no supervisord) | ||
| 332 | - ✅ You need event/git data coordination (purgatory) | ||
| 333 | - ✅ You want inline authorization (lower latency) | ||
| 334 | - ✅ You need comprehensive observability (Prometheus metrics) | ||
| 335 | - ✅ You're comfortable with more complex codebase (~25,000 lines) | ||
| 336 | - ✅ You want full GRASP-02 v4 multi-relay event discovery | ||
| 337 | |||
| 338 | ## Current Status | ||
| 339 | |||
| 340 | ### ngit-relay (Reference) | ||
| 341 | - ✅ GRASP-01 complete and production-ready | ||
| 342 | - ✅ Git data proactive sync (fetches from git servers) | ||
| 343 | - ❌ No Nostr event sync (relies on client pushes) | ||
| 344 | - ✅ Battle-tested in production | ||
| 345 | - 🔄 Community adoption growing | ||
| 346 | |||
| 347 | ### ngit-grasp (This Project) | ||
| 348 | - ✅ GRASP-01 complete with comprehensive testing | ||
| 349 | - ✅ GRASP-02 v4 multi-relay Nostr event sync with negentropy | ||
| 350 | - ✅ Git data proactive sync (via purgatory queue) | ||
| 351 | - ✅ Purgatory system for event/git coordination | ||
| 352 | - ✅ Prometheus metrics and health tracking | ||
| 353 | - ✅ NIP-77 negentropy support | ||
| 354 | - ✅ Full integration test suite | ||
| 355 | - 🔄 Production deployment validation ongoing | ||
| 356 | |||
| 357 | ## Conclusion | ||
| 358 | |||
| 359 | Both implementations are valid approaches to GRASP with different philosophies: | ||
| 360 | |||
| 361 | - **ngit-relay** prioritizes simplicity - clients push events, relay syncs git data (~1,866 lines) | ||
| 362 | - **ngit-grasp** prioritizes completeness - syncs both events and git data from network (~25,000 lines) | ||
| 363 | |||
| 364 | **The fundamental difference**: ngit-relay expects clients to push Nostr events to it. ngit-grasp proactively discovers and syncs events from other relays in the network. | ||
| 365 | |||
| 366 | The choice depends on your priorities: | ||
| 367 | |||
| 368 | | Priority | Recommendation | | ||
| 369 | |----------|---------------| | ||
| 370 | | **Simplicity** | ngit-relay | | ||
| 371 | | **Event Discovery** | ngit-grasp (syncs from network) | | ||
| 372 | | **Production Stability** | ngit-relay (more battle-tested) | | ||
| 373 | | **Event Completeness** | ngit-grasp (proactive sync) | | ||
| 374 | | **Low Resources** | ngit-grasp (single binary, lower memory) | | ||
| 375 | | **Quick Deploy** | ngit-relay (Docker Compose) | | ||
| 376 | | **Development** | ngit-grasp (better tooling, type safety) | | ||
| 377 | | **Network Resilience** | ngit-grasp (multi-relay sync) | | ||
| 378 | |||
| 379 | For deployments where **Nostr event sync** is important (discovering events from other relays), **ngit-grasp** is required. For simpler deployments where users will push events directly, **ngit-relay** is sufficient and battle-tested. | ||