diff options
Diffstat (limited to 'docs/explanation/architecture.md')
| -rw-r--r-- | docs/explanation/architecture.md | 131 |
1 files changed, 62 insertions, 69 deletions
diff --git a/docs/explanation/architecture.md b/docs/explanation/architecture.md index e0a57e5..09737df 100644 --- a/docs/explanation/architecture.md +++ b/docs/explanation/architecture.md | |||
| @@ -2,53 +2,7 @@ | |||
| 2 | 2 | ||
| 3 | ## Executive Summary | 3 | ## Executive Summary |
| 4 | 4 | ||
| 5 | `ngit-grasp` implements the GRASP protocol in Rust with **inline authorization** rather than Git hooks. The key architectural insight is that we can intercept and validate Git push operations at the HTTP handler level before reaching the Git repository, eliminating the need for pre-receive hooks. | 5 | `ngit-grasp` implements the GRASP protocol in Rust with **inline authorization** rather than Git hooks. Git push operations are intercepted and validated at the HTTP handler level before reaching the Git repository, eliminating the need for pre-receive hooks. |
| 6 | |||
| 7 | ## Architectural Decision: Inline vs. Hook-Based Authorization | ||
| 8 | |||
| 9 | ### Investigation Summary | ||
| 10 | |||
| 11 | After examining both the reference implementation and HTTP server options, we have two options: | ||
| 12 | |||
| 13 | #### Option 1: Hook-Based (Reference Implementation Approach) | ||
| 14 | |||
| 15 | - Use standard Git HTTP backend | ||
| 16 | - Create pre-receive and post-receive hooks | ||
| 17 | - Hooks query the Nostr relay and validate pushes | ||
| 18 | - **Pros**: Follows reference implementation closely | ||
| 19 | - **Cons**: Requires hook management, harder to test, less Rust-native | ||
| 20 | |||
| 21 | #### Option 2: Inline Authorization (Recommended) | ||
| 22 | |||
| 23 | - Intercept Git receive-pack requests in the HTTP handler | ||
| 24 | - Validate against Nostr state before spawning Git process | ||
| 25 | - Only forward valid pushes to Git | ||
| 26 | - **Pros**: Better error handling, easier testing, pure Rust, simpler deployment | ||
| 27 | - **Cons**: Requires custom Git protocol handling | ||
| 28 | |||
| 29 | ### Decision: Inline Authorization (Option 2) | ||
| 30 | |||
| 31 | **Rationale:** | ||
| 32 | |||
| 33 | 1. **Full control over HTTP layer**: Using Hyper directly gives us complete control over request handling, WebSocket upgrades, and CORS headers. | ||
| 34 | |||
| 35 | 2. **Better Developer Experience**: | ||
| 36 | |||
| 37 | - Validation errors can be returned as proper HTTP responses | ||
| 38 | - No need to parse hook stderr output | ||
| 39 | - Shared state between Git and Nostr components | ||
| 40 | - Pure Rust testing without shell scripts | ||
| 41 | |||
| 42 | 3. **Simpler Deployment**: | ||
| 43 | |||
| 44 | - Single binary | ||
| 45 | - No hook symlinks or permissions to manage | ||
| 46 | - No multi-process coordination | ||
| 47 | |||
| 48 | 4. **Performance**: | ||
| 49 | - Can parse incoming pack data once | ||
| 50 | - Avoid process spawn overhead for invalid pushes | ||
| 51 | - Better async integration | ||
| 52 | 6 | ||
| 53 | ## System Architecture | 7 | ## System Architecture |
| 54 | 8 | ||
| @@ -257,57 +211,73 @@ State events undergo authorization checks at multiple points: | |||
| 257 | 211 | ||
| 258 | ### 5. Purgatory System ([`src/purgatory/`](../../src/purgatory/)) | 212 | ### 5. Purgatory System ([`src/purgatory/`](../../src/purgatory/)) |
| 259 | 213 | ||
| 260 | The purgatory system solves the "which arrives first?" problem where either nostr events or git pushes can arrive in any order. It provides an in-memory holding area for events and git data awaiting their counterparts. | 214 | The purgatory system solves two related problems: |
| 215 | |||
| 216 | 1. **"Which arrives first?"** — Either nostr events or git pushes can arrive in any order. Purgatory holds events awaiting their git data counterparts. | ||
| 217 | 2. **Misleading empty repository announcements** — New announcements are held in purgatory until git data arrives, ensuring clients are never served announcements for repos with no content. | ||
| 261 | 218 | ||
| 262 | **Design Document**: See [`purgatory-design.md`](purgatory-design.md) for complete design specifications. | 219 | **Design Document**: See [`purgatory-design.md`](purgatory-design.md) for complete design specifications. |
| 263 | 220 | ||
| 264 | #### Architecture | 221 | #### Architecture |
| 265 | 222 | ||
| 266 | ```rust | 223 | ```rust |
| 267 | /// Main purgatory structure with two separate stores | 224 | /// Main purgatory structure with separate stores per event type |
| 268 | pub struct Purgatory { | 225 | pub struct Purgatory { |
| 226 | /// Announcement events (kind 30617) indexed by (owner, identifier) | ||
| 227 | /// Held until git data proves content exists | ||
| 228 | announcement_purgatory: DashMap<(PublicKey, String), AnnouncementPurgatoryEntry>, | ||
| 229 | |||
| 269 | /// State events (kind 30618) indexed by repository identifier | 230 | /// State events (kind 30618) indexed by repository identifier |
| 270 | state_events: Arc<DashMap<String, Vec<StatePurgatoryEntry>>>, | 231 | state_events: DashMap<String, Vec<StatePurgatoryEntry>>, |
| 271 | 232 | ||
| 272 | /// PR events (kind 1617/1618) or placeholders indexed by event ID | 233 | /// PR events (kind 1617/1618) or placeholders indexed by event ID |
| 273 | pr_events: Arc<DashMap<String, PrPurgatoryEntry>>, | 234 | pr_events: DashMap<String, PrPurgatoryEntry>, |
| 274 | } | 235 | } |
| 275 | ``` | 236 | ``` |
| 276 | 237 | ||
| 277 | **Key Design Principles:** | 238 | **Key Design Principles:** |
| 278 | 239 | ||
| 279 | 1. **Separate Storage**: State events and PR events use different indexing strategies | 240 | 1. **Separate Storage**: Each event type uses a different indexing strategy |
| 241 | - Announcements: Indexed by `(pubkey, identifier)` (unique per owner) | ||
| 280 | - State events: Indexed by `identifier` (multiple events can wait for same repo) | 242 | - State events: Indexed by `identifier` (multiple events can wait for same repo) |
| 281 | - PR events: Indexed by `event_id` (one-to-one mapping) | 243 | - PR events: Indexed by `event_id` (one-to-one mapping) |
| 282 | 244 | ||
| 283 | 2. **Late Binding**: State event refs are extracted at git push time, not event arrival | 245 | 2. **Announcement Purgatory**: New announcements are held until git data arrives |
| 246 | - Bare repo created immediately so pushes can succeed | ||
| 247 | - Announcement promoted to database only when git data proves content exists | ||
| 248 | - Two-phase soft expiry: bare repo deleted at 30 min, event retained 24h for revival | ||
| 249 | |||
| 250 | 3. **Late Binding**: State event refs are extracted at git push time, not event arrival | ||
| 284 | - Enables flexible matching when pushes arrive out-of-order | 251 | - Enables flexible matching when pushes arrive out-of-order |
| 285 | - Helper functions in [`helpers.rs`](../../src/purgatory/helpers.rs) handle ref extraction | 252 | - Helper functions in [`helpers.rs`](../../src/purgatory/helpers.rs) handle ref extraction |
| 286 | 253 | ||
| 287 | 3. **Bidirectional Waiting**: Either side can arrive first | 254 | 4. **Bidirectional Waiting**: Either side can arrive first |
| 288 | - **Event-first**: Event waits for git push | 255 | - **Event-first**: Event waits for git push |
| 289 | - **Git-first**: Placeholder created, waits for event | 256 | - **Git-first**: Placeholder created, waits for event |
| 290 | 257 | ||
| 291 | 4. **Automatic Expiry**: 30-minute default expiry, extensible during processing | 258 | 5. **Automatic Expiry**: 30-minute default expiry, extensible during processing |
| 292 | - Background cleanup task runs every 60 seconds | 259 | - Background cleanup task runs every 60 seconds |
| 293 | - Removes expired entries from both stores | 260 | - Removes expired entries from all stores |
| 294 | 261 | ||
| 295 | #### Data Types | 262 | #### Data Types |
| 296 | 263 | ||
| 297 | See [`types.rs`](../../src/purgatory/types.rs) for complete definitions: | 264 | See [`types.rs`](../../src/purgatory/types.rs) for complete definitions: |
| 298 | 265 | ||
| 299 | - **[`RefPair`](../../src/purgatory/types.rs:16)**: Ref name + object SHA pair | 266 | - **[`RefPair`](../../src/purgatory/types.rs:16)**: Ref name + object SHA pair |
| 267 | - **[`AnnouncementPurgatoryEntry`](../../src/purgatory/types.rs)**: Announcement with bare repo path, relays, and expiry | ||
| 300 | - **[`StatePurgatoryEntry`](../../src/purgatory/types.rs:29)**: State event with metadata | 268 | - **[`StatePurgatoryEntry`](../../src/purgatory/types.rs:29)**: State event with metadata |
| 301 | - **[`PrPurgatoryEntry`](../../src/purgatory/types.rs:52)**: PR event or placeholder with metadata | 269 | - **[`PrPurgatoryEntry`](../../src/purgatory/types.rs:52)**: PR event or placeholder with metadata |
| 302 | 270 | ||
| 303 | #### Integration Points | 271 | #### Integration Points |
| 304 | 272 | ||
| 305 | **Write Policy** ([`src/nostr/policy/`](../../src/nostr/policy/)): | 273 | **Write Policy** ([`src/nostr/policy/`](../../src/nostr/policy/)): |
| 306 | - State policy checks git data existence before adding to purgatory | 274 | - Announcement policy routes new announcements to purgatory; replacements accepted immediately |
| 275 | - State policy checks git data existence before adding to purgatory; checks purgatory announcements for authorization | ||
| 307 | - PR policy checks for placeholders before adding to purgatory | 276 | - PR policy checks for placeholders before adding to purgatory |
| 308 | - Events return "purgatory: will not be served until git data arrives" message | 277 | - Events return "purgatory: will not be served until git data arrives" message |
| 309 | 278 | ||
| 310 | **Git Handlers** ([`src/git/handlers.rs`](../../src/git/handlers.rs)): | 279 | **Git Handlers** ([`src/git/handlers.rs`](../../src/git/handlers.rs)): |
| 280 | - On git push: Promote announcement from purgatory to database if present | ||
| 311 | - On git push: Check purgatory for matching state events | 281 | - On git push: Check purgatory for matching state events |
| 312 | - On refs/nostr/* push: Check purgatory for PR events or create placeholders | 282 | - On refs/nostr/* push: Check purgatory for PR events or create placeholders |
| 313 | - Release events from purgatory when git data arrives | 283 | - Release events from purgatory when git data arrives |
| @@ -392,11 +362,25 @@ Configuration is loaded via **clap CLI > environment variables > .env > defaults | |||
| 392 | └─ Accept or reject | 362 | └─ Accept or reject |
| 393 | ↓ | 363 | ↓ |
| 394 | 4. If ACCEPTED: | 364 | 4. If ACCEPTED: |
| 395 | ├─ Event saved to database | 365 | ├─ Is there an active announcement for (pubkey, identifier) in DB? |
| 396 | └─ ensure_bare_repository() called | 366 | │ ├─ YES → Accept immediately (replacement, repo already proven) |
| 367 | │ └─ NO → Route to announcement purgatory | ||
| 397 | ↓ | 368 | ↓ |
| 398 | 5. Bare Git repository created at | 369 | 5. Announcement Purgatory path: |
| 399 | <git_data_path>/<npub>/<identifier>.git | 370 | ├─ Bare Git repository created immediately at |
| 371 | │ <git_data_path>/<npub>/<identifier>.git | ||
| 372 | ├─ Announcement held in purgatory (not served to clients) | ||
| 373 | └─ Awaiting git data to prove content exists | ||
| 374 | ↓ | ||
| 375 | 6. When git data arrives (push or background sync): | ||
| 376 | ├─ Announcement promoted from purgatory to database | ||
| 377 | ├─ Event now served to clients | ||
| 378 | └─ SyncManager upgrades to Full sync level | ||
| 379 | ↓ | ||
| 380 | 7. If no git data within 30 minutes: | ||
| 381 | ├─ Bare repo deleted (soft expiry) | ||
| 382 | ├─ Event retained 24h for potential revival | ||
| 383 | └─ Eventually discarded if no git data arrives | ||
| 400 | ``` | 384 | ``` |
| 401 | 385 | ||
| 402 | ### State Event Flow | 386 | ### State Event Flow |
| @@ -407,14 +391,25 @@ Configuration is loaded via **clap CLI > environment variables > .env > defaults | |||
| 407 | 2. Nostr relay receives event | 391 | 2. Nostr relay receives event |
| 408 | ↓ | 392 | ↓ |
| 409 | 3. Nip34WritePolicy::admit_event() | 393 | 3. Nip34WritePolicy::admit_event() |
| 410 | ├─ Check author is in maintainer set | 394 | ├─ Check author is in maintainer set (DB + purgatory announcements) |
| 411 | ├─ Validate state structure | 395 | ├─ Validate state structure |
| 412 | └─ Accept or reject | 396 | └─ Accept or reject |
| 413 | ↓ | 397 | ↓ |
| 414 | 4. If ACCEPTED and is latest state: | 398 | 4. If ACCEPTED: |
| 415 | ├─ Align repository refs to match state | 399 | ├─ Does git data already exist for this state? |
| 416 | ├─ Create/update/delete refs as needed | 400 | │ ├─ YES → Save to database immediately |
| 417 | └─ Set HEAD if commit available | 401 | │ └─ NO → Add to state purgatory |
| 402 | ↓ | ||
| 403 | 5. State Purgatory path: | ||
| 404 | ├─ Event held in purgatory (not served to clients) | ||
| 405 | ├─ Enqueued for background git data sync (3 min delay) | ||
| 406 | └─ Awaiting git push or background sync | ||
| 407 | ↓ | ||
| 408 | 6. When git push arrives: | ||
| 409 | ├─ Authorization checks both database AND purgatory | ||
| 410 | ├─ If authorized via purgatory state: push proceeds | ||
| 411 | ├─ After successful push: state event saved to database | ||
| 412 | └─ Removed from purgatory | ||
| 418 | ``` | 413 | ``` |
| 419 | 414 | ||
| 420 | ## Testing Strategy | 415 | ## Testing Strategy |
| @@ -613,9 +608,7 @@ WantedBy=multi-user.target | |||
| 613 | 608 | ||
| 614 | ## Conclusion | 609 | ## Conclusion |
| 615 | 610 | ||
| 616 | The inline authorization approach provides a cleaner, more maintainable architecture than hook-based authorization while maintaining full GRASP-01 compliance. Using Hyper for the HTTP layer gives us complete control over request handling, WebSocket upgrades, and CORS headers. | 611 | ngit-grasp uses inline authorization at the HTTP handler level, giving full control over request handling, WebSocket upgrades, and CORS headers while maintaining full GRASP-01 compliance. The purgatory system ensures that only repositories with actual git content are served to clients, and that events and git data are always consistent when released to the database. |
| 617 | |||
| 618 | The key insight is that we don't need to rely on Git's hook mechanism when we have full control over the HTTP layer that Git operates through. By intercepting at the HTTP handler level, we gain better error handling, easier testing, and tighter integration between the Git and Nostr components. | ||
| 619 | 612 | ||
| 620 | ## Related Documentation | 613 | ## Related Documentation |
| 621 | 614 | ||