diff options
Diffstat (limited to 'docs/learnings/grasp-audit.md')
| -rw-r--r-- | docs/learnings/grasp-audit.md | 209 |
1 files changed, 71 insertions, 138 deletions
diff --git a/docs/learnings/grasp-audit.md b/docs/learnings/grasp-audit.md index 14e5a2b..f4620d9 100644 --- a/docs/learnings/grasp-audit.md +++ b/docs/learnings/grasp-audit.md | |||
| @@ -1,13 +1,13 @@ | |||
| 1 | # GRASP Audit Tool - Patterns and Learnings | 1 | # GRASP Audit Tool - Patterns and Learnings |
| 2 | 2 | ||
| 3 | **Purpose:** Document grasp-audit architecture, patterns, and lessons learned | 3 | **Purpose:** Document grasp-audit architecture, patterns, and lessons learned |
| 4 | **Last Updated:** November 4, 2025 | 4 | **Last Updated:** December 4, 2025 |
| 5 | 5 | ||
| 6 | --- | 6 | --- |
| 7 | 7 | ||
| 8 | ## Overview | 8 | ## Overview |
| 9 | 9 | ||
| 10 | `grasp-audit` is a compliance testing tool for GRASP (Git Relays Authorized via Signed-Nostr Proofs) protocol implementations. It tests both Nostr relay compliance (NIP-01) and GRASP-specific functionality. | 10 | `grasp-audit` is a **fully implemented** compliance testing tool for GRASP (Git Relays Authorized via Signed-Nostr Proofs) protocol implementations. It tests both Nostr relay compliance (NIP-01) and GRASP-specific functionality. |
| 11 | 11 | ||
| 12 | --- | 12 | --- |
| 13 | 13 | ||
| @@ -32,10 +32,10 @@ | |||
| 32 | 32 | ||
| 33 | **Problem:** Test events pollute the relay and need cleanup without deletion events. | 33 | **Problem:** Test events pollute the relay and need cleanup without deletion events. |
| 34 | 34 | ||
| 35 | **Solution:** Use special tags to mark audit events: | 35 | **Solution:** Use special tags to mark audit events (implemented in [`grasp-audit/src/audit.rs`](grasp-audit/src/audit.rs)): |
| 36 | 36 | ||
| 37 | ```rust | 37 | ```rust |
| 38 | // Every audit event includes these tags | 38 | // Every audit event includes these tags (added automatically) |
| 39 | [ | 39 | [ |
| 40 | ["t", "grasp-audit-test-event"], // Marker | 40 | ["t", "grasp-audit-test-event"], // Marker |
| 41 | ["t", "audit-{run-id}"], // Run isolation | 41 | ["t", "audit-{run-id}"], // Run isolation |
| @@ -78,6 +78,8 @@ | |||
| 78 | 78 | ||
| 79 | ### Audit Configuration | 79 | ### Audit Configuration |
| 80 | 80 | ||
| 81 | From [`grasp-audit/src/audit.rs`](grasp-audit/src/audit.rs): | ||
| 82 | |||
| 81 | ```rust | 83 | ```rust |
| 82 | use grasp_audit::audit::AuditConfig; | 84 | use grasp_audit::audit::AuditConfig; |
| 83 | 85 | ||
| @@ -101,100 +103,41 @@ let config = AuditConfig::shared(); | |||
| 101 | 103 | ||
| 102 | ### Creating Audit Events | 104 | ### Creating Audit Events |
| 103 | 105 | ||
| 104 | ```rust | 106 | From [`grasp-audit/src/client.rs`](grasp-audit/src/client.rs): |
| 105 | use grasp_audit::audit::{AuditConfig, AuditEventBuilder}; | ||
| 106 | use nostr_sdk::prelude::*; | ||
| 107 | |||
| 108 | let config = AuditConfig::isolated(); | ||
| 109 | let keys = Keys::generate(); | ||
| 110 | |||
| 111 | // Create audit event | ||
| 112 | let event = AuditEventBuilder::new(&config, Kind::TextNote, "test content") | ||
| 113 | .build(&keys)?; | ||
| 114 | |||
| 115 | // Event automatically includes: | ||
| 116 | // - Audit marker tag | ||
| 117 | // - Run ID tag | ||
| 118 | // - Cleanup timestamp tag | ||
| 119 | ``` | ||
| 120 | |||
| 121 | --- | ||
| 122 | |||
| 123 | ### Querying Audit Events | ||
| 124 | 107 | ||
| 125 | ```rust | 108 | ```rust |
| 126 | use grasp_audit::client::AuditClient; | 109 | use grasp_audit::client::AuditClient; |
| 127 | use grasp_audit::audit::AuditConfig; | 110 | use grasp_audit::audit::AuditConfig; |
| 128 | 111 | ||
| 129 | let config = AuditConfig::isolated(); | 112 | let config = AuditConfig::isolated(); |
| 130 | let client = AuditClient::new(config, keys); | 113 | let client = AuditClient::new("ws://localhost:8080", config).await?; |
| 131 | |||
| 132 | // Connect to relay | ||
| 133 | client.add_relay("ws://localhost:7000").await?; | ||
| 134 | client.connect().await; | ||
| 135 | |||
| 136 | // Query audit events for this run | ||
| 137 | let events = client.query().await?; | ||
| 138 | |||
| 139 | // Events are filtered by: | ||
| 140 | // - "grasp-audit-test-event" marker | ||
| 141 | // - Current run ID | ||
| 142 | ``` | ||
| 143 | |||
| 144 | --- | ||
| 145 | |||
| 146 | ### Test Isolation | ||
| 147 | |||
| 148 | **Each test run is isolated by unique run ID:** | ||
| 149 | |||
| 150 | ```rust | ||
| 151 | // CI mode generates unique UUID per run | ||
| 152 | let config1 = AuditConfig::isolated(); | ||
| 153 | let config2 = AuditConfig::isolated(); | ||
| 154 | |||
| 155 | // config1.run_id != config2.run_id | ||
| 156 | // Tests won't interfere with each other | ||
| 157 | ``` | ||
| 158 | |||
| 159 | **Benefits:** | ||
| 160 | 114 | ||
| 161 | - ✅ Parallel CI/CD runs don't conflict | 115 | // Create and send an event - cleanup tags are added automatically |
| 162 | - ✅ Can run multiple test suites simultaneously | 116 | let event = client.event_builder() |
| 163 | - ✅ Easy to identify which run created which events | 117 | .kind(Kind::TextNote) |
| 164 | - ✅ Cleanup can target specific runs | 118 | .content("test content") |
| 165 | 119 | .build(&keys)?; | |
| 166 | --- | ||
| 167 | |||
| 168 | ### Cleanup Strategy | ||
| 169 | |||
| 170 | **Two-phase cleanup:** | ||
| 171 | |||
| 172 | 1. **Automatic expiry** via cleanup timestamp tag | ||
| 173 | 2. **Manual cleanup** by querying and deleting | ||
| 174 | |||
| 175 | ```rust | ||
| 176 | // Events include cleanup timestamp | ||
| 177 | ["t", "audit-cleanup-after-1730707200"] | ||
| 178 | 120 | ||
| 179 | // Cleanup process: | 121 | client.send_event(event).await?; |
| 180 | // 1. Query events with expired cleanup timestamp | ||
| 181 | // 2. Delete from database directly (no KIND 5) | ||
| 182 | // 3. Avoid deletion event pollution | ||
| 183 | ``` | 122 | ``` |
| 184 | 123 | ||
| 185 | **Implementation:** To be built in relay (not in audit tool) | ||
| 186 | |||
| 187 | --- | 124 | --- |
| 188 | 125 | ||
| 189 | ## Testing Strategy | 126 | ### Test Suites |
| 190 | 127 | ||
| 191 | ### Test Organization | 128 | From [`grasp-audit/src/specs/grasp01/mod.rs`](grasp-audit/src/specs/grasp01/mod.rs): |
| 192 | 129 | ||
| 193 | ``` | 130 | ``` |
| 194 | grasp-audit/src/specs/ | 131 | grasp-audit/src/specs/grasp01/ |
| 195 | ├── nip01_smoke.rs # NIP-01 basic functionality | 132 | ├── mod.rs # Module exports |
| 196 | ├── grasp_01_relay.rs # GRASP-01 relay requirements (planned) | 133 | ├── nip01_smoke.rs # NIP-01 basic functionality |
| 197 | └── mod.rs # Test suite registry | 134 | ├── nip11_document.rs # NIP-11 document tests |
| 135 | ├── event_acceptance_policy.rs # GRASP-01 event rules | ||
| 136 | ├── cors.rs # CORS header tests | ||
| 137 | ├── git_clone.rs # Git clone operations | ||
| 138 | ├── push_authorization.rs # Push validation tests | ||
| 139 | ├── repository_creation.rs # Repository lifecycle | ||
| 140 | └── spec_requirements.rs # Requirement definitions | ||
| 198 | ``` | 141 | ``` |
| 199 | 142 | ||
| 200 | ### Unit vs Integration Tests | 143 | ### Unit vs Integration Tests |
| @@ -229,17 +172,18 @@ mod tests { | |||
| 229 | 172 | ||
| 230 | ```bash | 173 | ```bash |
| 231 | # Unit tests (fast, no dependencies) | 174 | # Unit tests (fast, no dependencies) |
| 232 | cargo test --lib | 175 | cd grasp-audit && nix develop -c cargo test --lib |
| 233 | 176 | ||
| 234 | # Integration tests (requires relay) | 177 | # Integration tests (requires relay via test-ngit-relay.sh) |
| 235 | docker run --rm -p 7000:7000 scsibug/nostr-rs-relay | 178 | cd grasp-audit && nix develop -c bash test-ngit-relay.sh --mode test |
| 236 | cargo test -- --ignored | ||
| 237 | ``` | 179 | ``` |
| 238 | 180 | ||
| 239 | --- | 181 | --- |
| 240 | 182 | ||
| 241 | ### Test Result Reporting | 183 | ### Test Result Reporting |
| 242 | 184 | ||
| 185 | From [`grasp-audit/src/result.rs`](grasp-audit/src/result.rs): | ||
| 186 | |||
| 243 | ```rust | 187 | ```rust |
| 244 | use grasp_audit::result::AuditResult; | 188 | use grasp_audit::result::AuditResult; |
| 245 | 189 | ||
| @@ -255,7 +199,7 @@ for result in &results { | |||
| 255 | } | 199 | } |
| 256 | 200 | ||
| 257 | // Summary | 201 | // Summary |
| 258 | let passed = results.iter().filter(|r| r.is_pass()).count(); | 202 | let passed = results.iter().filter(|r| r.passed).count(); |
| 259 | let total = results.len(); | 203 | let total = results.len(); |
| 260 | println!("Results: {}/{} passed ({:.1}%)", | 204 | println!("Results: {}/{} passed ({:.1}%)", |
| 261 | passed, total, (passed as f64 / total as f64) * 100.0); | 205 | passed, total, (passed as f64 / total as f64) * 100.0); |
| @@ -291,7 +235,7 @@ grasp-audit audit \ | |||
| 291 | grasp-audit audit \ | 235 | grasp-audit audit \ |
| 292 | --relay wss://relay.example.com \ | 236 | --relay wss://relay.example.com \ |
| 293 | --mode production \ | 237 | --mode production \ |
| 294 | --run-id "audit-2025-11-04" \ | 238 | --run-id "audit-2025-12-04" \ |
| 295 | --verbose | 239 | --verbose |
| 296 | 240 | ||
| 297 | # Test all specs | 241 | # Test all specs |
| @@ -366,25 +310,23 @@ let events = client.query().await?; | |||
| 366 | 310 | ||
| 367 | --- | 311 | --- |
| 368 | 312 | ||
| 369 | ## Future Enhancements | 313 | ## What's Implemented |
| 370 | 314 | ||
| 371 | ### Planned Features | 315 | ### Completed Features |
| 372 | |||
| 373 | - [ ] **GRASP-01 Test Suite**: Repository announcement and state event tests | ||
| 374 | - [ ] **Test Report Generation**: JSON/HTML output for CI/CD | ||
| 375 | - [ ] **Performance Benchmarks**: Measure relay performance | ||
| 376 | - [ ] **Relay Comparison**: Side-by-side compliance comparison | ||
| 377 | - [ ] **Continuous Monitoring**: Periodic production audits | ||
| 378 | 316 | ||
| 379 | --- | 317 | - ✅ **GRASP-01 Test Suites**: All NIP-01, NIP-11, CORS, event acceptance tests |
| 318 | - ✅ **Spec Requirements Database**: Machine-readable requirements in [`spec_requirements.rs`](grasp-audit/src/specs/grasp01/spec_requirements.rs) | ||
| 319 | - ✅ **Automatic Cleanup Tags**: Production-safe event tagging | ||
| 320 | - ✅ **Test Isolation**: UUID run IDs for parallel execution | ||
| 321 | - ✅ **AuditClient**: Nostr client wrapper with audit features | ||
| 322 | - ✅ **Fixture Helpers**: Event creation helpers in [`fixtures.rs`](grasp-audit/src/fixtures.rs) | ||
| 380 | 323 | ||
| 381 | ### Possible Improvements | 324 | ### Future Enhancements |
| 382 | 325 | ||
| 383 | - [ ] **Parallel Test Execution**: Run specs in parallel | 326 | - [ ] **GRASP-02 Test Suite**: Proactive sync tests |
| 384 | - [ ] **Retry Logic**: Handle transient failures | 327 | - [ ] **HTML Report Generation**: Rich CI/CD reports |
| 385 | - [ ] **Custom Assertions**: Domain-specific test helpers | 328 | - [ ] **Performance Benchmarks**: Measure relay performance |
| 386 | - [ ] **Event Diff Tool**: Compare expected vs actual events | 329 | - [ ] **Relay Comparison**: Side-by-side compliance comparison |
| 387 | - [ ] **Cleanup Automation**: Auto-cleanup after tests | ||
| 388 | 330 | ||
| 389 | --- | 331 | --- |
| 390 | 332 | ||
| @@ -403,14 +345,12 @@ let events = client.query().await?; | |||
| 403 | **Solution:** | 345 | **Solution:** |
| 404 | 346 | ||
| 405 | ```bash | 347 | ```bash |
| 406 | # Start relay | 348 | # Use test-ngit-relay.sh for automated relay management |
| 407 | docker run --rm -p 7000:7000 scsibug/nostr-rs-relay | 349 | cd grasp-audit && nix develop -c bash test-ngit-relay.sh --mode test |
| 408 | |||
| 409 | # Verify relay is running | ||
| 410 | curl http://localhost:7000 | ||
| 411 | 350 | ||
| 412 | # Run tests | 351 | # Or manually: |
| 413 | cargo test -- --ignored | 352 | docker run --rm -p 18081:8081 ghcr.io/danconwaydev/ngit-relay:latest |
| 353 | RELAY_URL="ws://localhost:18081" nix develop -c cargo test --lib -- --ignored | ||
| 414 | ``` | 354 | ``` |
| 415 | 355 | ||
| 416 | --- | 356 | --- |
| @@ -471,46 +411,39 @@ let config = AuditConfig::isolated(); | |||
| 471 | let config = AuditConfig::shared(); | 411 | let config = AuditConfig::shared(); |
| 472 | ``` | 412 | ``` |
| 473 | 413 | ||
| 474 | ### Event Creation | ||
| 475 | |||
| 476 | ```rust | ||
| 477 | let event = AuditEventBuilder::new(&config, kind, content) | ||
| 478 | .build(&keys)?; | ||
| 479 | ``` | ||
| 480 | |||
| 481 | ### Client Usage | 414 | ### Client Usage |
| 482 | 415 | ||
| 483 | ```rust | 416 | ```rust |
| 484 | let client = AuditClient::new(config, keys); | 417 | let client = AuditClient::new("ws://localhost:7000", config).await?; |
| 485 | client.add_relay("ws://localhost:7000").await?; | 418 | assert!(client.is_connected().await); |
| 486 | client.connect().await; | ||
| 487 | let events = client.query().await?; | ||
| 488 | ``` | 419 | ``` |
| 489 | 420 | ||
| 490 | ### Running Tests | 421 | ### Running Tests |
| 491 | 422 | ||
| 492 | ```bash | 423 | ```bash |
| 493 | # Unit tests | 424 | # Unit tests (from grasp-audit/) |
| 494 | cargo test --lib | 425 | nix develop -c cargo test --lib |
| 495 | 426 | ||
| 496 | # Integration tests | 427 | # Integration tests with ngit-relay |
| 497 | cargo test -- --ignored | 428 | nix develop -c bash test-ngit-relay.sh --mode test |
| 498 | 429 | ||
| 499 | # CLI | 430 | # CLI audit |
| 500 | cargo run -- audit --relay ws://localhost:7000 | 431 | nix develop -c cargo run -- audit --relay ws://localhost:7000 |
| 501 | ``` | 432 | ``` |
| 502 | 433 | ||
| 503 | --- | 434 | ### Key Files |
| 504 | |||
| 505 | ## References | ||
| 506 | 435 | ||
| 507 | - **GRASP Protocol**: https://gitworkshop.dev/danconwaydev.com/grasp | 436 | | File | Purpose | |
| 508 | - **NIP-01**: https://github.com/nostr-protocol/nips/blob/master/01.md | 437 | |------|---------| |
| 509 | - **NIP-34**: https://github.com/nostr-protocol/nips/blob/master/34.md | 438 | | [`grasp-audit/src/lib.rs`](grasp-audit/src/lib.rs) | Public API | |
| 510 | - **grasp-audit README**: `grasp-audit/README.md` | 439 | | [`grasp-audit/src/client.rs`](grasp-audit/src/client.rs) | AuditClient implementation | |
| 511 | - **Tag Migration**: `docs/archive/2025-11-04-tag-migration.md` | 440 | | [`grasp-audit/src/audit.rs`](grasp-audit/src/audit.rs) | AuditConfig, cleanup tags | |
| 441 | | [`grasp-audit/src/specs/grasp01/mod.rs`](grasp-audit/src/specs/grasp01/mod.rs) | Test suite registry | | ||
| 442 | | [`grasp-audit/src/specs/grasp01/spec_requirements.rs`](grasp-audit/src/specs/grasp01/spec_requirements.rs) | Requirement database | | ||
| 512 | 443 | ||
| 513 | --- | 444 | --- |
| 514 | 445 | ||
| 515 | _Last updated: November 4, 2025_ | 446 | ## Related Documentation |
| 516 | _Status: Living document - update as grasp-audit evolves_ | 447 | |
| 448 | - [Test Strategy](../reference/test-strategy.md) - Overall testing approach | ||
| 449 | - [GRASP-01 Implementation](grasp-01-implementation.md) - Main project learnings \ No newline at end of file | ||