upleb.uk

Public git repos — served from a NIP-34 GRASP relay at git.upleb.uk

summaryrefslogtreecommitdiff
path: root/docs/reference/test-strategy.md
diff options
context:
space:
mode:
Diffstat (limited to 'docs/reference/test-strategy.md')
-rw-r--r--docs/reference/test-strategy.md1339
1 files changed, 207 insertions, 1132 deletions
diff --git a/docs/reference/test-strategy.md b/docs/reference/test-strategy.md
index cc1d5b0..7a31bdf 100644
--- a/docs/reference/test-strategy.md
+++ b/docs/reference/test-strategy.md
@@ -2,15 +2,15 @@
2 2
3## Overview 3## Overview
4 4
5This document outlines the comprehensive testing strategy for ngit-grasp, including a **reusable GRASP compliance testing tool** that can validate any GRASP implementation against the protocol specification. 5This document describes the testing strategy for ngit-grasp, including the **grasp-audit** reusable compliance testing tool and the **integration tests** in the main repository.
6 6
7## Testing Philosophy 7## Testing Philosophy
8 8
91. **Specification-Driven**: Tests mirror the GRASP protocol structure exactly 91. **Specification-Driven**: Tests mirror GRASP-01 protocol structure exactly
102. **Compliance-First**: Every requirement in the spec has a corresponding test 102. **Compliance-First**: Every requirement in the spec has a corresponding test
113. **Reusable**: Compliance tests can validate any GRASP implementation 113. **Reusable**: The grasp-audit tool can validate any GRASP implementation
124. **Clear Failures**: Test failures cite exact spec lines/sections 124. **Isolated**: Each test runs with its own relay instance via [`TestRelay`](tests/common/relay.rs:14)
135. **Comprehensive**: Unit, integration, and compliance testing 135. **Clear Failures**: Test failures cite exact spec requirements
14 14
15## Test Pyramid 15## Test Pyramid
16 16
@@ -20,1219 +20,294 @@ This document outlines the comprehensive testing strategy for ngit-grasp, includ
20 ╱ E2E╲ ~ 10% End-to-end with real Git 20 ╱ E2E╲ ~ 10% End-to-end with real Git
21 ╱──────╲ 21 ╱──────╲
22 ╱ ╲ 22 ╱ ╲
23 ╱Compliance╲ ~ 20% GRASP spec validation 23 ╱Compliance╲ ~ 30% GRASP-01 spec validation
24 ╱────────────╲ 24 ╱────────────╲ (grasp-audit)
25 ╱ ╲ 25 ╱ ╲
26 ╱ Integration ╲ ~ 30% Component interaction 26 ╱ Integration ╲ ~ 30% Component interaction
27 ╱──────────────────╲ 27 ╱──────────────────╲ (tests/)
28 ╱ ╲ 28 ╱ ╲
29 ╱ Unit Tests ╲ ~ 40% Individual functions 29 ╱ Unit Tests ╲ ~ 30% Individual functions
30 ╱────────────────────────╲ 30 ╱────────────────────────╲ (src/**/tests)
31``` 31```
32 32
33## GRASP Compliance Testing Tool 33## Project Structure
34 34
35### Design Goals 35### Actual Test Layout
36
371. **Reusable**: Can test ngit-grasp or any other GRASP implementation
382. **Spec-Mirrored**: Test structure matches GRASP protocol documents
393. **Clear Reporting**: Failures cite exact spec requirements
404. **Automated**: Can run in CI/CD
415. **Extensible**: Easy to add new GRASP versions (GRASP-02, GRASP-05)
42
43### Project Structure
44 36
45``` 37```
46grasp-compliance-tests/ 38ngit-grasp/
47├── Cargo.toml # Standalone crate 39├── tests/ # Integration tests for ngit-grasp
48├── README.md # Usage instructions 40│ ├── common/
49├── src/ 41│ │ ├── mod.rs # Test utilities module
50│ ├── lib.rs # Public API 42│ │ └── relay.rs # TestRelay fixture
51│ ├── client.rs # Test client utilities 43│ ├── nip01_compliance.rs # NIP-01 relay compliance
52│ ├── assertions.rs # Spec-based assertions 44│ ├── nip11_document.rs # NIP-11 document tests
53│ └── specs/ 45│ ├── nip34_announcements.rs # Repository announcement tests
54│ ├── mod.rs # Spec registry 46│ ├── repository_creation.rs # Git repo creation tests
55│ ├── grasp_01.rs # GRASP-01 tests 47│ ├── push_authorization.rs # Push validation tests
56│ ├── grasp_02.rs # GRASP-02 tests 48│ ├── cors.rs # CORS header tests
57│ └── grasp_05.rs # GRASP-05 tests 49│ └── git_clone.rs # Git clone tests
58├── fixtures/ 50│
59│ ├── repos/ # Test repositories 51└── grasp-audit/ # Reusable GRASP compliance tool
60│ ├── events/ # Nostr event fixtures 52 ├── Cargo.toml
61│ └── keys/ # Test keypairs 53 ├── flake.nix
62└── examples/ 54 └── src/
63 └── test_implementation.rs # Example usage 55 ├── lib.rs # Public API
56 ├── client.rs # AuditClient
57 ├── audit.rs # AuditConfig, cleanup tags
58 ├── fixtures.rs # Test fixtures
59 └── specs/
60 └── grasp01/ # GRASP-01 specification tests
61 ├── mod.rs # Module exports
62 ├── nip01_smoke.rs # NIP-01 smoke tests
63 ├── nip11_document.rs
64 ├── event_acceptance_policy.rs
65 ├── cors.rs
66 ├── git_clone.rs
67 ├── push_authorization.rs
68 ├── repository_creation.rs
69 └── spec_requirements.rs # Requirement definitions
64``` 70```
65 71
66### Spec-Mirrored Test Structure 72## Integration Tests (tests/)
73
74### TestRelay Fixture
67 75
68Each GRASP spec document maps to a test module with identical structure: 76The [`TestRelay`](tests/common/relay.rs:14) fixture provides automatic relay lifecycle management:
69 77
70```rust 78```rust
71// src/specs/grasp_01.rs 79// From tests/common/relay.rs
72 80
73use crate::{TestContext, SpecRequirement, ComplianceResult}; 81/// Test relay fixture that manages relay lifecycle
74 82///
75/// GRASP-01 - Core Service Requirements 83/// Automatically starts and stops the ngit-grasp relay for testing.
76/// Reference: https://gitworkshop.dev/danconwaydev.com/grasp/01.md 84/// Uses a random port to avoid conflicts and cleans up created repositories.
77pub struct Grasp01Spec; 85pub struct TestRelay {
78 86 process: Child,
79impl Grasp01Spec { 87 url: String,
80 /// Run all GRASP-01 compliance tests 88 port: u16,
81 pub async fn test_compliance(ctx: &TestContext) -> ComplianceResult { 89}
82 let mut results = ComplianceResult::new("GRASP-01"); 90
83 91impl TestRelay {
84 // Section: Nostr Relay 92 /// Start a test relay instance
85 results.add(Self::test_nostr_relay_nip01_compliance(ctx).await); 93 pub async fn start() -> Self { ... }
86 results.add(Self::test_accepts_repository_announcements(ctx).await);
87 results.add(Self::test_accepts_repository_state_announcements(ctx).await);
88 results.add(Self::test_rejects_unlisted_announcements(ctx).await);
89 results.add(Self::test_accepts_related_events(ctx).await);
90 results.add(Self::test_serves_nip11_document(ctx).await);
91 results.add(Self::test_nip11_has_supported_grasps(ctx).await);
92 results.add(Self::test_nip11_has_repo_acceptance_criteria(ctx).await);
93 results.add(Self::test_nip11_has_curation_policy(ctx).await);
94
95 // Section: Git Smart HTTP Service
96 results.add(Self::test_serves_git_at_correct_path(ctx).await);
97 results.add(Self::test_accepts_matching_pushes(ctx).await);
98 results.add(Self::test_rejects_mismatched_pushes(ctx).await);
99 results.add(Self::test_respects_recursive_maintainers(ctx).await);
100 results.add(Self::test_sets_head_from_state(ctx).await);
101 results.add(Self::test_accepts_nostr_refs(ctx).await);
102 results.add(Self::test_rejects_pr_branches(ctx).await);
103 results.add(Self::test_deletes_orphaned_nostr_refs(ctx).await);
104 results.add(Self::test_allows_reachable_sha1_in_want(ctx).await);
105 results.add(Self::test_allows_tip_sha1_in_want(ctx).await);
106 results.add(Self::test_serves_webpage(ctx).await);
107
108 // Section: CORS Support
109 results.add(Self::test_cors_allow_origin(ctx).await);
110 results.add(Self::test_cors_allow_methods(ctx).await);
111 results.add(Self::test_cors_allow_headers(ctx).await);
112 results.add(Self::test_cors_options_request(ctx).await);
113
114 results
115 }
116
117 // ================================================================
118 // NOSTR RELAY TESTS
119 // ================================================================
120
121 /// MUST serve a NIP-01 compliant nostr relay at `/`
122 ///
123 /// Spec: GRASP-01, Line 9-10
124 /// > MUST serve a [NIP-01](https://nips.nostr.com/1) compliant nostr
125 /// > relay at `/` that accepts [git repository announcements]...
126 async fn test_nostr_relay_nip01_compliance(ctx: &TestContext) -> TestResult {
127 TestResult::new(
128 "nostr_relay_nip01_compliance",
129 "GRASP-01:9-10",
130 "MUST serve a NIP-01 compliant nostr relay at `/`",
131 )
132 .run(async {
133 // Test WebSocket upgrade at /
134 let ws = ctx.connect_websocket("/").await?;
135
136 // Test NIP-01 REQ/EVENT/CLOSE/NOTICE messages
137 ws.send_req("test-sub", vec![]).await?;
138 let response = ws.recv().await?;
139 assert_nip01_eose(response)?;
140
141 Ok(())
142 })
143 .await
144 }
145
146 /// MUST reject announcements that do not list the service in both
147 /// `clone` and `relays` tags unless implementing `GRASP-05`
148 ///
149 /// Spec: GRASP-01, Line 12-13
150 /// > MUST reject [git repository announcements] that do not list the
151 /// > service in both `clone` and `relays` tags unless implementing `GRASP-05`.
152 async fn test_rejects_unlisted_announcements(ctx: &TestContext) -> TestResult {
153 TestResult::new(
154 "rejects_unlisted_announcements",
155 "GRASP-01:12-13",
156 "MUST reject announcements not listing service in clone and relays",
157 )
158 .run(async {
159 let event = ctx.create_announcement()
160 .without_clone_tag(ctx.domain())
161 .build()
162 .await?;
163
164 let result = ctx.send_event(event).await?;
165
166 assert_eq!(
167 result.ok, false,
168 "Expected rejection of announcement without clone tag"
169 );
170 assert!(
171 result.message.contains("clone") || result.message.contains("relays"),
172 "Expected rejection message to mention clone/relays requirement"
173 );
174
175 Ok(())
176 })
177 .await
178 }
179
180 /// MUST accept other events that tag, or are tagged by, accepted announcements
181 ///
182 /// Spec: GRASP-01, Line 17-20
183 /// > MUST accept other events that tag, or are tagged by, either:
184 /// > 1. accepted [git repository announcements]; or
185 /// > 2. accepted [issues] or [patches]
186 async fn test_accepts_related_events(ctx: &TestContext) -> TestResult {
187 TestResult::new(
188 "accepts_related_events",
189 "GRASP-01:17-20",
190 "MUST accept events that tag or are tagged by accepted announcements",
191 )
192 .run(async {
193 // First, create and accept an announcement
194 let announcement = ctx.create_announcement()
195 .with_clone_tag(ctx.domain())
196 .with_relay_tag(ctx.domain())
197 .build()
198 .await?;
199
200 ctx.send_event(announcement.clone()).await?;
201
202 // Now send an issue that tags the announcement
203 let issue = ctx.create_issue()
204 .tag_announcement(&announcement)
205 .build()
206 .await?;
207
208 let result = ctx.send_event(issue).await?;
209
210 assert_eq!(
211 result.ok, true,
212 "Expected acceptance of issue tagging accepted announcement"
213 );
214
215 Ok(())
216 })
217 .await
218 }
219
220 /// MUST serve a NIP-11 document with required fields
221 ///
222 /// Spec: GRASP-01, Line 24-27
223 /// > MUST serve a [NIP-11] document:
224 /// > 1. MUST list each supported GRASP under `supported_grasps`
225 /// > 2. MUST list repository acceptance criteria under `repo_acceptance_criteria`
226 /// > 3. MUST list curation policy under `curation` if events are curated
227 async fn test_serves_nip11_document(ctx: &TestContext) -> TestResult {
228 TestResult::new(
229 "serves_nip11_document",
230 "GRASP-01:24-27",
231 "MUST serve a NIP-11 document",
232 )
233 .run(async {
234 let nip11 = ctx.fetch_nip11().await?;
235
236 assert!(
237 nip11.contains_key("supported_nips"),
238 "NIP-11 document must have supported_nips"
239 );
240
241 Ok(())
242 })
243 .await
244 }
245
246 /// NIP-11 MUST list supported GRASPs
247 ///
248 /// Spec: GRASP-01, Line 25
249 /// > 1. MUST list each supported GRASP under `supported_grasps`
250 /// > in format `GRASP-XX` eg `GRASP-01` as a string array
251 async fn test_nip11_has_supported_grasps(ctx: &TestContext) -> TestResult {
252 TestResult::new(
253 "nip11_has_supported_grasps",
254 "GRASP-01:25",
255 "NIP-11 MUST list supported_grasps as string array",
256 )
257 .run(async {
258 let nip11 = ctx.fetch_nip11().await?;
259
260 let grasps = nip11.get("supported_grasps")
261 .ok_or("NIP-11 missing supported_grasps field")?
262 .as_array()
263 .ok_or("supported_grasps must be an array")?;
264
265 assert!(
266 grasps.iter().any(|g| g.as_str() == Some("GRASP-01")),
267 "supported_grasps must include 'GRASP-01'"
268 );
269
270 // Validate format: GRASP-XX
271 for grasp in grasps {
272 let s = grasp.as_str().ok_or("GRASP must be a string")?;
273 assert!(
274 s.starts_with("GRASP-") && s.len() >= 8,
275 "GRASP format must be 'GRASP-XX', got: {}", s
276 );
277 }
278
279 Ok(())
280 })
281 .await
282 }
283
284 // ================================================================
285 // GIT SMART HTTP SERVICE TESTS
286 // ================================================================
287
288 /// MUST serve a git repository via git smart http at /<npub>/<identifier>.git
289 ///
290 /// Spec: GRASP-01, Line 31-32
291 /// > MUST serve a git repository via an unauthenticated [git smart http service]
292 /// > at `/<npub>/<identifier>.git` for each accepted announcement
293 async fn test_serves_git_at_correct_path(ctx: &TestContext) -> TestResult {
294 TestResult::new(
295 "serves_git_at_correct_path",
296 "GRASP-01:31-32",
297 "MUST serve git at /<npub>/<identifier>.git",
298 )
299 .run(async {
300 // Create and send announcement
301 let announcement = ctx.create_announcement()
302 .with_identifier("test-repo")
303 .with_clone_tag(ctx.domain())
304 .with_relay_tag(ctx.domain())
305 .build()
306 .await?;
307
308 let npub = announcement.author_npub();
309 ctx.send_event(announcement).await?;
310
311 // Wait for repo creation
312 tokio::time::sleep(Duration::from_secs(2)).await;
313
314 // Test git info/refs endpoint
315 let path = format!("/{}/test-repo.git/info/refs?service=git-upload-pack", npub);
316 let response = ctx.http_get(&path).await?;
317
318 assert_eq!(
319 response.status(), 200,
320 "Git info/refs must return 200 OK"
321 );
322
323 assert_eq!(
324 response.headers().get("content-type").unwrap(),
325 "application/x-git-upload-pack-advertisement",
326 "Git info/refs must have correct content-type"
327 );
328
329 Ok(())
330 })
331 .await
332 }
333
334 /// MUST accept pushes that match the latest state announcement
335 ///
336 /// Spec: GRASP-01, Line 34-35
337 /// > MUST accept pushes via this service that match the latest
338 /// > [repo state announcement] on the relay, respecting the recursive maintainer set.
339 async fn test_accepts_matching_pushes(ctx: &TestContext) -> TestResult {
340 TestResult::new(
341 "accepts_matching_pushes",
342 "GRASP-01:34-35",
343 "MUST accept pushes matching latest state announcement",
344 )
345 .run(async {
346 // Setup: Create repo with announcement and state
347 let (announcement, state) = ctx.create_repo_with_state()
348 .branch("main", "a1b2c3d4...")
349 .build()
350 .await?;
351
352 // Push matching state
353 let result = ctx.git_push(&announcement, "main", "a1b2c3d4...").await?;
354
355 assert!(
356 result.success,
357 "Push matching state must succeed, got: {}", result.stderr
358 );
359
360 Ok(())
361 })
362 .await
363 }
364
365 /// MUST reject pushes that don't match the state announcement
366 ///
367 /// Spec: GRASP-01, Line 34-35 (inverse requirement)
368 /// Implied by "MUST accept pushes... that match"
369 async fn test_rejects_mismatched_pushes(ctx: &TestContext) -> TestResult {
370 TestResult::new(
371 "rejects_mismatched_pushes",
372 "GRASP-01:34-35",
373 "MUST reject pushes not matching state announcement",
374 )
375 .run(async {
376 // Setup: Create repo with state pointing to commit A
377 let (announcement, state) = ctx.create_repo_with_state()
378 .branch("main", "aaaa1111...")
379 .build()
380 .await?;
381
382 // Try to push different commit B
383 let result = ctx.git_push(&announcement, "main", "bbbb2222...").await;
384
385 assert!(
386 result.is_err() || !result.unwrap().success,
387 "Push not matching state must be rejected"
388 );
389
390 Ok(())
391 })
392 .await
393 }
394
395 /// MUST accept pushes to refs/nostr/<event-id>
396 ///
397 /// Spec: GRASP-01, Line 42-44
398 /// > MUST accept pushes via this service to `refs/nostr/<event-id>` but
399 /// > SHOULD reject if event exists on relay listing a different tip
400 async fn test_accepts_nostr_refs(ctx: &TestContext) -> TestResult {
401 TestResult::new(
402 "accepts_nostr_refs",
403 "GRASP-01:42-44",
404 "MUST accept pushes to refs/nostr/<event-id>",
405 )
406 .run(async {
407 let (announcement, _) = ctx.create_repo_with_state().build().await?;
408
409 // Create a PR event
410 let pr_event = ctx.create_pr_event()
411 .for_repo(&announcement)
412 .build()
413 .await?;
414
415 let event_id = pr_event.id();
416
417 // Push to refs/nostr/<event-id>
418 let result = ctx.git_push(
419 &announcement,
420 &format!("refs/nostr/{}", event_id),
421 "commit-sha..."
422 ).await?;
423
424 assert!(
425 result.success,
426 "Push to refs/nostr/<event-id> must succeed"
427 );
428
429 Ok(())
430 })
431 .await
432 }
433
434 /// MUST reject pr/* branches
435 ///
436 /// Spec: GRASP-01, Line 42-44 (implied)
437 /// PRs should use refs/nostr/, not refs/heads/pr/*
438 async fn test_rejects_pr_branches(ctx: &TestContext) -> TestResult {
439 TestResult::new(
440 "rejects_pr_branches",
441 "GRASP-01:42-44",
442 "MUST reject refs/heads/pr/* (use refs/nostr/ instead)",
443 )
444 .run(async {
445 let (announcement, _) = ctx.create_repo_with_state().build().await?;
446
447 // Try to push to pr/* branch
448 let result = ctx.git_push(
449 &announcement,
450 "refs/heads/pr/123",
451 "commit-sha..."
452 ).await;
453
454 assert!(
455 result.is_err() || !result.unwrap().success,
456 "Push to refs/heads/pr/* must be rejected"
457 );
458
459 Ok(())
460 })
461 .await
462 }
463
464 /// MUST include allow-reachable-sha1-in-want and allow-tip-sha1-in-want
465 ///
466 /// Spec: GRASP-01, Line 48-49
467 /// > MUST include `allow-reachable-sha1-in-want` and `allow-tip-sha1-in-want`
468 /// > in advertisement and serve available oids.
469 async fn test_allows_tip_sha1_in_want(ctx: &TestContext) -> TestResult {
470 TestResult::new(
471 "allows_tip_sha1_in_want",
472 "GRASP-01:48-49",
473 "MUST advertise and support allow-tip-sha1-in-want",
474 )
475 .run(async {
476 let (announcement, _) = ctx.create_repo_with_state()
477 .branch("main", "a1b2c3d4...")
478 .build()
479 .await?;
480
481 // Fetch git capabilities
482 let caps = ctx.git_capabilities(&announcement).await?;
483
484 assert!(
485 caps.contains("allow-tip-sha1-in-want"),
486 "Git advertisement must include allow-tip-sha1-in-want"
487 );
488
489 assert!(
490 caps.contains("allow-reachable-sha1-in-want"),
491 "Git advertisement must include allow-reachable-sha1-in-want"
492 );
493
494 Ok(())
495 })
496 .await
497 }
498 94
499 // ================================================================ 95 /// Get the relay WebSocket URL
500 // CORS SUPPORT TESTS 96 pub fn url(&self) -> &str { ... }
501 // ================================================================
502 97
503 /// MUST set Access-Control-Allow-Origin: * on ALL responses 98 /// Get the relay domain (host:port)
504 /// 99 pub fn domain(&self) -> String { ... }
505 /// Spec: GRASP-01, Line 57
506 /// > 1. Set `Access-Control-Allow-Origin: *` on ALL responses
507 async fn test_cors_allow_origin(ctx: &TestContext) -> TestResult {
508 TestResult::new(
509 "cors_allow_origin",
510 "GRASP-01:57",
511 "MUST set Access-Control-Allow-Origin: * on ALL responses",
512 )
513 .run(async {
514 let paths = vec![
515 "/",
516 "/test-npub/test-repo.git/info/refs?service=git-upload-pack",
517 ];
518
519 for path in paths {
520 let response = ctx.http_get(path).await?;
521
522 assert_eq!(
523 response.headers().get("access-control-allow-origin").unwrap(),
524 "*",
525 "Path {} must have Access-Control-Allow-Origin: *", path
526 );
527 }
528
529 Ok(())
530 })
531 .await
532 }
533 100
534 /// MUST respond to OPTIONS requests with 204 No Content 101 /// Stop the relay
535 /// 102 pub async fn stop(mut self) { ... }
536 /// Spec: GRASP-01, Line 60
537 /// > 4. Respond to OPTIONS requests with 204 No Content
538 async fn test_cors_options_request(ctx: &TestContext) -> TestResult {
539 TestResult::new(
540 "cors_options_request",
541 "GRASP-01:60",
542 "MUST respond to OPTIONS with 204 No Content",
543 )
544 .run(async {
545 let response = ctx.http_options("/test-npub/test-repo.git/info/refs").await?;
546
547 assert_eq!(
548 response.status(), 204,
549 "OPTIONS request must return 204 No Content"
550 );
551
552 Ok(())
553 })
554 .await
555 }
556} 103}
557``` 104```
558 105
559### Test Result Reporting 106### Using TestRelay in Integration Tests
107
108From [`tests/nip01_compliance.rs`](tests/nip01_compliance.rs):
560 109
561```rust 110```rust
562/// Test result with spec citation 111use common::TestRelay;
563pub struct TestResult { 112use grasp_audit::*;
564 pub name: String,
565 pub spec_ref: String, // e.g., "GRASP-01:12-13"
566 pub requirement: String, // Exact text from spec
567 pub passed: bool,
568 pub error: Option<String>,
569 pub duration: Duration,
570}
571 113
572impl TestResult { 114/// Macro to generate isolated integration tests
573 /// Create a new test result 115macro_rules! isolated_test {
574 pub fn new(name: &str, spec_ref: &str, requirement: &str) -> Self { 116 ($test_name:ident) => {
575 TestResult { 117 #[tokio::test]
576 name: name.to_string(), 118 async fn $test_name() {
577 spec_ref: spec_ref.to_string(), 119 let relay = TestRelay::start().await;
578 requirement: requirement.to_string(), 120 let config = AuditConfig::isolated();
579 passed: false, 121 let client = AuditClient::new(relay.url(), config)
580 error: None, 122 .await
581 duration: Duration::default(), 123 .expect("Failed to create audit client");
582 }
583 }
584
585 /// Run the test
586 pub async fn run<F, Fut>(mut self, test_fn: F) -> Self
587 where
588 F: FnOnce() -> Fut,
589 Fut: Future<Output = Result<(), String>>,
590 {
591 let start = Instant::now();
592
593 match test_fn().await {
594 Ok(()) => {
595 self.passed = true;
596 }
597 Err(e) => {
598 self.passed = false;
599 self.error = Some(e);
600 }
601 }
602
603 self.duration = start.elapsed();
604 self
605 }
606}
607 124
608/// Collection of test results for a spec 125 let result = specs::Nip01SmokeTests::$test_name(&client).await;
609pub struct ComplianceResult { 126
610 pub spec: String, 127 relay.stop().await;
611 pub results: Vec<TestResult>,
612}
613 128
614impl ComplianceResult { 129 assert!(
615 pub fn report(&self) -> String { 130 result.passed,
616 let mut output = String::new(); 131 "{} failed: {}",
617 132 stringify!($test_name),
618 output.push_str(&format!("\n{} Compliance Report\n", self.spec)); 133 result.error.as_deref().unwrap_or("unknown error")
619 output.push_str(&"=".repeat(60)); 134 );
620 output.push_str("\n\n");
621
622 let passed = self.results.iter().filter(|r| r.passed).count();
623 let total = self.results.len();
624
625 output.push_str(&format!("Results: {}/{} passed\n\n", passed, total));
626
627 for result in &self.results {
628 let status = if result.passed { "✓" } else { "✗" };
629
630 output.push_str(&format!(
631 "{} {} ({})\n",
632 status, result.name, result.spec_ref
633 ));
634
635 output.push_str(&format!(" Requirement: {}\n", result.requirement));
636
637 if let Some(error) = &result.error {
638 output.push_str(&format!(" Error: {}\n", error));
639 }
640
641 output.push_str(&format!(" Duration: {:?}\n\n", result.duration));
642 } 135 }
643 136 };
644 output
645 }
646} 137}
647```
648 138
649### Usage Example 139// Generate isolated tests for all NIP-01 smoke tests
650 140isolated_test!(test_websocket_connection);
651```rust 141isolated_test!(test_send_receive_event);
652// examples/test_implementation.rs 142isolated_test!(test_create_subscription);
653
654use grasp_compliance_tests::{TestContext, Grasp01Spec};
655
656#[tokio::main]
657async fn main() {
658 // Configure the implementation to test
659 let ctx = TestContext::builder()
660 .base_url("http://localhost:8080")
661 .websocket_url("ws://localhost:8080")
662 .domain("localhost:8080")
663 .build();
664
665 // Run GRASP-01 compliance tests
666 let results = Grasp01Spec::test_compliance(&ctx).await;
667
668 // Print report
669 println!("{}", results.report());
670
671 // Exit with error if any tests failed
672 if !results.all_passed() {
673 std::process::exit(1);
674 }
675}
676``` 143```
677 144
678### Integration with ngit-grasp 145### Running Integration Tests
679 146
680In `ngit-grasp/tests/compliance.rs`: 147```bash
148# Run all integration tests
149cargo test --test '*'
681 150
682```rust 151# Run specific test file
683use grasp_compliance_tests::{TestContext, Grasp01Spec}; 152cargo test --test nip01_compliance
684 153
685#[tokio::test] 154# Run with output
686async fn test_grasp_01_compliance() { 155cargo test --test nip01_compliance -- --nocapture
687 // Start test server
688 let server = start_test_server().await;
689
690 // Configure test context
691 let ctx = TestContext::builder()
692 .base_url(&server.url())
693 .websocket_url(&server.ws_url())
694 .domain(&server.domain())
695 .build();
696
697 // Run compliance tests
698 let results = Grasp01Spec::test_compliance(&ctx).await;
699
700 // Assert all tests passed
701 assert!(
702 results.all_passed(),
703 "GRASP-01 compliance failed:\n{}",
704 results.report()
705 );
706}
707``` 156```
708 157
709## Unit Testing Strategy 158## GRASP Audit Tool (grasp-audit/)
710 159
711### Git Module Tests 160### Purpose
712 161
713```rust 162The grasp-audit tool is a **reusable GRASP compliance testing library** that can:
714// src/git/parser.rs tests
715 163
716#[cfg(test)] 164- Test ngit-grasp for self-validation
717mod tests { 165- Test any other GRASP implementation (like ngit-relay)
718 use super::*; 166- Run in CI/CD for continuous compliance verification
719 167- Generate compliance reports
720 #[test]
721 fn test_parse_pkt_line() {
722 let data = b"0006a\n";
723 let (length, payload) = parse_pkt_line(data).unwrap();
724 assert_eq!(length, 6);
725 assert_eq!(payload, b"a\n");
726 }
727
728 #[test]
729 fn test_parse_flush_packet() {
730 let data = b"0000";
731 let result = parse_pkt_line(data).unwrap();
732 assert_eq!(result.0, 0);
733 }
734
735 #[test]
736 fn test_parse_ref_updates() {
737 let body = b"00820000000000000000000000000000000000000000 \
738 a1b2c3d4e5f6789012345678901234567890abcd \
739 refs/heads/main\0 report-status\n\
740 0000";
741
742 let updates = parse_ref_updates(body).unwrap();
743 assert_eq!(updates.len(), 1);
744 assert_eq!(updates[0].ref_name, "refs/heads/main");
745 }
746}
747```
748 168
749### Authorization Module Tests 169### Test Suites
750 170
751```rust 171From [`grasp-audit/src/specs/grasp01/mod.rs`](grasp-audit/src/specs/grasp01/mod.rs):
752// src/git/authorization.rs tests
753 172
754#[cfg(test)] 173| Suite | Description | Requirements |
755mod tests { 174|-------|-------------|--------------|
756 use super::*; 175| [`Nip01SmokeTests`](grasp-audit/src/specs/grasp01/nip01_smoke.rs) | Basic NIP-01 relay functionality | WebSocket only |
757 176| [`Nip11DocumentTests`](grasp-audit/src/specs/grasp01/nip11_document.rs) | NIP-11 relay information document | WebSocket only |
758 #[test] 177| [`EventAcceptancePolicyTests`](grasp-audit/src/specs/grasp01/event_acceptance_policy.rs) | Event acceptance rules | WebSocket only |
759 fn test_get_maintainers_single() { 178| [`CorsTests`](grasp-audit/src/specs/grasp01/cors.rs) | CORS headers on Git HTTP endpoints | git-data-dir |
760 let events = vec![ 179| [`GitCloneTests`](grasp-audit/src/specs/grasp01/git_clone.rs) | Git clone operations | git-data-dir |
761 create_test_announcement("alice", "repo1", vec![]), 180| [`PushAuthorizationTests`](grasp-audit/src/specs/grasp01/push_authorization.rs) | Push authorization | git-data-dir |
762 ]; 181| [`RepositoryCreationTests`](grasp-audit/src/specs/grasp01/repository_creation.rs) | Repository creation | git-data-dir |
763
764 let maintainers = get_maintainers(&events, "alice", "repo1");
765 assert_eq!(maintainers, vec!["alice"]);
766 }
767
768 #[test]
769 fn test_get_maintainers_recursive() {
770 let events = vec![
771 create_test_announcement("alice", "repo1", vec!["bob"]),
772 create_test_announcement("bob", "repo1", vec![]),
773 ];
774
775 let maintainers = get_maintainers(&events, "alice", "repo1");
776 assert!(maintainers.contains(&"alice".to_string()));
777 assert!(maintainers.contains(&"bob".to_string()));
778 }
779
780 #[test]
781 fn test_get_maintainers_circular() {
782 let events = vec![
783 create_test_announcement("alice", "repo1", vec!["bob"]),
784 create_test_announcement("bob", "repo1", vec!["alice"]),
785 ];
786
787 let maintainers = get_maintainers(&events, "alice", "repo1");
788 assert_eq!(maintainers.len(), 2);
789 }
790
791 #[test]
792 fn test_validate_state_ref_matching() {
793 let state = RepositoryState {
794 branches: HashMap::from([
795 ("main".into(), "a1b2c3d4...".into()),
796 ]),
797 tags: HashMap::new(),
798 };
799
800 let update = RefUpdate {
801 old_oid: "0000...".into(),
802 new_oid: "a1b2c3d4...".into(),
803 ref_name: "refs/heads/main".into(),
804 };
805
806 assert!(validate_state_ref(&state, &update).is_ok());
807 }
808
809 #[test]
810 fn test_validate_state_ref_mismatch() {
811 let state = RepositoryState {
812 branches: HashMap::from([
813 ("main".into(), "aaaa1111...".into()),
814 ]),
815 tags: HashMap::new(),
816 };
817
818 let update = RefUpdate {
819 old_oid: "0000...".into(),
820 new_oid: "bbbb2222...".into(),
821 ref_name: "refs/heads/main".into(),
822 };
823
824 assert!(validate_state_ref(&state, &update).is_err());
825 }
826}
827```
828 182
829## Integration Testing Strategy 183### Spec Requirements Database
830 184
831### Repository Lifecycle Tests 185From [`grasp-audit/src/specs/grasp01/spec_requirements.rs`](grasp-audit/src/specs/grasp01/spec_requirements.rs):
832 186
833```rust 187```rust
834// tests/integration/repository_lifecycle.rs 188pub struct SpecRequirement {
835 189 pub id: &'static str, // e.g., "GRASP-01:L9"
836#[tokio::test] 190 pub section: &'static str, // e.g., "Nostr Relay"
837async fn test_repository_creation_on_announcement() { 191 pub level: RequirementLevel, // MUST, SHOULD, MAY
838 let app = test_app().await; 192 pub text: &'static str, // Exact text from spec
839 193 pub line: u32, // Line number in spec
840 // Send repository announcement
841 let announcement = create_announcement()
842 .with_identifier("test-repo")
843 .with_clone_tag(app.domain())
844 .with_relay_tag(app.domain())
845 .sign()
846 .await;
847
848 app.send_event(announcement).await.unwrap();
849
850 // Wait for async processing
851 tokio::time::sleep(Duration::from_secs(1)).await;
852
853 // Verify repository was created
854 let repo_path = app.git_data_path()
855 .join(announcement.author_npub())
856 .join("test-repo.git");
857
858 assert!(repo_path.exists());
859 assert!(repo_path.join("HEAD").exists());
860 assert!(repo_path.join("config").exists());
861} 194}
862 195
863#[tokio::test] 196pub enum RequirementLevel {
864async fn test_push_validation_flow() { 197 Must,
865 let app = test_app().await; 198 Should,
866 199 May,
867 // Create repository with state
868 let (announcement, state) = app.create_repo_with_state()
869 .branch("main", "commit-sha-123")
870 .build()
871 .await;
872
873 // Attempt push matching state
874 let result = app.git_push("main", "commit-sha-123").await;
875 assert!(result.success);
876
877 // Attempt push NOT matching state
878 let result = app.git_push("main", "different-sha-456").await;
879 assert!(!result.success);
880 assert!(result.stderr.contains("state event"));
881} 200}
882``` 201```
883 202
884### Multi-Maintainer Tests 203### Automatic Cleanup Tags
204
205All audit events include cleanup tags for production safety (from [`grasp-audit/src/audit.rs`](grasp-audit/src/audit.rs)):
885 206
886```rust 207```rust
887#[tokio::test] 208// Automatically added to EVERY audit event:
888async fn test_multi_maintainer_push() { 209["t", "grasp-audit-test-event"] // Marker
889 let app = test_app().await; 210["t", "audit-{run_id}"] // Run isolation
890 211["t", "audit-cleanup-after-{unix_timestamp}"] // Cleanup time
891 // Alice creates repo, lists Bob as maintainer
892 let alice_announcement = create_announcement()
893 .author("alice")
894 .maintainers(vec!["bob"])
895 .build();
896
897 app.send_event(alice_announcement).await.unwrap();
898
899 // Bob creates state event
900 let bob_state = create_state()
901 .author("bob")
902 .branch("main", "commit-123")
903 .build();
904
905 app.send_event(bob_state).await.unwrap();
906
907 // Bob's push should succeed
908 let result = app.git_push_as("bob", "main", "commit-123").await;
909 assert!(result.success);
910}
911``` 212```
912 213
913## End-to-End Testing 214### Running grasp-audit
914 215
915### Real Git Client Tests 216**Testing the reference implementation (ngit-relay):**
916 217
917```rust 218```bash
918// tests/e2e/git_client.rs 219# Use test-ngit-relay.sh for automated relay management
919 220cd grasp-audit && nix develop -c bash test-ngit-relay.sh --mode test
920#[tokio::test]
921async fn test_real_git_clone() {
922 let app = test_app().await;
923
924 // Setup repository
925 let (announcement, _) = app.create_repo_with_commits()
926 .commit("Initial commit", "file.txt", "content")
927 .build()
928 .await;
929
930 // Clone with real git client
931 let temp_dir = TempDir::new().unwrap();
932 let clone_url = format!(
933 "http://{}/{}/{}.git",
934 app.domain(),
935 announcement.author_npub(),
936 announcement.identifier()
937 );
938
939 let output = Command::new("git")
940 .args(&["clone", &clone_url])
941 .current_dir(&temp_dir)
942 .output()
943 .await
944 .unwrap();
945
946 assert!(output.status.success());
947 assert!(temp_dir.path().join(announcement.identifier()).exists());
948}
949 221
950#[tokio::test] 222# Or manually:
951async fn test_real_git_push() { 223docker run --rm -p 18081:8081 ghcr.io/danconwaydev/ngit-relay:latest
952 let app = test_app().await; 224cd grasp-audit
953 225RELAY_URL="ws://localhost:18081" nix develop -c cargo test --lib -- --ignored --nocapture
954 // Create repository
955 let (announcement, keys) = app.create_repo().await;
956
957 // Clone it
958 let temp_dir = TempDir::new().unwrap();
959 git_clone(&app, &announcement, &temp_dir).await;
960
961 // Make changes
962 let repo_dir = temp_dir.path().join(announcement.identifier());
963 tokio::fs::write(repo_dir.join("new-file.txt"), "content").await.unwrap();
964
965 // Commit
966 git_commit(&repo_dir, "Add new file").await;
967
968 // Send state event for new commit
969 let new_commit = git_rev_parse(&repo_dir, "HEAD").await;
970 app.send_state(&announcement, "main", &new_commit, &keys).await;
971
972 // Push
973 let output = Command::new("git")
974 .args(&["push", "origin", "main"])
975 .current_dir(&repo_dir)
976 .output()
977 .await
978 .unwrap();
979
980 assert!(output.status.success());
981}
982``` 226```
983 227
984## Performance Testing 228**Testing ngit-grasp (the main project):**
985 229
986### Load Tests 230```bash
231# Integration tests use TestRelay fixture - just run:
232cargo test --test '*'
233```
987 234
988```rust 235## Test Patterns
989// tests/performance/load.rs
990 236
991#[tokio::test] 237### Isolated Test Pattern
992async fn test_concurrent_pushes() { 238
993 let app = test_app().await; 239Each test runs with its own fresh relay instance:
994
995 let num_concurrent = 100;
996 let mut handles = vec![];
997
998 for i in 0..num_concurrent {
999 let app = app.clone();
1000 let handle = tokio::spawn(async move {
1001 let (announcement, state) = app.create_repo_with_state()
1002 .branch("main", &format!("commit-{}", i))
1003 .build()
1004 .await;
1005
1006 app.git_push("main", &format!("commit-{}", i)).await
1007 });
1008 handles.push(handle);
1009 }
1010
1011 let results = futures::future::join_all(handles).await;
1012
1013 // All should succeed
1014 for result in results {
1015 assert!(result.unwrap().success);
1016 }
1017}
1018 240
241```rust
1019#[tokio::test] 242#[tokio::test]
1020async fn test_event_ingestion_throughput() { 243async fn test_something() {
1021 let app = test_app().await; 244 // Start fresh relay
1022 245 let relay = TestRelay::start().await;
1023 let num_events = 1000;
1024 let start = Instant::now();
1025
1026 for i in 0..num_events {
1027 let event = create_announcement()
1028 .with_identifier(&format!("repo-{}", i))
1029 .build();
1030 app.send_event(event).await.unwrap();
1031 }
1032 246
1033 let duration = start.elapsed(); 247 // Run test
1034 let throughput = num_events as f64 / duration.as_secs_f64(); 248 let client = AuditClient::new(relay.url(), AuditConfig::isolated()).await?;
249 // ... test logic ...
1035 250
1036 println!("Event throughput: {:.2} events/sec", throughput); 251 // Cleanup
1037 assert!(throughput > 100.0, "Throughput too low"); 252 relay.stop().await;
1038} 253}
1039``` 254```
1040 255
1041## Test Utilities 256### Macro-Based Test Generation
1042 257
1043### Test Fixtures 258For test suites that follow the same pattern, use macros:
1044 259
1045```rust 260```rust
1046// tests/common/fixtures.rs 261macro_rules! isolated_test {
1047 262 ($test_name:ident) => {
1048pub struct TestEventBuilder { 263 #[tokio::test]
1049 kind: Kind, 264 async fn $test_name() {
1050 content: String, 265 let relay = TestRelay::start().await;
1051 tags: Vec<Tag>, 266 // ... standard setup and teardown ...
1052 keys: Option<Keys>,
1053}
1054
1055impl TestEventBuilder {
1056 pub fn announcement() -> Self {
1057 TestEventBuilder {
1058 kind: Kind::RepositoryAnnouncement,
1059 content: String::new(),
1060 tags: vec![],
1061 keys: None,
1062 } 267 }
1063 } 268 };
1064
1065 pub fn with_identifier(mut self, id: &str) -> Self {
1066 self.tags.push(Tag::Identifier(id.to_string()));
1067 self
1068 }
1069
1070 pub fn with_clone_tag(mut self, url: &str) -> Self {
1071 self.tags.push(Tag::new("clone", vec![url]));
1072 self
1073 }
1074
1075 pub async fn build(self) -> Event {
1076 let keys = self.keys.unwrap_or_else(|| Keys::generate());
1077 EventBuilder::new(self.kind, self.content, self.tags)
1078 .to_event(&keys)
1079 .await
1080 .unwrap()
1081 }
1082}
1083```
1084
1085### Test Server
1086
1087```rust
1088// tests/common/server.rs
1089
1090pub struct TestServer {
1091 addr: SocketAddr,
1092 handle: JoinHandle<()>,
1093}
1094
1095impl TestServer {
1096 pub async fn start() -> Self {
1097 let config = Config {
1098 domain: "localhost:0".to_string(),
1099 git_data_path: TempDir::new().unwrap().into_path(),
1100 relay_data_path: TempDir::new().unwrap().into_path(),
1101 // ... other config
1102 };
1103
1104 let app = create_app(config).await;
1105 let listener = TcpListener::bind("127.0.0.1:0").await.unwrap();
1106 let addr = listener.local_addr().unwrap();
1107
1108 let handle = tokio::spawn(async move {
1109 axum::serve(listener, app).await.unwrap();
1110 });
1111
1112 // Wait for server to be ready
1113 tokio::time::sleep(Duration::from_millis(100)).await;
1114
1115 TestServer { addr, handle }
1116 }
1117
1118 pub fn url(&self) -> String {
1119 format!("http://{}", self.addr)
1120 }
1121
1122 pub fn ws_url(&self) -> String {
1123 format!("ws://{}", self.addr)
1124 }
1125} 269}
1126```
1127 270
1128## CI/CD Integration 271isolated_test!(test_websocket_connection);
1129 272isolated_test!(test_send_receive_event);
1130### GitHub Actions Workflow
1131
1132```yaml
1133# .github/workflows/test.yml
1134
1135name: Test
1136
1137on: [push, pull_request]
1138
1139jobs:
1140 unit-tests:
1141 runs-on: ubuntu-latest
1142 steps:
1143 - uses: actions/checkout@v3
1144 - uses: actions-rs/toolchain@v1
1145 with:
1146 toolchain: stable
1147 - name: Run unit tests
1148 run: cargo test --lib
1149
1150 integration-tests:
1151 runs-on: ubuntu-latest
1152 steps:
1153 - uses: actions/checkout@v3
1154 - uses: actions-rs/toolchain@v1
1155 with:
1156 toolchain: stable
1157 - name: Install Git
1158 run: sudo apt-get install -y git
1159 - name: Run integration tests
1160 run: cargo test --test '*'
1161
1162 compliance-tests:
1163 runs-on: ubuntu-latest
1164 steps:
1165 - uses: actions/checkout@v3
1166 - uses: actions-rs/toolchain@v1
1167 with:
1168 toolchain: stable
1169 - name: Run GRASP-01 compliance tests
1170 run: cargo test --test compliance
1171 - name: Generate compliance report
1172 run: cargo run --example compliance-report > compliance-report.txt
1173 - name: Upload compliance report
1174 uses: actions/upload-artifact@v3
1175 with:
1176 name: compliance-report
1177 path: compliance-report.txt
1178``` 273```
1179 274
1180## Test Coverage 275## Coverage Targets
1181 276
1182### Target Coverage 277| Test Type | Coverage Target |
278|-----------|-----------------|
279| Unit Tests | >80% line coverage of `src/` |
280| Integration Tests | All critical user paths |
281| GRASP-01 Compliance | 100% of MUST requirements |
1183 282
1184- **Unit Tests**: >80% line coverage 283## CI/CD Integration
1185- **Integration Tests**: All critical paths
1186- **Compliance Tests**: 100% of GRASP-01 requirements
1187- **E2E Tests**: Key user workflows
1188 284
1189### Measuring Coverage 285### Running All Tests
1190 286
1191```bash 287```bash
1192# Install tarpaulin 288# Unit tests (fast, no external dependencies)
1193cargo install cargo-tarpaulin 289cargo test --lib
1194 290
1195# Run with coverage 291# Integration tests (requires relay binary built)
1196cargo tarpaulin --out Html --output-dir coverage 292cargo build --release
293cargo test --test '*'
1197 294
1198# View report 295# Compliance tests against ngit-relay reference
1199open coverage/index.html 296cd grasp-audit && nix develop -c bash test-ngit-relay.sh --mode test
1200```
1201
1202## Documentation Testing
1203
1204### Doc Tests
1205
1206```rust
1207/// Parse a pkt-line from Git protocol
1208///
1209/// # Examples
1210///
1211/// ```
1212/// use ngit_grasp::git::parse_pkt_line;
1213///
1214/// let data = b"0006a\n";
1215/// let (length, payload) = parse_pkt_line(data).unwrap();
1216/// assert_eq!(length, 6);
1217/// assert_eq!(payload, b"a\n");
1218/// ```
1219pub fn parse_pkt_line(data: &[u8]) -> Result<(usize, &[u8])> {
1220 // implementation
1221}
1222``` 297```
1223 298
1224## Summary 299## Summary
1225 300
1226This comprehensive test strategy ensures: 301| What | Where | Purpose |
302|------|-------|---------|
303| Unit tests | `src/**/tests` modules | Test individual functions |
304| Integration tests | `tests/*.rs` | Test ngit-grasp as a whole |
305| TestRelay fixture | [`tests/common/relay.rs`](tests/common/relay.rs) | Manage relay lifecycle |
306| GRASP audit library | `grasp-audit/` | Reusable compliance testing |
307| GRASP-01 specs | [`grasp-audit/src/specs/grasp01/`](grasp-audit/src/specs/grasp01/) | Spec requirement tests |
1227 308
12281. **Spec Compliance**: Every GRASP requirement has a corresponding test 309## Related Documentation
12292. **Reusability**: Compliance tests can validate any GRASP implementation
12303. **Clear Failures**: Test failures cite exact spec lines
12314. **Comprehensive Coverage**: Unit, integration, compliance, and E2E tests
12325. **Maintainability**: Tests mirror spec structure for easy updates
1233 310
1234The compliance testing tool is a standalone crate that can be: 311- [Architecture](../explanation/architecture.md) - System design
1235- Used by ngit-grasp for self-validation 312- [GRASP-01 Implementation Learnings](../learnings/grasp-01-implementation.md) - Patterns and lessons
1236- Published for other GRASP implementations to use 313- [GRASP Audit Learnings](../learnings/grasp-audit.md) - Audit tool patterns \ No newline at end of file
1237- Updated as new GRASP specs are released (GRASP-02, GRASP-05)
1238- Run in CI/CD for continuous compliance verification
generated by cgit v1.2.3 (git 2.39.1) at 2026-06-05 11:26:59 +0000