upleb.uk

Public git repos — served from a NIP-34 GRASP relay at git.upleb.uk

summaryrefslogtreecommitdiff
path: root/COMPLIANCE_TEST_PROPOSAL.md
diff options
context:
space:
mode:
authorDanConwayDev <DanConwayDev@protonmail.com>2025-11-04 09:31:57 +0000
committerDanConwayDev <DanConwayDev@protonmail.com>2025-11-04 09:31:57 +0000
commit22557f15d6a7b77f72d4597fc05aa06346495a33 (patch)
treee31e0cdecfc4cb1e28246227a7ef295b71687b09 /COMPLIANCE_TEST_PROPOSAL.md
parentb3031800cd95601c2d9cd2d24034364d1496b073 (diff)
docs: major cleanup and reorganization
- Archive 30 completed session documents to docs/archive/ - Extract learnings to docs/learnings/ (nix-flakes, nostr-sdk, grasp-audit) - Create CURRENT_STATUS.md as single source of truth - Create AGENTS.md with documentation guidelines - Create docs/archive/README.md for archive organization - Clean root directory: 32 files → 4 files Root directory now contains only: - README.md (project overview) - AGENTS.md (documentation guidelines) - CURRENT_STATUS.md (current state) - CLEANUP_SUMMARY.md (cleanup report) All historical documents preserved in docs/archive/ with proper dating. All reusable knowledge extracted to docs/learnings/. Benefits: - Easy to find current information - Clear document lifecycle - No more documentation sprawl - Learnings are accessible and reusable - Better onboarding for new developers/agents File counts: - Root: 4 (was 32) - Permanent docs: 7 - Learnings: 3 (new) - Archive: 32 (new) - Total: 49 well-organized docs
Diffstat (limited to 'COMPLIANCE_TEST_PROPOSAL.md')
-rw-r--r--COMPLIANCE_TEST_PROPOSAL.md500
1 files changed, 0 insertions, 500 deletions
diff --git a/COMPLIANCE_TEST_PROPOSAL.md b/COMPLIANCE_TEST_PROPOSAL.md
deleted file mode 100644
index 792f375..0000000
--- a/COMPLIANCE_TEST_PROPOSAL.md
+++ /dev/null
@@ -1,500 +0,0 @@
1# GRASP Compliance Test Tool - Implementation Proposal
2
3## Executive Summary
4
5This document proposes the implementation of a **reusable GRASP compliance testing tool** as a standalone Rust crate. The first phase focuses on testing GRASP-01's requirement: "MUST serve a NIP-01 compliant nostr relay at / that accepts git repository announcements and their corresponding repo state announcements."
6
7## Key Question: How Much NIP-01 Testing Do We Need?
8
9### Analysis
10
11**NIP-01** specifies the basic Nostr protocol including:
121. Event structure and validation (id, pubkey, created_at, kind, tags, content, sig)
132. Event ID calculation (SHA256 of serialized event)
143. Signature verification (Schnorr signatures on secp256k1)
154. WebSocket message types (EVENT, REQ, CLOSE, NOTICE, OK, EOSE, CLOSED, AUTH)
165. Subscription filters
176. Message format and serialization rules
18
19**rust-nostr's `nostr-relay-builder`** already provides:
20- ✅ Full NIP-01 event validation
21- ✅ WebSocket message handling
22- ✅ Signature verification
23- ✅ Event ID validation
24- ✅ Subscription management
25- ✅ Comprehensive test suite for all of the above
26
27### Recommendation: Smoke Tests Only for NIP-01 Core
28
29**We should NOT re-test what rust-nostr already tests extensively.**
30
31Instead, we should focus on:
32
331. **Smoke Tests** (10-15 tests):
34 - WebSocket connection works
35 - Can send/receive basic EVENT messages
36 - Can create subscriptions with REQ
37 - Receive EOSE for subscriptions
38 - Basic event validation works (reject invalid events)
39 - Can close subscriptions with CLOSE
40
412. **GRASP-Specific Tests** (majority of effort):
42 - Accepts NIP-34 repository announcements (kind 30617)
43 - Accepts NIP-34 repository state events (kind 30618)
44 - Rejects announcements without required clone/relay tags
45 - Accepts events that tag accepted announcements
46 - NIP-11 document has GRASP-specific fields
47 - Repository creation triggered by announcements
48 - State events update repository HEAD
49
50**Rationale:**
51- rust-nostr has 1000+ tests for NIP-01 compliance
52- We're using their relay builder, not implementing NIP-01 from scratch
53- Our value-add is GRASP protocol logic, not Nostr basics
54- Testing what's already tested wastes time and creates maintenance burden
55- Focus on integration points and GRASP-specific behavior
56
57## Proposed Test Structure
58
59### Phase 1: Exportable Test Tool Foundation
60
61Create `grasp-compliance-tests/` as a standalone crate that can be:
62- Used by ngit-grasp
63- Published for other GRASP implementations
64- Run against any GRASP service (Go, Rust, Python, etc.)
65
66### Directory Structure
67
68```
69grasp-compliance-tests/
70├── Cargo.toml
71├── README.md
72├── src/
73│ ├── lib.rs # Public API
74│ ├── client.rs # HTTP/WebSocket/Git test clients
75│ ├── assertions.rs # Spec-based assertions
76│ ├── fixtures.rs # Event/repo builders
77│ └── specs/
78│ ├── mod.rs # Spec registry
79│ ├── nip01_smoke.rs # Minimal NIP-01 smoke tests
80│ └── grasp_01.rs # GRASP-01 compliance tests
81├── fixtures/
82│ ├── repos/ # Test git repositories
83│ ├── events/ # Nostr event JSON fixtures
84│ └── keys/ # Test keypairs (deterministic)
85└── examples/
86 └── test_server.rs # Example: test any GRASP server
87```
88
89## Test Breakdown: GRASP-01 First Requirement
90
91**Requirement:** "MUST serve a NIP-01 compliant nostr relay at / that accepts git repository announcements and their corresponding repo state announcements."
92
93### Proposed Tests (18 total)
94
95#### NIP-01 Smoke Tests (6 tests)
96
97These verify basic Nostr relay functionality:
98
991. **websocket_connection**
100 - Spec: NIP-01 basic requirement
101 - Test: Can establish WebSocket connection to `/`
102 - Assertion: Upgrade successful, connection stays open
103
1042. **send_receive_event**
105 - Spec: NIP-01 EVENT message
106 - Test: Send valid EVENT, receive OK response
107 - Assertion: OK response with event ID
108
1093. **create_subscription**
110 - Spec: NIP-01 REQ message
111 - Test: Send REQ with filters, receive EOSE
112 - Assertion: EOSE received for subscription ID
113
1144. **close_subscription**
115 - Spec: NIP-01 CLOSE message
116 - Test: Send CLOSE, verify subscription closed
117 - Assertion: No more events for closed subscription
118
1195. **reject_invalid_event**
120 - Spec: NIP-01 event validation
121 - Test: Send event with invalid signature
122 - Assertion: OK response with ok=false
123
1246. **reject_invalid_event_id**
125 - Spec: NIP-01 event ID validation
126 - Test: Send event with wrong ID
127 - Assertion: OK response with ok=false, error message
128
129#### GRASP-01 Specific Tests (12 tests)
130
131These verify GRASP protocol requirements:
132
1337. **accepts_repository_announcement**
134 - Spec: GRASP-01:9-10
135 - Test: Send NIP-34 kind 30617 with clone/relay tags
136 - Assertion: Event accepted (OK with ok=true)
137
1388. **accepts_repository_state**
139 - Spec: GRASP-01:9-10
140 - Test: Send NIP-34 kind 30618 state event
141 - Assertion: Event accepted
142
1439. **rejects_announcement_without_clone_tag**
144 - Spec: GRASP-01:12-13
145 - Test: Send announcement missing clone tag for this service
146 - Assertion: Event rejected with descriptive error
147
14810. **rejects_announcement_without_relay_tag**
149 - Spec: GRASP-01:12-13
150 - Test: Send announcement missing relay tag for this service
151 - Assertion: Event rejected with descriptive error
152
15311. **accepts_announcement_with_multiple_clones**
154 - Spec: GRASP-01:12-13 (inverse - should accept if listed)
155 - Test: Announcement with multiple clone URLs including ours
156 - Assertion: Event accepted
157
15812. **accepts_events_tagging_announcement**
159 - Spec: GRASP-01:17-20
160 - Test: Send issue (kind 1621) tagging accepted announcement
161 - Assertion: Event accepted
162
16313. **accepts_events_tagged_by_announcement**
164 - Spec: GRASP-01:17-20
165 - Test: Send event that announcement tags
166 - Assertion: Event accepted
167
16814. **rejects_events_tagging_rejected_announcement**
169 - Spec: GRASP-01:17-20 (inverse)
170 - Test: Send issue tagging announcement we rejected
171 - Assertion: Event rejected
172
17315. **query_announcements_by_identifier**
174 - Spec: GRASP-01 (implied - must be queryable)
175 - Test: REQ filter for kind 30617, specific identifier
176 - Assertion: Can retrieve accepted announcements
177
17816. **query_state_events**
179 - Spec: GRASP-01 (implied - must be queryable)
180 - Test: REQ filter for kind 30618
181 - Assertion: Can retrieve state events
182
18317. **state_replaces_previous**
184 - Spec: NIP-01 replaceable events
185 - Test: Send two state events with same d-tag
186 - Assertion: Only latest state returned in queries
187
18818. **concurrent_event_submission**
189 - Spec: General reliability
190 - Test: Send 100 events concurrently
191 - Assertion: All valid events accepted, no race conditions
192
193## Can We Reuse rust-nostr Tests?
194
195### Direct Reuse: No
196
197We cannot directly import rust-nostr's test suite because:
1981. Their tests are internal to their crates
1992. They test library functions, not running servers
2003. They don't test GRASP-specific behavior
201
202### Indirect Reuse: Yes
203
204We can learn from their test patterns:
205
2061. **Event Building Patterns**: Use similar builder patterns from `nostr-sdk`
207 ```rust
208 use nostr_sdk::prelude::*;
209
210 let event = EventBuilder::new(Kind::Custom(30617), "", [
211 Tag::identifier("my-repo"),
212 Tag::custom(TagKind::Custom("clone".into()), vec![domain]),
213 ])
214 .to_event(&keys)?;
215 ```
216
2172. **Assertion Helpers**: Adapt their validation logic
218 ```rust
219 // They test event.verify() - we test server accepts it
220 assert!(event.verify().is_ok()); // Their test
221 assert!(server.send_event(event).await?.ok); // Our test
222 ```
223
2243. **Test Fixtures**: Use their event generation utilities
225 ```rust
226 use nostr_sdk::Keys;
227
228 // Generate deterministic test keys (same as they do)
229 let keys = Keys::from_mnemonic("test seed phrase", None)?;
230 ```
231
232### What We Leverage from rust-nostr
233
234Since we're using `nostr-relay-builder`, we get:
235- ✅ Event validation (don't need to test)
236- ✅ Signature verification (don't need to test)
237- ✅ WebSocket handling (smoke test only)
238- ✅ Subscription management (smoke test only)
239
240We focus on testing:
241- 🎯 GRASP policy enforcement (our code)
242- 🎯 Repository announcement acceptance (our code)
243- 🎯 Integration between Nostr relay and Git service (our code)
244
245## Implementation Plan
246
247### Step 1: Create Standalone Crate (Week 1)
248
249```bash
250# Create the compliance test crate
251cargo new --lib grasp-compliance-tests
252cd grasp-compliance-tests
253```
254
255**Dependencies:**
256```toml
257[dependencies]
258nostr-sdk = "0.43"
259tokio = { version = "1", features = ["full"] }
260tokio-tungstenite = "0.21" # WebSocket client
261reqwest = { version = "0.11", features = ["json"] }
262serde = { version = "1", features = ["derive"] }
263serde_json = "1"
264anyhow = "1"
265thiserror = "1"
266
267[dev-dependencies]
268tokio-test = "0.4"
269```
270
271### Step 2: Implement Test Client (Week 1)
272
273```rust
274// src/client.rs
275
276pub struct GraspTestClient {
277 http_client: reqwest::Client,
278 base_url: String,
279 ws_url: String,
280}
281
282impl GraspTestClient {
283 pub fn new(base_url: &str) -> Self { /* ... */ }
284
285 pub async fn websocket_connect(&self) -> Result<WebSocketClient> { /* ... */ }
286
287 pub async fn send_event(&self, event: Event) -> Result<OkResponse> { /* ... */ }
288
289 pub async fn subscribe(&self, filters: Vec<Filter>) -> Result<Subscription> { /* ... */ }
290
291 pub async fn fetch_nip11(&self) -> Result<RelayInformationDocument> { /* ... */ }
292}
293```
294
295### Step 3: Implement NIP-01 Smoke Tests (Week 1)
296
297```rust
298// src/specs/nip01_smoke.rs
299
300pub async fn test_nip01_smoke(client: &GraspTestClient) -> ComplianceResult {
301 let mut results = ComplianceResult::new("NIP-01 Smoke Tests");
302
303 results.add(test_websocket_connection(client).await);
304 results.add(test_send_receive_event(client).await);
305 results.add(test_create_subscription(client).await);
306 results.add(test_close_subscription(client).await);
307 results.add(test_reject_invalid_event(client).await);
308 results.add(test_reject_invalid_event_id(client).await);
309
310 results
311}
312```
313
314### Step 4: Implement GRASP-01 Tests (Week 2)
315
316```rust
317// src/specs/grasp_01.rs
318
319pub async fn test_grasp_01_relay_requirements(
320 client: &GraspTestClient
321) -> ComplianceResult {
322 let mut results = ComplianceResult::new("GRASP-01: Relay Requirements");
323
324 results.add(test_accepts_repository_announcement(client).await);
325 results.add(test_accepts_repository_state(client).await);
326 results.add(test_rejects_announcement_without_clone_tag(client).await);
327 // ... etc
328
329 results
330}
331```
332
333### Step 5: Create Fixtures and Builders (Week 2)
334
335```rust
336// src/fixtures.rs
337
338pub struct AnnouncementBuilder {
339 keys: Keys,
340 identifier: String,
341 clone_urls: Vec<String>,
342 relay_urls: Vec<String>,
343 maintainers: Vec<String>,
344}
345
346impl AnnouncementBuilder {
347 pub fn new(identifier: &str) -> Self { /* ... */ }
348
349 pub fn with_clone(mut self, url: &str) -> Self {
350 self.clone_urls.push(url.to_string());
351 self
352 }
353
354 pub fn with_relay(mut self, url: &str) -> Self {
355 self.relay_urls.push(url.to_string());
356 self
357 }
358
359 pub async fn build(self) -> Result<Event> {
360 EventBuilder::new(Kind::Custom(30617), "", [
361 Tag::identifier(&self.identifier),
362 // Add clone tags
363 // Add relay tags
364 // Add maintainer tags
365 ])
366 .to_event(&self.keys)
367 }
368}
369```
370
371## Example Usage
372
373```rust
374// examples/test_server.rs
375
376use grasp_compliance_tests::*;
377
378#[tokio::main]
379async fn main() -> Result<()> {
380 // Test any GRASP implementation
381 let client = GraspTestClient::new("http://localhost:8080");
382
383 // Run NIP-01 smoke tests
384 println!("Running NIP-01 smoke tests...");
385 let nip01_results = test_nip01_smoke(&client).await;
386 nip01_results.print_report();
387
388 // Run GRASP-01 relay tests
389 println!("\nRunning GRASP-01 relay tests...");
390 let grasp01_results = test_grasp_01_relay_requirements(&client).await;
391 grasp01_results.print_report();
392
393 // Exit with error if any failed
394 if !nip01_results.all_passed() || !grasp01_results.all_passed() {
395 std::process::exit(1);
396 }
397
398 Ok(())
399}
400```
401
402## Test Output Format
403
404```
405GRASP-01: Relay Requirements
406════════════════════════════════════════════════════════════
407
408✓ accepts_repository_announcement (GRASP-01:9-10)
409 Requirement: MUST accept NIP-34 repository announcements
410 Duration: 45ms
411
412✓ accepts_repository_state (GRASP-01:9-10)
413 Requirement: MUST accept NIP-34 repository state events
414 Duration: 32ms
415
416✗ rejects_announcement_without_clone_tag (GRASP-01:12-13)
417 Requirement: MUST reject announcements without clone tag
418 Error: Event was accepted but should have been rejected
419 Expected: OK response with ok=false
420 Got: OK response with ok=true
421 Duration: 28ms
422
423Results: 2/3 passed (66.7%)
424
425━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
426
427Overall: 17/18 tests passed (94.4%)
428```
429
430## Benefits of This Approach
431
4321. **Focused Testing**: Test GRASP-specific behavior, not generic Nostr
4332. **Reusable Tool**: Any GRASP implementation can use this
4343. **Clear Failures**: Failures cite exact spec requirements
4354. **Maintainable**: Only 18 tests instead of 100+ redundant tests
4365. **Fast**: Smoke tests run in seconds, not minutes
4376. **Exportable**: Can be published as `grasp-compliance-tests` crate
438
439## Questions for You
440
4411. **Scope Confirmation**: Do you agree we should do smoke tests for NIP-01 rather than comprehensive testing?
442
4432. **Test Count**: Are 18 tests (6 smoke + 12 GRASP-specific) sufficient for the first requirement?
444
4453. **Implementation Order**: Should we:
446 - a) Build the test tool first, then implement ngit-grasp to pass it?
447 - b) Build them in parallel?
448 - c) Start with minimal ngit-grasp, then add tests?
449
4504. **Fixture Strategy**: Should we use:
451 - a) Deterministic test keys (same keys every run)?
452 - b) Random keys (new keys each run)?
453 - c) Configurable (support both)?
454
4555. **Integration**: Should the compliance tests:
456 - a) Be a separate crate from day one?
457 - b) Start in ngit-grasp, extract later?
458 - c) Hybrid (some tests in both places)?
459
460## Recommended Next Steps
461
462**Option A: Test-First Approach (Recommended)**
4631. Create `grasp-compliance-tests/` crate
4642. Implement all 18 tests (they will all fail)
4653. Implement ngit-grasp to pass tests
4664. Iterate until all tests pass
467
468**Option B: Parallel Development**
4691. Create minimal ngit-grasp skeleton
4702. Create test tool in parallel
4713. Wire them together
4724. Fix failing tests
473
474**Option C: Implementation-First**
4751. Build ngit-grasp based on architecture docs
4762. Create tests to verify it works
4773. Extract tests to standalone crate
478
479I recommend **Option A** because:
480- Tests serve as executable specification
481- Forces us to think through edge cases
482- Tests are reusable immediately
483- TDD approach ensures testability
484
485## Timeline Estimate
486
487- **Week 1**: Test tool foundation + NIP-01 smoke tests
488- **Week 2**: GRASP-01 relay tests + fixtures
489- **Week 3**: Integration with ngit-grasp skeleton
490- **Week 4**: Iterate until all tests pass
491
492Total: **4 weeks** to prove the concept with working tests and passing implementation.
493
494---
495
496**Ready to proceed?** Please advise on:
4971. Approach (A, B, or C)
4982. Any changes to test scope
4993. Priority of specific tests
5004. Any additional tests you want included