upleb.uk

Public git repos — served from a NIP-34 GRASP relay at git.upleb.uk

summaryrefslogtreecommitdiff
path: root/grasp-audit/README.md
diff options
context:
space:
mode:
Diffstat (limited to 'grasp-audit/README.md')
-rw-r--r--grasp-audit/README.md313
1 files changed, 149 insertions, 164 deletions
diff --git a/grasp-audit/README.md b/grasp-audit/README.md
index 0ff3d71..4d2401f 100644
--- a/grasp-audit/README.md
+++ b/grasp-audit/README.md
@@ -4,7 +4,7 @@ A reusable audit and compliance testing tool for GRASP protocol implementations.
4 4
5## Features 5## Features
6 6
7- ✅ **Shared Fixtures**: Fixtures cached and reused across tests (default for CLI) 7- ✅ **Shared Fixtures**: Fixtures cached and reused across tests (default for CLI) to reduce rate-limitting
8- ✅ **Isolated Testing**: Fresh fixtures per test for parallel test isolation 8- ✅ **Isolated Testing**: Fresh fixtures per test for parallel test isolation
9- ✅ **Clean Audit Events**: Special tags for easy cleanup (no deletion trails) 9- ✅ **Clean Audit Events**: Special tags for easy cleanup (no deletion trails)
10- ✅ **Spec-Mirrored Tests**: Test structure matches GRASP protocol exactly 10- ✅ **Spec-Mirrored Tests**: Test structure matches GRASP protocol exactly
@@ -12,27 +12,37 @@ A reusable audit and compliance testing tool for GRASP protocol implementations.
12 12
13## Quick Start 13## Quick Start
14 14
15The fastest way to run GRASP-01 compliance tests: 15Run GRASP compliance tests against any GRASP relay:
16 16
17```bash 17```bash
18# Run the test suite against ngit-relay 18# Install
19cd grasp-audit 19cd grasp-audit
20nix develop -c bash test-ngit-relay.sh --mode test 20cargo install --path .
21``` 21
22# Audit a production relay
23grasp-audit audit --relay wss://relay.ngit.dev
22 24
23This automatically: 25# Or audit a local development relay
26grasp-audit audit --relay ws://localhost:7000
27```
24 28
25- ✅ Starts ngit-relay in an isolated Docker container 29## Usage Examples
26- ✅ Runs all GRASP-01 compliance tests
27- ✅ Cleans up resources when finished
28 30
29For more options: 31### As a CLI Tool
30 32
31```bash 33```bash
32./test-ngit-relay.sh --help 34# Install
33``` 35cargo install --path .
34 36
35## Usage Examples 37# Audit a production GRASP relay (shared fixtures - default)
38grasp-audit audit --relay wss://relay.ngit.dev
39
40# Audit local development relay
41grasp-audit audit --relay ws://localhost:7000 --spec nip01-smoke
42
43# Run with isolated fixtures (for testing/debugging)
44grasp-audit audit --relay ws://localhost:7000 --mode isolated --spec push-auth
45```
36 46
37### As a Library 47### As a Library
38 48
@@ -41,8 +51,9 @@ use grasp_audit::*;
41 51
42#[tokio::main] 52#[tokio::main]
43async fn main() -> Result<()> { 53async fn main() -> Result<()> {
44 // Create audit client with shared fixtures (default for CLI) 54 // Create audit client with isolated fixtures (recommended for library use)
45 let config = AuditConfig::shared(); 55 let config = AuditConfig::isolated();
56 // let config = AuditConfig::shared(); // Alternative: shared fixtures
46 let client = AuditClient::new("ws://localhost:7000", config).await?; 57 let client = AuditClient::new("ws://localhost:7000", config).await?;
47 58
48 // Run NIP-01 smoke tests 59 // Run NIP-01 smoke tests
@@ -57,24 +68,25 @@ async fn main() -> Result<()> {
57} 68}
58``` 69```
59 70
60### As a CLI Tool 71## Test Specifications
61 72
62```bash 73The audit tool provides **good test coverage of GRASP-01** requirements, with additional smoke tests for basic Nostr relay functionality and git over HTTP.
63# Install
64cargo install --path .
65 74
66# Run smoke tests against local relay (shared fixtures - default) 75### GRASP-01 Tests
67grasp-audit audit --relay ws://localhost:7000 --spec nip01-smoke
68 76
69# Run with isolated fixtures (each test gets fresh fixtures) 77Test coverage of GRASP-01 specification:
70grasp-audit audit --relay ws://localhost:7000 --mode isolated --spec all
71```
72 78
73## Test Specifications 79- Repository announcement acceptance
80- State event handling
81- Push authorization (owner, maintainer, recursive maintainer)
82- Event acceptance policy
83- Git clone over HTTP
84- CORS headers
85- NIP-11 relay information document
74 86
75### NIP-01 Smoke Tests (6 tests) 87### NIP-01 Smoke Tests (6 tests)
76 88
77Basic Nostr relay functionality: 89Basic Nostr relay functionality validation:
78 90
791. `websocket_connection` - Can connect to / 911. `websocket_connection` - Can connect to /
802. `send_receive_event` - Can send EVENT, get OK 922. `send_receive_event` - Can send EVENT, get OK
@@ -85,12 +97,60 @@ Basic Nostr relay functionality:
85 97
86**Why only smoke tests?** rust-nostr already has 1000+ tests for NIP-01 compliance. We focus on GRASP-specific behavior. 98**Why only smoke tests?** rust-nostr already has 1000+ tests for NIP-01 compliance. We focus on GRASP-specific behavior.
87 99
88### GRASP-01 Tests (Coming Soon) 100### Git over HTTP Smoke Tests
89 101
90- Repository announcement acceptance 102Basic validation that git clone works over HTTP.
91- State event handling 103
92- Policy enforcement 104## Fixture Modes
93- And more... 105
106The audit tool supports two fixture caching modes that control how test prerequisites are managed. This is a key feature for controlling test isolation and resource efficiency.
107
108### Shared Mode (Default for CLI)
109
110**Default for CLI usage.** Fixtures are cached and reused across all tests for efficiency.
111
112Use this when:
113
114- Auditing production or development relays
115
116```bash
117# CLI uses shared mode by default
118grasp-audit audit --relay wss://relay.ngit.dev
119```
120
121```rust
122let config = AuditConfig::shared();
123```
124
125### Isolated Mode (Recommended for Library)
126
127**Recommended for library/test usage.** Each test creates fresh fixtures for complete isolation.
128
129Use this when:
130
131- Using grasp-audit as a library
132- Running `cargo test` in parallel
133- Tests must not interfere with each other
134- Debugging test failures
135
136```bash
137# Use isolated mode explicitly
138grasp-audit audit --relay ws://localhost:7000 --mode isolated
139```
140
141```rust
142let config = AuditConfig::isolated();
143```
144
145### When to Use Each Mode
146
147| Scenario | Recommended Mode |
148| ----------------------------- | ---------------- |
149| CLI auditing production relay | Shared (default) |
150| CLI auditing local relay | Shared (default) |
151| Library usage / `cargo test` | Isolated |
152| CI/CD pipeline | Isolated |
153| Debugging a single test | Isolated |
94 154
95## Audit Event Strategy 155## Audit Event Strategy
96 156
@@ -123,106 +183,59 @@ All audit events automatically include special tags for isolation and cleanup:
123- **No deletion trails**: No NIP-09 deletion events needed 183- **No deletion trails**: No NIP-09 deletion events needed
124- **Discovery**: Easy to query all audit events via hashtag 184- **Discovery**: Easy to query all audit events via hashtag
125 185
126## Fixture Modes 186## Architecture
127
128The audit tool supports two fixture caching modes that control how test prerequisites are managed.
129
130### Shared Mode (Default for CLI)
131
132Fixtures are cached and reused across all tests. Use this when:
133- Running the CLI audit tool sequentially
134- Tests build on each other's fixtures
135- You want efficient resource usage
136
137```rust
138let config = AuditConfig::shared();
139```
140
141### Isolated Mode (For Parallel Tests)
142
143Each test creates fresh fixtures for complete isolation. Use this when:
144- Running `cargo test` in parallel
145- Tests must not interfere with each other
146- Testing the fixture system itself
147
148```rust
149let config = AuditConfig::isolated();
150```
151
152### CLI Usage
153
154```bash
155# Default: shared fixtures (efficient for sequential CLI runs)
156grasp-audit audit --relay ws://localhost:7000 --spec all
157
158# Isolated fixtures (for testing)
159grasp-audit audit --relay ws://localhost:7000 --mode isolated --spec all
160```
161
162## Examples
163
164See `examples/` directory:
165 187
166```bash
167# Simple audit example
168cargo run --example simple_audit
169``` 188```
170 189grasp-audit/
171## Testing 190├── src/
172 191│ ├── lib.rs # Public API
173> **TL;DR:** See the [Quick Start](#quick-start) section for the fastest way to run tests. 192│ ├── audit.rs # Audit config and event tagging
174 193│ ├── client.rs # Audit client
175### Unit Tests 194│ ├── fixtures.rs # TestContext and FixtureKind
176 195│ ├── result.rs # Test result types
177```bash 196│ ├── isolation.rs # Test isolation utilities
178# Enter dev environment (NixOS) 197│ └── specs/
179nix develop 198│ ├── mod.rs
180 199│ ├── nip01_smoke.rs # NIP-01 smoke tests
181# Run unit tests (no relay required) 200│ └── grasp01/ # GRASP-01 compliance tests
182cargo test 201└── bin/
202 └── grasp-audit.rs # CLI tool
183``` 203```
184 204
185### Integration Tests 205## Roadmap
186 206
187The recommended approach is [`test-ngit-relay.sh`](test-ngit-relay.sh), which handles all relay lifecycle management automatically. 207Planned features and improvements:
188 208
189See the [Quick Start](#quick-start) section for common usage patterns. 209### Near-term
190 210
191**Advanced: Manual Relay Setup** 211- [ ] **Configurable backoffs for rate limiting** - Allow configuring retry delays when relays rate-limit requests
212- [ ] **Delete events per pubkey** - Send NIP-09 deletion events grouped by pubkey for better cleanup on relays that support it
213- [ ] **Delete event handling** - Respect NIP-09 support flagged in NIP-11 relay information document
192 214
193<details> 215### Future
194<summary>Click to expand manual testing instructions</summary>
195 216
196For advanced use cases where you need direct control over the relay: 217- [ ] **GRASP-05 support** - Add test coverage for GRASP-05 specification
197 218
198```bash 219### Out of Scope
199# Start relay on a specific port (example uses 18081)
200docker run --rm -p 18081:8081 ghcr.io/danconwaydev/ngit-relay:latest
201
202# In another terminal, run tests with RELAY_URL
203grasp-audit audit --relay ws://localhost:18081 --mode ci
204 220
205# or run all ignored tests via cargo 221- **GRASP-02 (Proactive Sync)** - Testing proactive synchronization behavior is inherently difficult due to its asynchronous nature and reliance on external state. This specification is out of scope for automated compliance testing.
206RELAY_URL="ws://localhost:18081" cargo test --lib -- --ignored --nocapture
207 222
208# or run specific test via cargo 223## Development
209RELAY_URL="ws://localhost:18081" cargo test --lib test_grasp01_nostr_relay_against_relay -- --ignored --nocapture
210```
211 224
212</details> 225This section covers patterns and guidelines for contributing new audit tests.
213 226
214## Test Design Pattern: Fixture-First 227### Test Design Pattern: Fixture-First
215 228
216To prevent rate-limiting from production relays during testing, we use a **fixture-first** approach that minimizes relay interactions. 229To prevent rate-limiting from production relays during testing, we use a **fixture-first** approach that minimizes relay interactions.
217 230
218### Quick Start for New Tests 231#### Quick Start for New Tests
219 232
2201. Create TestContext at test start 2331. Create TestContext at test start
2212. Get prerequisites via `ctx.get_fixture(FixtureKind::...)` 2342. Get prerequisites via `ctx.get_fixture(FixtureKind::...)`
2223. Build test-specific events using fixtures as base 2353. Build test-specific events using fixtures as base
2234. Verify outcomes via `send_and_verify_accepted/rejected` 2364. Verify outcomes via `send_and_verify_accepted/rejected`
224 237
225### Pattern Template 238#### Pattern Template
226 239
227```rust 240```rust
228pub async fn test_something(client: &AuditClient) -> TestResult { 241pub async fn test_something(client: &AuditClient) -> TestResult {
@@ -246,7 +259,7 @@ pub async fn test_something(client: &AuditClient) -> TestResult {
246} 259}
247``` 260```
248 261
249### Three-Layer Architecture 262#### Three-Layer Architecture
250 263
251``` 264```
252┌─────────────────────────────────────────────────────────────────┐ 265┌─────────────────────────────────────────────────────────────────┐
@@ -262,35 +275,35 @@ pub async fn test_something(client: &AuditClient) -> TestResult {
262└─────────────────────────────────────────────────────────────────┘ 275└─────────────────────────────────────────────────────────────────┘
263``` 276```
264 277
265### Available Fixtures 278#### Available Fixtures
266 279
267| FixtureKind | Provides | Use When | 280| FixtureKind | Provides | Use When |
268| ----------- | -------- | -------- | 281| ------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------ |
269| `ValidRepo` | Accepted repo announcement (kind 30617). Signed by owner keys, lists maintainer in maintainers tag. | Need a repo as prerequisite | 282| `ValidRepo` | Accepted repo announcement (kind 30617). Signed by owner keys, lists maintainer in maintainers tag. | Need a repo as prerequisite |
270| `RepoWithIssue` | Repo + accepted issue (kind 1621) | Testing issue-dependent events | 283| `RepoWithIssue` | Repo + accepted issue (kind 1621) | Testing issue-dependent events |
271| `RepoWithComment` | Repo + issue + comment (kind 1111) | Testing comment-dependent events | 284| `RepoWithComment` | Repo + issue + comment (kind 1111) | Testing comment-dependent events |
272| `RepoState` | Repo + state event (kind 30618). Signed by owner, points to `DETERMINISTIC_COMMIT_HASH`. | Testing owner state events | 285| `RepoState` | Repo + state event (kind 30618). Signed by owner, points to `DETERMINISTIC_COMMIT_HASH`. | Testing owner state events |
273| `PREvent` | Repo + PR event (kind 1618). Signed by PR author, points to `PR_TEST_COMMIT_HASH`. | Testing PR-dependent events | 286| `PREvent` | Repo + PR event (kind 1618). Signed by PR author, points to `PR_TEST_COMMIT_HASH`. | Testing PR-dependent events |
274| `PREventGenerated` | PR event built but NOT sent to relay. | Need PR event ID before publishing | 287| `PREventGenerated` | PR event built but NOT sent to relay. | Need PR event ID before publishing |
275| `PRWrongCommitPushedBeforeEvent` | Wrong commit pushed to `refs/nostr/<pr-event-id>` before PR event sent. Returns unsent PR event. | Testing pre-event ref cleanup | 288| `PRWrongCommitPushedBeforeEvent` | Wrong commit pushed to `refs/nostr/<pr-event-id>` before PR event sent. Returns unsent PR event. | Testing pre-event ref cleanup |
276| `PREventSentAfterWrongPush` | PR event sent after wrong commit was pushed. Tests cleanup behavior. | Testing post-event ref cleanup | 289| `PREventSentAfterWrongPush` | PR event sent after wrong commit was pushed. Tests cleanup behavior. | Testing post-event ref cleanup |
277| `OwnerStateDataPushed` | Full owner push flow: state event + git data pushed. Points to `DETERMINISTIC_COMMIT_HASH`. | Testing owner push authorization | 290| `OwnerStateDataPushed` | Full owner push flow: state event + git data pushed. Points to `DETERMINISTIC_COMMIT_HASH`. | Testing owner push authorization |
278| `MaintainerStateDataPushed` | Full maintainer push flow: force-pushes over owner's data. Points to `MAINTAINER_DETERMINISTIC_COMMIT_HASH`. | Testing maintainer push authorization | 291| `MaintainerStateDataPushed` | Full maintainer push flow: force-pushes over owner's data. Points to `MAINTAINER_DETERMINISTIC_COMMIT_HASH`. | Testing maintainer push authorization |
279| `RecursiveMaintainerStateDataPushed` | Full recursive maintainer push flow: Owner → Maintainer → RecursiveMaintainer chain. Points to `RECURSIVE_MAINTAINER_DETERMINISTIC_COMMIT_HASH`. | Testing recursive maintainer authorization | 292| `RecursiveMaintainerStateDataPushed` | Full recursive maintainer push flow: Owner → Maintainer → RecursiveMaintainer chain. Points to `RECURSIVE_MAINTAINER_DETERMINISTIC_COMMIT_HASH`. | Testing recursive maintainer authorization |
280| `HeadSetToDevelopBranch` | State event with HEAD=refs/heads/develop. Depends on RecursiveMaintainerStateDataPushed. | Testing HEAD branch switching | 293| `HeadSetToDevelopBranch` | State event with HEAD=refs/heads/develop. Depends on RecursiveMaintainerStateDataPushed. | Testing HEAD branch switching |
281 294
282### Deterministic Commit Hashes 295#### Deterministic Commit Hashes
283 296
284Fixtures use deterministic commit hashes for reproducible testing: 297Fixtures use deterministic commit hashes for reproducible testing:
285 298
286| Constant | Hash | Used By | 299| Constant | Hash | Used By |
287| -------- | ---- | ------- | 300| ------------------------------------------------ | ------------------------------------------ | ------------------------------------------------ |
288| `DETERMINISTIC_COMMIT_HASH` | `64ea71d79a57a7acb334cd9651f8aec067c0ce5d` | Owner fixtures (RepoState, OwnerStateDataPushed) | 301| `DETERMINISTIC_COMMIT_HASH` | `64ea71d79a57a7acb334cd9651f8aec067c0ce5d` | Owner fixtures (RepoState, OwnerStateDataPushed) |
289| `MAINTAINER_DETERMINISTIC_COMMIT_HASH` | `1c2d472c9b71ed51968a66500281a3c4a6840464` | MaintainerStateDataPushed | 302| `MAINTAINER_DETERMINISTIC_COMMIT_HASH` | `1c2d472c9b71ed51968a66500281a3c4a6840464` | MaintainerStateDataPushed |
290| `RECURSIVE_MAINTAINER_DETERMINISTIC_COMMIT_HASH` | `05939b82de66fbdb9c077d0a64fc68522f3cb8e0` | RecursiveMaintainerStateDataPushed | 303| `RECURSIVE_MAINTAINER_DETERMINISTIC_COMMIT_HASH` | `05939b82de66fbdb9c077d0a64fc68522f3cb8e0` | RecursiveMaintainerStateDataPushed |
291| `PR_TEST_COMMIT_HASH` | `5d40fb1555a0c28bf4d650515a73aaa54d4d9bfb` | PR fixtures (PREvent, PREventGenerated) | 304| `PR_TEST_COMMIT_HASH` | `5d40fb1555a0c28bf4d650515a73aaa54d4d9bfb` | PR fixtures (PREvent, PREventGenerated) |
292 305
293### Fixture Dependencies 306#### Fixture Dependencies
294 307
295Fixtures automatically resolve their dependencies: 308Fixtures automatically resolve their dependencies:
296 309
@@ -306,7 +319,7 @@ ValidRepo (base)
306 └── HeadSetToDevelopBranch 319 └── HeadSetToDevelopBranch
307``` 320```
308 321
309### Fixture Lifecycle: Generate → Send → Verify → DataPushed 322#### Fixture Lifecycle: Generate → Send → Verify → DataPushed
310 323
311Every fixture follows a lifecycle (some stop earlier): 324Every fixture follows a lifecycle (some stop earlier):
312 325
@@ -319,7 +332,7 @@ Caching happens after the fixture completes - same fixture request returns cache
319 332
320**Note:** Some fixtures handle their own event sending (e.g., `OwnerStateDataPushed`, `MaintainerStateDataPushed`). These are marked with `sends_own_events() -> true`. 333**Note:** Some fixtures handle their own event sending (e.g., `OwnerStateDataPushed`, `MaintainerStateDataPushed`). These are marked with `sends_own_events() -> true`.
321 334
322### How TestContext Correlates Events 335#### How TestContext Correlates Events
323 336
324Each TestContext shares a `run_id` with all events: 337Each TestContext shares a `run_id` with all events:
325 338
@@ -336,7 +349,7 @@ This enables:
336- Production relay cleanup scripts 349- Production relay cleanup scripts
337- Test isolation between runs 350- Test isolation between runs
338 351
339### When NOT to Use Fixtures 352#### When NOT to Use Fixtures
340 353
341Use direct event building (NOT fixtures) when: 354Use direct event building (NOT fixtures) when:
342 355
@@ -354,7 +367,7 @@ let invalid_event = client.event_builder(Kind::GitRepoAnnouncement, "")
354send_and_verify_rejected(client, invalid_event, "missing clone tag").await?; 367send_and_verify_rejected(client, invalid_event, "missing clone tag").await?;
355``` 368```
356 369
357### Anti-Patterns to Avoid 370#### Anti-Patterns to Avoid
358 371
359❌ **Creating TestContext inside helper functions** - Tests lose cache control 372❌ **Creating TestContext inside helper functions** - Tests lose cache control
360 373
@@ -368,34 +381,6 @@ send_and_verify_rejected(client, invalid_event, "missing clone tag").await?;
368 381
369✅ **Build invalid events directly** - Only for rejection tests 382✅ **Build invalid events directly** - Only for rejection tests
370 383
371## Architecture
372
373```
374grasp-audit/
375├── src/
376│ ├── lib.rs # Public API
377│ ├── audit.rs # Audit config and event tagging
378│ ├── client.rs # Audit client
379│ ├── fixtures.rs # TestContext and FixtureKind
380│ ├── result.rs # Test result types
381│ ├── isolation.rs # Test isolation utilities
382│ └── specs/
383│ ├── mod.rs
384│ └── nip01_smoke.rs # NIP-01 smoke tests
385├── examples/
386│ └── simple_audit.rs # Example usage
387└── bin/
388 └── grasp-audit.rs # CLI tool
389```
390
391## Development Status
392
393- ✅ Audit framework
394- ✅ NIP-01 smoke tests (6 tests)
395- 🚧 GRASP-01 relay tests (planned)
396- 🚧 GRASP-01 git tests (planned)
397- 🚧 Cleanup utilities (planned)
398
399## Contributing 384## Contributing
400 385
401This tool is designed to be reusable by any GRASP implementation. Contributions welcome! 386This tool is designed to be reusable by any GRASP implementation. Contributions welcome!
generated by cgit v1.2.3 (git 2.39.1) at 2026-06-05 02:22:23 +0000