diff options
| author | DanConwayDev <DanConwayDev@protonmail.com> | 2025-11-04 21:58:23 +0000 |
|---|---|---|
| committer | DanConwayDev <DanConwayDev@protonmail.com> | 2025-11-04 21:58:23 +0000 |
| commit | 652c5913f695ba7e8dfd78cd0cbe5cc3de67fa59 (patch) | |
| tree | e76bc0bcdc9eeec466ddf6e5e75a7f63a9d6650e /docs/how-to | |
| parent | c2c0cdba4af434043f3fa707231d8f5a7e3fd882 (diff) | |
test: migrate to TestRelay fixture pattern and add compliance docs
- Remove unnecessary 'nix' dev dependency (Unix syscalls crate, not needed)
- Migrate announcement tests to new TestRelay fixture pattern
- Delete legacy test files (announcement_tests.rs, test_relay.sh)
- Add comprehensive test documentation (docs/how-to/test-compliance.md)
- Update README.md with new test commands
- All 18 integration tests passing (NIP-01 + NIP-34)
Benefits:
- Automatic relay lifecycle management
- No manual setup required
- Pure Rust integration tests
- Better developer experience
- CI/CD ready
Diffstat (limited to 'docs/how-to')
| -rw-r--r-- | docs/how-to/test-compliance.md | 440 |
1 files changed, 440 insertions, 0 deletions
diff --git a/docs/how-to/test-compliance.md b/docs/how-to/test-compliance.md new file mode 100644 index 0000000..370fb9a --- /dev/null +++ b/docs/how-to/test-compliance.md | |||
| @@ -0,0 +1,440 @@ | |||
| 1 | # How to Test GRASP Compliance | ||
| 2 | |||
| 3 | **Purpose:** Guide for running compliance tests against ngit-grasp relay | ||
| 4 | **Audience:** Developers, contributors, CI/CD maintainers | ||
| 5 | **Category:** How-To (task-oriented) | ||
| 6 | |||
| 7 | --- | ||
| 8 | |||
| 9 | ## Overview | ||
| 10 | |||
| 11 | This guide shows you how to run GRASP protocol compliance tests for the ngit-grasp relay. We have two test suites: | ||
| 12 | |||
| 13 | 1. **Integration Tests** - Built into ngit-grasp, test core functionality | ||
| 14 | 2. **GRASP Audit Tool** - Standalone compliance checker for any GRASP relay | ||
| 15 | |||
| 16 | --- | ||
| 17 | |||
| 18 | ## Quick Start | ||
| 19 | |||
| 20 | ```bash | ||
| 21 | # Run all integration tests (automatic relay management) | ||
| 22 | nix develop -c cargo test --test nip01_compliance --test nip34_announcements | ||
| 23 | |||
| 24 | # Run NIP-01 compliance tests only | ||
| 25 | nix develop -c cargo test --test nip01_compliance | ||
| 26 | |||
| 27 | # Run NIP-34 announcement tests only | ||
| 28 | nix develop -c cargo test --test nip34_announcements | ||
| 29 | |||
| 30 | # Run with detailed output | ||
| 31 | nix develop -c cargo test --test nip01_compliance -- --nocapture | ||
| 32 | ``` | ||
| 33 | |||
| 34 | **No manual setup needed!** Tests automatically start and stop relay instances. | ||
| 35 | |||
| 36 | --- | ||
| 37 | |||
| 38 | ## Integration Tests | ||
| 39 | |||
| 40 | ### What They Test | ||
| 41 | |||
| 42 | **NIP-01 Compliance (`tests/nip01_compliance.rs`)** | ||
| 43 | - Basic WebSocket connectivity | ||
| 44 | - Event publishing and subscription | ||
| 45 | - REQ/EVENT/CLOSE message handling | ||
| 46 | - Filter-based event queries | ||
| 47 | - Relay connection lifecycle | ||
| 48 | |||
| 49 | **NIP-34 Announcements (`tests/nip34_announcements.rs`)** | ||
| 50 | - Repository announcement acceptance (kind 30617) | ||
| 51 | - Repository state event acceptance (kind 30618) | ||
| 52 | - Clone URL validation | ||
| 53 | - Relay URL validation | ||
| 54 | - Domain matching | ||
| 55 | - Multi-branch state events | ||
| 56 | - Event queries by kind and tags | ||
| 57 | |||
| 58 | ### Test Architecture | ||
| 59 | |||
| 60 | All integration tests use the **TestRelay fixture pattern**: | ||
| 61 | |||
| 62 | ```rust | ||
| 63 | use crate::common::relay::TestRelay; | ||
| 64 | |||
| 65 | #[tokio::test] | ||
| 66 | async fn test_something() { | ||
| 67 | // Automatic relay startup on random port | ||
| 68 | let relay = TestRelay::start().await; | ||
| 69 | |||
| 70 | // Test code here | ||
| 71 | // ... | ||
| 72 | |||
| 73 | // Automatic cleanup when relay drops | ||
| 74 | } | ||
| 75 | ``` | ||
| 76 | |||
| 77 | **Benefits:** | ||
| 78 | - ✅ Automatic relay lifecycle management | ||
| 79 | - ✅ Random port allocation (no conflicts) | ||
| 80 | - ✅ Isolated test environments | ||
| 81 | - ✅ Automatic cleanup on test completion | ||
| 82 | - ✅ No manual relay management needed | ||
| 83 | |||
| 84 | ### Running Specific Tests | ||
| 85 | |||
| 86 | ```bash | ||
| 87 | # Run a specific test by name | ||
| 88 | nix develop -c cargo test --test nip01_compliance test_nip01_smoke | ||
| 89 | |||
| 90 | # List all tests without running | ||
| 91 | nix develop -c cargo test --test nip34_announcements -- --list | ||
| 92 | |||
| 93 | # Run tests matching a pattern | ||
| 94 | nix develop -c cargo test --test nip34_announcements test_accepts | ||
| 95 | ``` | ||
| 96 | |||
| 97 | ### Test Output | ||
| 98 | |||
| 99 | ```bash | ||
| 100 | $ nix develop -c cargo test --test nip01_compliance | ||
| 101 | |||
| 102 | running 6 tests | ||
| 103 | test test_nip01_smoke ... ok | ||
| 104 | test test_subscription ... ok | ||
| 105 | test test_event_publishing ... ok | ||
| 106 | test test_filter_queries ... ok | ||
| 107 | test test_connection_lifecycle ... ok | ||
| 108 | test test_relay_lifecycle ... ignored | ||
| 109 | |||
| 110 | test result: ok. 5 passed; 0 failed; 1 ignored; 0 measured; 0 filtered out | ||
| 111 | ``` | ||
| 112 | |||
| 113 | --- | ||
| 114 | |||
| 115 | ## GRASP Audit Tool | ||
| 116 | |||
| 117 | ### What It Does | ||
| 118 | |||
| 119 | The `grasp-audit` tool is a standalone compliance checker that can test **any** GRASP relay (local or remote). | ||
| 120 | |||
| 121 | **Located:** `grasp-audit/` subdirectory (separate Rust project) | ||
| 122 | |||
| 123 | ### Running the Audit Tool | ||
| 124 | |||
| 125 | ```bash | ||
| 126 | # Enter the grasp-audit directory | ||
| 127 | cd grasp-audit | ||
| 128 | |||
| 129 | # Run against local relay | ||
| 130 | nix develop -c cargo run -- --url ws://127.0.0.1:7000 | ||
| 131 | |||
| 132 | # Run against remote relay | ||
| 133 | nix develop -c cargo run -- --url wss://relay.example.com | ||
| 134 | |||
| 135 | # Run with verbose output | ||
| 136 | nix develop -c cargo run -- --url ws://127.0.0.1:7000 --verbose | ||
| 137 | ``` | ||
| 138 | |||
| 139 | ### What It Tests | ||
| 140 | |||
| 141 | - NIP-01 basic relay functionality | ||
| 142 | - NIP-34 repository announcement handling | ||
| 143 | - GRASP-01 core service requirements | ||
| 144 | - Domain validation | ||
| 145 | - Event acceptance/rejection rules | ||
| 146 | |||
| 147 | ### Example Output | ||
| 148 | |||
| 149 | ```bash | ||
| 150 | $ cd grasp-audit | ||
| 151 | $ nix develop -c cargo run -- --url ws://127.0.0.1:7000 | ||
| 152 | |||
| 153 | GRASP Compliance Audit | ||
| 154 | ====================== | ||
| 155 | Relay: ws://127.0.0.1:7000 | ||
| 156 | |||
| 157 | ✅ NIP-01: Basic Connectivity | ||
| 158 | ✅ NIP-01: Event Publishing | ||
| 159 | ✅ NIP-01: Subscriptions | ||
| 160 | ✅ NIP-34: Repository Announcements | ||
| 161 | ✅ NIP-34: State Events | ||
| 162 | ✅ GRASP-01: Domain Validation | ||
| 163 | |||
| 164 | Summary: 6/6 tests passed | ||
| 165 | Status: COMPLIANT | ||
| 166 | ``` | ||
| 167 | |||
| 168 | --- | ||
| 169 | |||
| 170 | ## Testing Workflow | ||
| 171 | |||
| 172 | ### For Development | ||
| 173 | |||
| 174 | **1. Quick Validation (after code changes)** | ||
| 175 | ```bash | ||
| 176 | # Run all integration tests | ||
| 177 | nix develop -c cargo test --test nip01_compliance --test nip34_announcements | ||
| 178 | ``` | ||
| 179 | |||
| 180 | **2. Deep Compliance Check (before release)** | ||
| 181 | ```bash | ||
| 182 | # Start your relay | ||
| 183 | nix develop -c cargo run | ||
| 184 | |||
| 185 | # In another terminal, run audit tool | ||
| 186 | cd grasp-audit | ||
| 187 | nix develop -c cargo run -- --url ws://127.0.0.1:8080 | ||
| 188 | ``` | ||
| 189 | |||
| 190 | ### For CI/CD | ||
| 191 | |||
| 192 | **Recommended CI pipeline:** | ||
| 193 | |||
| 194 | ```yaml | ||
| 195 | # .github/workflows/test.yml example | ||
| 196 | test: | ||
| 197 | runs-on: ubuntu-latest | ||
| 198 | steps: | ||
| 199 | - uses: actions/checkout@v3 | ||
| 200 | - uses: cachix/install-nix-action@v22 | ||
| 201 | - name: Run integration tests | ||
| 202 | run: nix develop -c cargo test --test nip01_compliance --test nip34_announcements | ||
| 203 | ``` | ||
| 204 | |||
| 205 | **Why this works:** | ||
| 206 | - No external relay needed | ||
| 207 | - Tests manage their own relay instances | ||
| 208 | - Fast parallel execution | ||
| 209 | - Clean isolation | ||
| 210 | |||
| 211 | --- | ||
| 212 | |||
| 213 | ## Test Configuration | ||
| 214 | |||
| 215 | ### Environment Variables | ||
| 216 | |||
| 217 | Tests use these environment variables (set automatically by TestRelay): | ||
| 218 | |||
| 219 | - `NGIT_DOMAIN` - Domain for clone URL validation (auto-set to bind address) | ||
| 220 | - `NGIT_RELAY_DATA_PATH` - Temporary directory for relay data | ||
| 221 | - `RUST_LOG` - Logging level (optional, for debugging) | ||
| 222 | |||
| 223 | **Example: Enable debug logging** | ||
| 224 | ```bash | ||
| 225 | RUST_LOG=debug nix develop -c cargo test --test nip01_compliance -- --nocapture | ||
| 226 | ``` | ||
| 227 | |||
| 228 | ### Test Data Locations | ||
| 229 | |||
| 230 | Integration tests use temporary directories: | ||
| 231 | |||
| 232 | ``` | ||
| 233 | /tmp/ngit-test-XXXXXX/ # Relay data (auto-cleaned) | ||
| 234 | ├── events/ # Nostr events | ||
| 235 | └── git/ # Git repositories (if tested) | ||
| 236 | ``` | ||
| 237 | |||
| 238 | **Cleanup:** Automatic when test completes (or on failure). | ||
| 239 | |||
| 240 | --- | ||
| 241 | |||
| 242 | ## Troubleshooting | ||
| 243 | |||
| 244 | ### Test Hangs or Times Out | ||
| 245 | |||
| 246 | **Problem:** Test hangs waiting for relay to start | ||
| 247 | |||
| 248 | **Solution:** | ||
| 249 | ```bash | ||
| 250 | # Check if port is already in use | ||
| 251 | lsof -i :7000 | ||
| 252 | |||
| 253 | # Kill any stray relay processes | ||
| 254 | pkill -f ngit-grasp | ||
| 255 | |||
| 256 | # Re-run test | ||
| 257 | nix develop -c cargo test --test nip01_compliance | ||
| 258 | ``` | ||
| 259 | |||
| 260 | ### Connection Refused | ||
| 261 | |||
| 262 | **Problem:** `Connection refused` error in tests | ||
| 263 | |||
| 264 | **Cause:** Relay failed to start (check for port conflicts) | ||
| 265 | |||
| 266 | **Solution:** | ||
| 267 | ```bash | ||
| 268 | # Tests use random ports, but check for system issues | ||
| 269 | netstat -tuln | grep LISTEN | ||
| 270 | |||
| 271 | # Check relay logs | ||
| 272 | RUST_LOG=debug nix develop -c cargo test --test nip01_compliance -- --nocapture | ||
| 273 | ``` | ||
| 274 | |||
| 275 | ### Tests Pass Locally but Fail in CI | ||
| 276 | |||
| 277 | **Problem:** CI environment differences | ||
| 278 | |||
| 279 | **Common causes:** | ||
| 280 | - Network restrictions (WebSocket blocked) | ||
| 281 | - Insufficient resources (slow startup) | ||
| 282 | - Missing dependencies | ||
| 283 | |||
| 284 | **Solution:** | ||
| 285 | ```bash | ||
| 286 | # Ensure Nix is installed in CI | ||
| 287 | # Use longer timeouts for slow systems | ||
| 288 | # Check CI logs for specific errors | ||
| 289 | ``` | ||
| 290 | |||
| 291 | ### Audit Tool Can't Connect | ||
| 292 | |||
| 293 | **Problem:** `grasp-audit` fails to connect to relay | ||
| 294 | |||
| 295 | **Checklist:** | ||
| 296 | 1. Is the relay running? (`ps aux | grep ngit-grasp`) | ||
| 297 | 2. Is the URL correct? (ws:// for local, wss:// for remote) | ||
| 298 | 3. Is the port accessible? (`telnet 127.0.0.1 7000`) | ||
| 299 | 4. Check firewall rules | ||
| 300 | |||
| 301 | --- | ||
| 302 | |||
| 303 | ## Writing New Tests | ||
| 304 | |||
| 305 | ### Integration Test Pattern | ||
| 306 | |||
| 307 | **1. Create test file in `tests/` directory** | ||
| 308 | |||
| 309 | ```rust | ||
| 310 | // tests/my_new_tests.rs | ||
| 311 | mod common; | ||
| 312 | |||
| 313 | use common::relay::TestRelay; | ||
| 314 | use tokio_tungstenite::connect_async; | ||
| 315 | |||
| 316 | #[tokio::test] | ||
| 317 | async fn test_my_feature() { | ||
| 318 | // Start relay | ||
| 319 | let relay = TestRelay::start().await; | ||
| 320 | |||
| 321 | // Connect | ||
| 322 | let (mut ws, _) = connect_async(relay.ws_url()) | ||
| 323 | .await | ||
| 324 | .expect("Failed to connect"); | ||
| 325 | |||
| 326 | // Test your feature | ||
| 327 | // ... | ||
| 328 | |||
| 329 | // Cleanup automatic when relay drops | ||
| 330 | } | ||
| 331 | ``` | ||
| 332 | |||
| 333 | **2. Run your test** | ||
| 334 | ```bash | ||
| 335 | nix develop -c cargo test --test my_new_tests | ||
| 336 | ``` | ||
| 337 | |||
| 338 | ### Adding to Audit Tool | ||
| 339 | |||
| 340 | **1. Edit `grasp-audit/src/main.rs`** | ||
| 341 | |||
| 342 | Add your test function following existing patterns. | ||
| 343 | |||
| 344 | **2. Test it** | ||
| 345 | ```bash | ||
| 346 | cd grasp-audit | ||
| 347 | nix develop -c cargo run -- --url ws://127.0.0.1:7000 | ||
| 348 | ``` | ||
| 349 | |||
| 350 | --- | ||
| 351 | |||
| 352 | ## Test Coverage | ||
| 353 | |||
| 354 | ### Current Coverage | ||
| 355 | |||
| 356 | **NIP-01 (Nostr Relay):** | ||
| 357 | - ✅ WebSocket connectivity | ||
| 358 | - ✅ Event publishing | ||
| 359 | - ✅ Subscriptions (REQ/EVENT/EOSE/CLOSE) | ||
| 360 | - ✅ Filter queries | ||
| 361 | - ✅ Connection lifecycle | ||
| 362 | |||
| 363 | **NIP-34 (Git Stuff):** | ||
| 364 | - ✅ Repository announcements (kind 30617) | ||
| 365 | - ✅ Repository state events (kind 30618) | ||
| 366 | - ✅ Clone URL validation | ||
| 367 | - ✅ Relay URL validation | ||
| 368 | - ✅ Domain matching | ||
| 369 | - ✅ Multi-branch support | ||
| 370 | - ✅ Event queries | ||
| 371 | |||
| 372 | **GRASP-01 (Core Service):** | ||
| 373 | - ✅ Nostr relay at `/` | ||
| 374 | - ✅ NIP-34 event acceptance | ||
| 375 | - ✅ Domain validation | ||
| 376 | - ⏳ Git HTTP backend (planned) | ||
| 377 | - ⏳ Push authorization (planned) | ||
| 378 | |||
| 379 | ### Gaps (TODO) | ||
| 380 | |||
| 381 | - Git Smart HTTP protocol tests | ||
| 382 | - Push authorization validation | ||
| 383 | - Multi-maintainer scenarios | ||
| 384 | - PR reference handling (`refs/nostr/<event-id>`) | ||
| 385 | - CORS headers | ||
| 386 | - NIP-11 relay info document | ||
| 387 | |||
| 388 | --- | ||
| 389 | |||
| 390 | ## Performance Testing | ||
| 391 | |||
| 392 | ### Load Testing (Future) | ||
| 393 | |||
| 394 | ```bash | ||
| 395 | # Planned: Load test with multiple concurrent connections | ||
| 396 | # TODO: Add load testing tools | ||
| 397 | ``` | ||
| 398 | |||
| 399 | ### Benchmarking (Future) | ||
| 400 | |||
| 401 | ```bash | ||
| 402 | # Planned: Benchmark event processing throughput | ||
| 403 | # TODO: Add criterion benchmarks | ||
| 404 | ``` | ||
| 405 | |||
| 406 | --- | ||
| 407 | |||
| 408 | ## Related Documentation | ||
| 409 | |||
| 410 | - **[Test Strategy](../reference/test-strategy.md)** - Overall testing approach | ||
| 411 | - **[Architecture](../explanation/architecture.md)** - System design | ||
| 412 | - **[Getting Started](../tutorials/getting-started.md)** - Initial setup | ||
| 413 | - **[Nix Flakes](./nix-flakes.md)** - Nix development environment | ||
| 414 | |||
| 415 | --- | ||
| 416 | |||
| 417 | ## Summary | ||
| 418 | |||
| 419 | **For quick validation:** | ||
| 420 | ```bash | ||
| 421 | nix develop -c cargo test --test nip01_compliance --test nip34_announcements | ||
| 422 | ``` | ||
| 423 | |||
| 424 | **For deep compliance check:** | ||
| 425 | ```bash | ||
| 426 | cd grasp-audit | ||
| 427 | nix develop -c cargo run -- --url ws://127.0.0.1:8080 | ||
| 428 | ``` | ||
| 429 | |||
| 430 | **Key points:** | ||
| 431 | - ✅ No manual relay management needed | ||
| 432 | - ✅ Automatic cleanup and isolation | ||
| 433 | - ✅ Fast parallel execution | ||
| 434 | - ✅ Works in CI/CD | ||
| 435 | - ✅ Tests both local and remote relays | ||
| 436 | |||
| 437 | --- | ||
| 438 | |||
| 439 | **Last Updated:** November 4, 2025 | ||
| 440 | **Status:** ✅ Complete and current | ||