diff options
| author | DanConwayDev <DanConwayDev@protonmail.com> | 2026-01-10 01:49:33 +0000 |
|---|---|---|
| committer | DanConwayDev <DanConwayDev@protonmail.com> | 2026-01-10 01:49:33 +0000 |
| commit | 730f430c906c6c2d43ea8f2e5fc3b408a3de128b (patch) | |
| tree | 97d9161b5a5ad717918eff59b62a496ec59b0415 /docs/how-to/production-sync-testing.md | |
| parent | 52430fe5db959dcc23fa965dd7aabcdc29db17a5 (diff) | |
docs: save both raw and sanitized logs for flexible analysis
- Update production sync testing workflow to save both sync-raw.log and sync.log
- Raw log contains complete untruncated messages (rejection reasons, event data, etc.)
- Sanitized log remains for quick scanning and pattern recognition
- Add guidance on when to use each log and how to retrieve full details from raw log
- Resolves truncated rejection warning issue by making full details accessible
Diffstat (limited to 'docs/how-to/production-sync-testing.md')
| -rw-r--r-- | docs/how-to/production-sync-testing.md | 48 |
1 files changed, 37 insertions, 11 deletions
diff --git a/docs/how-to/production-sync-testing.md b/docs/how-to/production-sync-testing.md index 3a14791..3b1d4e4 100644 --- a/docs/how-to/production-sync-testing.md +++ b/docs/how-to/production-sync-testing.md | |||
| @@ -80,13 +80,13 @@ Start with short runs (30 seconds) to capture manageable log volumes. Each run c | |||
| 80 | RUN_DIR="tmp/run-$(date +%Y%m%d-%H%M%S)" | 80 | RUN_DIR="tmp/run-$(date +%Y%m%d-%H%M%S)" |
| 81 | mkdir -p "$RUN_DIR" | 81 | mkdir -p "$RUN_DIR" |
| 82 | 82 | ||
| 83 | # Run for 30 seconds with sanitized output | 83 | # Run for 30 seconds, saving both raw and sanitized logs |
| 84 | timeout 30s cargo run -- \ | 84 | timeout 30s cargo run -- \ |
| 85 | --sync-bootstrap-relay-url wss://git.shakespeare.diy \ | 85 | --sync-bootstrap-relay-url wss://git.shakespeare.diy \ |
| 86 | --domain ngit.danconwaydev.com \ | 86 | --domain ngit.danconwaydev.com \ |
| 87 | --git-data-path "$RUN_DIR/git-data" \ | 87 | --git-data-path "$RUN_DIR/git-data" \ |
| 88 | --relay-data-path "$RUN_DIR/relay-data" \ | 88 | --relay-data-path "$RUN_DIR/relay-data" \ |
| 89 | 2>&1 | ./scripts/sanitize-logs.sh | tee "$RUN_DIR/sync.log" | 89 | 2>&1 | tee "$RUN_DIR/sync-raw.log" | ./scripts/sanitize-logs.sh | tee "$RUN_DIR/sync.log" |
| 90 | ``` | 90 | ``` |
| 91 | 91 | ||
| 92 | **Note:** The `timeout` command returns exit code 124, which is expected. | 92 | **Note:** The `timeout` command returns exit code 124, which is expected. |
| @@ -97,9 +97,14 @@ tmp/ | |||
| 97 | └── run-20260109-143022/ | 97 | └── run-20260109-143022/ |
| 98 | ├── git-data/ # Git repository data | 98 | ├── git-data/ # Git repository data |
| 99 | ├── relay-data/ # Relay database | 99 | ├── relay-data/ # Relay database |
| 100 | └── sync.log # Sanitized log output | 100 | ├── sync.log # Sanitized log (for quick analysis) |
| 101 | └── sync-raw.log # Raw log (for full details when needed) | ||
| 101 | ``` | 102 | ``` |
| 102 | 103 | ||
| 104 | **When to use which log:** | ||
| 105 | - **sync.log** - Use for quick scanning and pattern recognition (long lines truncated) | ||
| 106 | - **sync-raw.log** - Use when you need full details (e.g., complete rejection reasons, full event data) | ||
| 107 | |||
| 103 | ## Log Sanitization | 108 | ## Log Sanitization |
| 104 | 109 | ||
| 105 | Raw logs include full events and hundreds of event IDs per line, making them unwieldy for analysis. The sanitizer truncates long lines: | 110 | Raw logs include full events and hundreds of event IDs per line, making them unwieldy for analysis. The sanitizer truncates long lines: |
| @@ -120,6 +125,23 @@ Example output: | |||
| 120 | 2024-01-09T10:00:00Z DEBUG sync: Processing events ids=[abc123, def456, ghi789, jkl012...<1847 chars>...xyz999, end123] | 125 | 2024-01-09T10:00:00Z DEBUG sync: Processing events ids=[abc123, def456, ghi789, jkl012...<1847 chars>...xyz999, end123] |
| 121 | ``` | 126 | ``` |
| 122 | 127 | ||
| 128 | ### Retrieving Full Details from Raw Logs | ||
| 129 | |||
| 130 | When sanitized logs show truncated messages (e.g., rejection reasons), use the raw log to see the complete content: | ||
| 131 | |||
| 132 | ```bash | ||
| 133 | # Find specific error in raw log | ||
| 134 | grep "Rejected repository announcement" "$RUN_DIR/sync-raw.log" | ||
| 135 | |||
| 136 | # Extract full line for specific event ID | ||
| 137 | grep "note1z5ys7wf3ms5yxhnp3kfw7hpu5asfkx4jngzt5zgs4tm4tnvggnsqjfqeyt" "$RUN_DIR/sync-raw.log" | ||
| 138 | |||
| 139 | # View context around a truncated warning | ||
| 140 | grep -A 2 -B 2 "pattern from sanitized log" "$RUN_DIR/sync-raw.log" | ||
| 141 | ``` | ||
| 142 | |||
| 143 | The raw log contains complete, untruncated messages including full rejection reasons, event data, and debug details. | ||
| 144 | |||
| 123 | ## What to Look For | 145 | ## What to Look For |
| 124 | 146 | ||
| 125 | ### Phase 1: Connection & Bootstrap (0-5 seconds) | 147 | ### Phase 1: Connection & Bootstrap (0-5 seconds) |
| @@ -265,16 +287,16 @@ When `work/active-issues/` is empty (or only contains README.md): | |||
| 265 | RUN_DIR="tmp/run-$(date +%Y%m%d-%H%M%S)" | 287 | RUN_DIR="tmp/run-$(date +%Y%m%d-%H%M%S)" |
| 266 | mkdir -p "$RUN_DIR" | 288 | mkdir -p "$RUN_DIR" |
| 267 | 289 | ||
| 268 | # Run 30-second test | 290 | # Run 30-second test, saving both raw and sanitized logs |
| 269 | timeout 30s cargo run -- \ | 291 | timeout 30s cargo run -- \ |
| 270 | --sync-bootstrap-relay-url wss://git.shakespeare.diy \ | 292 | --sync-bootstrap-relay-url wss://git.shakespeare.diy \ |
| 271 | --domain ngit.danconwaydev.com \ | 293 | --domain ngit.danconwaydev.com \ |
| 272 | --git-data-path "$RUN_DIR/git-data" \ | 294 | --git-data-path "$RUN_DIR/git-data" \ |
| 273 | --relay-data-path "$RUN_DIR/relay-data" \ | 295 | --relay-data-path "$RUN_DIR/relay-data" \ |
| 274 | 2>&1 | ./scripts/sanitize-logs.sh | tee "$RUN_DIR/sync.log" | 296 | 2>&1 | tee "$RUN_DIR/sync-raw.log" | ./scripts/sanitize-logs.sh | tee "$RUN_DIR/sync.log" |
| 275 | ``` | 297 | ``` |
| 276 | 298 | ||
| 277 | Each run is isolated in its own timestamped directory under `tmp/`, keeping data and logs organized. | 299 | Each run is isolated in its own timestamped directory under `tmp/`, keeping data and logs organized. Both raw and sanitized logs are saved for flexible analysis. |
| 278 | 300 | ||
| 279 | ### Step 2: Analyze Logs | 301 | ### Step 2: Analyze Logs |
| 280 | 302 | ||
| @@ -282,11 +304,15 @@ Scan for errors and unexpected patterns: | |||
| 282 | ```bash | 304 | ```bash |
| 283 | # Find the most recent run | 305 | # Find the most recent run |
| 284 | LATEST_RUN=$(ls -1t tmp/run-*/sync.log | head -n1) | 306 | LATEST_RUN=$(ls -1t tmp/run-*/sync.log | head -n1) |
| 307 | LATEST_RAW=$(ls -1t tmp/run-*/sync-raw.log | head -n1) | ||
| 285 | 308 | ||
| 286 | # Analyze for issues | 309 | # Analyze sanitized log for quick scanning |
| 287 | grep -i error "$LATEST_RUN" | 310 | grep -i error "$LATEST_RUN" |
| 288 | grep -i warn "$LATEST_RUN" | 311 | grep -i warn "$LATEST_RUN" |
| 289 | grep -i panic "$LATEST_RUN" | 312 | grep -i panic "$LATEST_RUN" |
| 313 | |||
| 314 | # If you find truncated messages, check the raw log for full details | ||
| 315 | grep "pattern from truncated message" "$LATEST_RAW" | ||
| 290 | ``` | 316 | ``` |
| 291 | 317 | ||
| 292 | ### Step 3: Document Issues | 318 | ### Step 3: Document Issues |
| @@ -435,13 +461,13 @@ Check work/active-issues/ | |||
| 435 | RUN_DIR="tmp/run-$(date +%Y%m%d-%H%M%S)" | 461 | RUN_DIR="tmp/run-$(date +%Y%m%d-%H%M%S)" |
| 436 | mkdir -p "$RUN_DIR" | 462 | mkdir -p "$RUN_DIR" |
| 437 | 463 | ||
| 438 | # Run test | 464 | # Run test with both raw and sanitized logs |
| 439 | timeout 30s cargo run -- \ | 465 | timeout 30s cargo run -- \ |
| 440 | --sync-bootstrap-relay-url wss://git.shakespeare.diy \ | 466 | --sync-bootstrap-relay-url wss://git.shakespeare.diy \ |
| 441 | --domain ngit.danconwaydev.com \ | 467 | --domain ngit.danconwaydev.com \ |
| 442 | --git-data-path "$RUN_DIR/git-data" \ | 468 | --git-data-path "$RUN_DIR/git-data" \ |
| 443 | --relay-data-path "$RUN_DIR/relay-data" \ | 469 | --relay-data-path "$RUN_DIR/relay-data" \ |
| 444 | 2>&1 | ./scripts/sanitize-logs.sh | tee "$RUN_DIR/sync.log" | 470 | 2>&1 | tee "$RUN_DIR/sync-raw.log" | ./scripts/sanitize-logs.sh | tee "$RUN_DIR/sync.log" |
| 445 | ``` | 471 | ``` |
| 446 | 472 | ||
| 447 | ### With Metrics Endpoint | 473 | ### With Metrics Endpoint |
| @@ -451,14 +477,14 @@ timeout 30s cargo run -- \ | |||
| 451 | RUN_DIR="tmp/run-$(date +%Y%m%d-%H%M%S)" | 477 | RUN_DIR="tmp/run-$(date +%Y%m%d-%H%M%S)" |
| 452 | mkdir -p "$RUN_DIR" | 478 | mkdir -p "$RUN_DIR" |
| 453 | 479 | ||
| 454 | # Run with metrics | 480 | # Run with metrics and both log formats |
| 455 | timeout 30s cargo run -- \ | 481 | timeout 30s cargo run -- \ |
| 456 | --sync-bootstrap-relay-url wss://git.shakespeare.diy \ | 482 | --sync-bootstrap-relay-url wss://git.shakespeare.diy \ |
| 457 | --domain ngit.danconwaydev.com \ | 483 | --domain ngit.danconwaydev.com \ |
| 458 | --git-data-path "$RUN_DIR/git-data" \ | 484 | --git-data-path "$RUN_DIR/git-data" \ |
| 459 | --relay-data-path "$RUN_DIR/relay-data" \ | 485 | --relay-data-path "$RUN_DIR/relay-data" \ |
| 460 | --metrics-address 127.0.0.1:9090 \ | 486 | --metrics-address 127.0.0.1:9090 \ |
| 461 | 2>&1 | ./scripts/sanitize-logs.sh | tee "$RUN_DIR/sync.log" | 487 | 2>&1 | tee "$RUN_DIR/sync-raw.log" | ./scripts/sanitize-logs.sh | tee "$RUN_DIR/sync.log" |
| 462 | ``` | 488 | ``` |
| 463 | 489 | ||
| 464 | Then in another terminal: `curl http://127.0.0.1:9090/metrics` | 490 | Then in another terminal: `curl http://127.0.0.1:9090/metrics` |