upleb.uk

Public git repos — served from a NIP-34 GRASP relay at git.upleb.uk

summaryrefslogtreecommitdiff
path: root/docs
diff options
context:
space:
mode:
authorDanConwayDev <DanConwayDev@protonmail.com>2026-01-23 14:58:31 +0000
committerDanConwayDev <DanConwayDev@protonmail.com>2026-01-27 20:38:03 +0000
commit6a6c8cf8b70bc387ea7241b5c9ec457cb525eb40 (patch)
treea810ca522872e82e6fc053eb53aa3a3914b310b3 /docs
parentb61d388162438d12df37aa3fcd40bc9d3344d5bd (diff)
Update migration guide with lessons learned from relay.ngit.dev analysis
- Add Gotchas section with common issues: git installation, localhost-only archive relays, non-standard git paths, service name variations, and permission requirements - Add relay.ngit.dev-specific migration notes with actual paths, service names, and analysis results (315 repos need re-sync, 382 purgatory expired) - Enhance Running the Analysis section with path discovery guidance - Expand Troubleshooting section with solutions for git not found, archive connection failures, and wrong git paths - Add git --version to prerequisite checks - Update examples to use realistic localhost archive URLs
Diffstat (limited to 'docs')
-rw-r--r--docs/how-to/migrate-to-ngit-grasp.md257
1 files changed, 252 insertions, 5 deletions
diff --git a/docs/how-to/migrate-to-ngit-grasp.md b/docs/how-to/migrate-to-ngit-grasp.md
index f4dff86..9b812a5 100644
--- a/docs/how-to/migrate-to-ngit-grasp.md
+++ b/docs/how-to/migrate-to-ngit-grasp.md
@@ -62,11 +62,112 @@ See [Running the Analysis](#running-the-analysis) for detailed options.
62# Check required tools 62# Check required tools
63nak --version 63nak --version
64jq --version 64jq --version
65git --version
65 66
66# Check optional tools (for VPS phases) 67# Check optional tools (for VPS phases)
67journalctl --version 68journalctl --version
68``` 69```
69 70
71## Gotchas and Common Issues
72
73Before running the analysis, be aware of these common issues discovered during real migrations:
74
75### Git Must Be Installed
76
77The analysis scripts require `git` to be installed and in PATH. This may not be present on minimal VPS installations.
78
79```bash
80# Check if git is available
81which git || echo "Git not found - install it first"
82
83# Install on Debian/Ubuntu
84apt install git
85
86# Install on NixOS (add to configuration.nix)
87environment.systemPackages = [ pkgs.git ];
88```
89
90### Archive Relay May Only Be Accessible Locally
91
92If your archive relay is configured to listen only on localhost (e.g., `ws://localhost:7443`), you must run the analysis **on the VPS itself**, not from a remote machine.
93
94```bash
95# Check if archive relay is accessible
96# This will fail if run remotely against a localhost-only relay
97nak req -k 30618 --limit 1 ws://localhost:7443
98
99# Solution: SSH into the VPS and run analysis there
100ssh user@your-vps
101cd /path/to/scripts
102./run-migration-analysis.sh --archive-relay ws://localhost:7443 ...
103```
104
105### Git Data Paths May Differ from Defaults
106
107Different deployments store git data in different locations. **Always verify paths before running the analysis.**
108
109```bash
110# Find actual git data paths from service configuration
111systemctl cat ngit-relay.service | grep -E 'ExecStart|WorkingDirectory|Environment'
112systemctl cat ngit-grasp-*.service | grep -E 'ExecStart|WorkingDirectory|Environment'
113
114# Common locations:
115# - /var/lib/ngit-relay/git (default)
116# - /var/lib/ngit-grasp/git (default)
117# - /persistent/*/data/repos (custom deployments)
118
119# Verify the path exists and contains expected structure
120ls /path/to/git/npub1*/ # Should show *.git directories
121```
122
123### Phase 4 Needs the Correct Service Name
124
125Phase 4 extracts structured logs (`[PARSE_FAIL]`, `[PURGATORY_EXPIRED]`) from journald. You must specify the service that has these logs - typically the **archive** service (ngit-grasp), not the production service (ngit-relay).
126
127```bash
128# Find all ngit-related services
129systemctl list-units 'ngit-*' --all
130
131# Check which service has structured logging
132journalctl -u ngit-grasp-*.service | grep -E '\[PARSE_FAIL\]|\[PURGATORY_EXPIRED\]' | head -5
133
134# Use the archive service name for Phase 4
135./run-migration-analysis.sh ... --service ngit-grasp-relay-ngit-dev.service
136```
137
138### Permission Issues with Service-Owned Directories
139
140Git data directories are typically owned by the service user and may require elevated permissions to read.
141
142```bash
143# Check directory permissions
144ls -la /var/lib/ngit-grasp/git
145
146# Options:
147# 1. Run as root/sudo
148sudo ./run-migration-analysis.sh ...
149
150# 2. Run as the service user
151sudo -u ngit-grasp ./run-migration-analysis.sh ...
152
153# 3. Add your user to the service group
154sudo usermod -aG ngit-grasp $USER
155# (logout/login required)
156```
157
158### Service Names Vary by Deployment
159
160NixOS multi-instance deployments use service names like `ngit-grasp-<instance>.service`. Always check actual service names.
161
162```bash
163# List all ngit services
164systemctl list-units 'ngit-*' --all --no-pager
165
166# Example output:
167# ngit-relay.service loaded active running ngit-relay
168# ngit-grasp-relay-ngit-dev.service loaded active running ngit-grasp (relay-ngit-dev)
169```
170
70## Migration Overview 171## Migration Overview
71 172
72The migration process has three stages: 173The migration process has three stages:
@@ -98,6 +199,26 @@ Once all issues are resolved:
98 199
99## Running the Analysis 200## Running the Analysis
100 201
202### Before You Start
203
204**Verify paths and service names** before running the analysis. Incorrect paths are the most common source of errors.
205
206```bash
207# 1. Find actual git data paths
208systemctl cat ngit-relay.service | grep -E 'ExecStart|data|git'
209systemctl cat ngit-grasp-*.service | grep -E 'ExecStart|data|git'
210
211# 2. Find service names
212systemctl list-units 'ngit-*' --all --no-pager
213
214# 3. Verify git data exists at the paths
215ls /path/to/prod/git/npub1*/ | head -5
216ls /path/to/archive/git/npub1*/ | head -5
217
218# 4. Check if archive relay is accessible
219nak req -k 30618 --limit 1 ws://localhost:7443 # or your archive URL
220```
221
101### Basic Usage 222### Basic Usage
102 223
103```bash 224```bash
@@ -115,13 +236,18 @@ Once all issues are resolved:
115 236
116### Full Analysis on VPS 237### Full Analysis on VPS
117 238
239**Important:** If your archive relay is localhost-only, you must run this on the VPS.
240
118```bash 241```bash
242# First, discover your actual paths (see "Before You Start" above)
243# Then run with the correct values:
244
119./run-migration-analysis.sh \ 245./run-migration-analysis.sh \
120 --prod-relay wss://source-relay.example.com \ 246 --prod-relay wss://source-relay.example.com \
121 --archive-relay wss://target-relay.example.com \ 247 --archive-relay ws://localhost:7443 \
122 --prod-git /var/lib/grasp-relay/git \ 248 --prod-git /path/to/prod/git \
123 --archive-git /var/lib/ngit-grasp/git \ 249 --archive-git /path/to/archive/git \
124 --service ngit-grasp.service 250 --service ngit-grasp-your-instance.service
125``` 251```
126 252
127### Phase Control 253### Phase Control
@@ -226,6 +352,21 @@ go install github.com/fiatjaf/nak@latest
226# Or download binary from releases 352# Or download binary from releases
227``` 353```
228 354
355### "git not found"
356
357Git must be installed and in PATH:
358
359```bash
360# Check if git is available
361which git
362
363# Install on Debian/Ubuntu
364sudo apt install git
365
366# Install on NixOS (add to configuration.nix)
367environment.systemPackages = [ pkgs.git ];
368```
369
229### "Permission denied" on git directories 370### "Permission denied" on git directories
230 371
231Run with sudo or ensure your user has read access: 372Run with sudo or ensure your user has read access:
@@ -234,8 +375,38 @@ Run with sudo or ensure your user has read access:
234# Check permissions 375# Check permissions
235ls -la /var/lib/grasp-relay/git 376ls -la /var/lib/grasp-relay/git
236 377
237# Run with sudo if needed 378# Option 1: Run with sudo
238sudo ./run-migration-analysis.sh ... 379sudo ./run-migration-analysis.sh ...
380
381# Option 2: Run as service user
382sudo -u ngit-grasp ./run-migration-analysis.sh ...
383```
384
385### Archive relay connection failed
386
387If you get connection errors to the archive relay:
388
389```bash
390# Check if relay is running
391systemctl status ngit-grasp-*.service
392
393# Check if it's localhost-only
394# If archive is ws://localhost:7443, you MUST run on the VPS
395ssh user@your-vps
396./run-migration-analysis.sh --archive-relay ws://localhost:7443 ...
397```
398
399### Wrong git paths / "No such file or directory"
400
401Git data paths vary by deployment. Discover the actual paths:
402
403```bash
404# Find paths from service configuration
405systemctl cat ngit-relay.service | grep -E 'ExecStart|WorkingDirectory|Environment'
406systemctl cat ngit-grasp-*.service | grep -E 'ExecStart|WorkingDirectory|Environment'
407
408# Verify the path contains git repos
409ls /discovered/path/npub1*/
239``` 410```
240 411
241### Phase 2 takes too long 412### Phase 2 takes too long
@@ -254,6 +425,18 @@ This is expected if:
254 425
255The analysis will continue without log data. 426The analysis will continue without log data.
256 427
428### Phase 4 finds no structured logs
429
430Structured logging (`[PARSE_FAIL]`, `[PURGATORY_EXPIRED]`) is only available in ngit-grasp. If checking an ngit-relay service, no structured logs will be found.
431
432```bash
433# Verify you're checking the right service (should be ngit-grasp)
434journalctl -u ngit-grasp-*.service | grep -E '\[PARSE_FAIL\]|\[PURGATORY_EXPIRED\]' | head -5
435
436# If checking ngit-relay, structured logs won't exist
437# Use --service with the ngit-grasp archive service name instead
438```
439
257### Event counts are multiples of 250 440### Event counts are multiples of 250
258 441
259This suggests pagination may have failed. The scripts use `--paginate` by default, but if you see exactly 250, 500, 750 events, verify the relay is responding correctly. 442This suggests pagination may have failed. The scripts use `--paginate` by default, but if you see exactly 250, 500, 750 events, verify the relay is responding correctly.
@@ -410,3 +593,67 @@ For advanced usage, you can run individual phase scripts:
410``` 593```
411 594
412Each script has detailed help available with `--help` or by reading the script header. 595Each script has detailed help available with `--help` or by reading the script header.
596
597## relay.ngit.dev Migration Notes
598
599This section documents the specific configuration and lessons learned from migrating relay.ngit.dev from ngit-relay to ngit-grasp. Use this as a reference for similar deployments.
600
601### Deployment Configuration
602
603| Component | Value |
604|-----------|-------|
605| **Production relay** | `wss://relay.ngit.dev` |
606| **Production service** | `ngit-relay.service` |
607| **Production git path** | `/persistent/relay-ngit-dev-ngit-relay/data/repos` |
608| **Archive relay** | `ws://localhost:7443` (localhost only) |
609| **Archive service** | `ngit-grasp-relay-ngit-dev.service` |
610| **Archive git path** | `/persistent/grasp/relay-ngit-dev/git` |
611
612### Key Differences from Defaults
613
6141. **Git paths are non-standard**: The production relay uses `/persistent/relay-ngit-dev-ngit-relay/data/repos` instead of `/var/lib/ngit-relay/git`
615
6162. **Archive is localhost-only**: The archive relay listens on `ws://localhost:7443`, not a public URL. All analysis must run on the VPS.
617
6183. **Service names include instance**: NixOS multi-instance deployment uses `ngit-grasp-relay-ngit-dev.service`, not `ngit-grasp.service`
619
620### Analysis Command
621
622```bash
623# Run on VPS (archive is localhost-only)
624./docs/how-to/migration-scripts/run-migration-analysis.sh \
625 --prod-relay wss://relay.ngit.dev \
626 --archive-relay ws://localhost:7443 \
627 --prod-git /persistent/relay-ngit-dev-ngit-relay/data/repos \
628 --archive-git /persistent/grasp/relay-ngit-dev/git \
629 --service ngit-grasp-relay-ngit-dev.service
630```
631
632### Analysis Results (January 2026)
633
634| Category | Count | Notes |
635|----------|-------|-------|
636| Complete in both | ~400 | Ready for migration |
637| Complete in prod, missing from archive | 315 | Need re-sync |
638| Empty in both | 100 | Users never pushed git data |
639| Manual investigation | 5 | Unusual states |
640| Purgatory expired | 382 | Structured logging working |
641
642### Lessons Learned
643
6441. **Always verify paths first**: The default paths in examples didn't match the actual deployment. Use `systemctl cat <service>` to find real paths.
645
6462. **Check archive accessibility**: We initially tried to run analysis remotely, but the archive relay was localhost-only. Had to SSH to VPS.
647
6483. **Use archive service for Phase 4**: Structured logging (`[PARSE_FAIL]`, `[PURGATORY_EXPIRED]`) is in the ngit-grasp archive service, not the ngit-relay production service.
649
6504. **Install git on VPS**: Git wasn't installed on the minimal VPS. The scripts now check for this in prerequisites.
651
6525. **Permissions matter**: Some directories required `sudo` to access. Running as root or the service user resolved this.
653
654### Next Steps for relay.ngit.dev
655
6561. **Re-sync 315 repos**: Trigger archive to re-fetch from production
6572. **Investigate 5 edge cases**: Manual review of unusual states
6583. **Monitor purgatory**: 382 expired entries indicate sync issues to investigate
6594. **Plan cutover**: Once re-sync complete, switch DNS/proxy to ngit-grasp