Merge relay.ngit.dev migration: bug fixes and migration tooling

This merge includes critical bug fixes and comprehensive migration tooling
developed during the relay.ngit.dev migration effort.

Bug Fixes:
- Fix git protocol error handling to return HTTP 200 with ERR pkt-line
- Fix naughty list false positives and DNS failure identification
- Fix database query filters in load_existing_events (remove .since())
- Fix OID fetch tracking to distinguish 0 OIDs from successful fetches
- Fix purgatory event source tracking for filtered expiry logging
- Implement OID retry logic for 'not our ref' errors

Migration Tools & Documentation:
- Complete 5-phase migration analysis pipeline with orchestration script
- Phase 1: Event fetching from source relay
- Phase 2: Git sync verification
- Phase 3: Categorization and relay comparison
- Phase 4: Log extraction (parse failures, purgatory expiry)
- Phase 5: Action classification for migration decisions
- Comprehensive migration guide with lessons learned
- Troubleshooting guide for permission and corruption issues

Configuration:
- Add NGIT_LOG_LEVEL configuration option
- Update git throttle limits to 60/minute
- Improve logging throughout for better observability

1 files changed, 66 insertions, 0 deletions

diff --git a/src/git/handlers.rs b/src/git/handlers.rs
index 017eee4..e3a6ad4 100644
--- a/src/git/handlers.rs
+++ b/src/git/handlers.rs
@@ -99,6 +99,42 @@ pub async fn handle_info_refs(
         .unwrap())
 }
 
+/// Build an HTTP 200 OK response with an ERR pkt-line for git protocol errors.
+///
+/// Per the git smart HTTP protocol spec, protocol-level errors (like "not our ref")
+/// should be returned as HTTP 200 OK with the error message in pkt-line format:
+/// `PKT-LINE("ERR" SP explanation-text)`
+///
+/// This allows git clients to properly parse and display the error message.
+fn build_git_protocol_error_response(
+    service: GitService,
+    error_message: &str,
+) -> Response<Full<Bytes>> {
+    // Format: "ERR <message>\n"
+    let err_content = format!("ERR {}\n", error_message.trim());
+    let err_pktline = PktLine::data(err_content.as_bytes()).encode();
+
+    Response::builder()
+        .status(StatusCode::OK)
+        .header("content-type", service.result_content_type())
+        .header("cache-control", "no-cache")
+        .body(Full::new(Bytes::from(err_pktline)))
+        .unwrap()
+}
+
+/// Check if a git process failure is a protocol error (vs transport error).
+///
+/// Protocol errors are communicated via stderr when git exits with code 128.
+/// These should be returned to the client as HTTP 200 with ERR pkt-line.
+///
+/// Transport errors (process spawn failures, I/O errors, signals) should
+/// remain as HTTP 500 errors.
+fn is_git_protocol_error(exit_code: Option<i32>, stderr: &[u8]) -> bool {
+    // Git uses exit code 128 for protocol/usage errors
+    // If there's stderr content, it's a protocol error message
+    exit_code == Some(128) && !stderr.is_empty()
+}
+
 /// Handle POST /git-upload-pack (clone/fetch)
 pub async fn handle_upload_pack(
     repo_path: PathBuf,
@@ -150,6 +186,21 @@ pub async fn handle_upload_pack(
 
     if !status.success() {
         let stderr_str = String::from_utf8_lossy(&stderr_output);
+
+        // Check if this is a git protocol error (exit code 128 with stderr)
+        // Protocol errors should be returned as HTTP 200 with ERR pkt-line
+        if is_git_protocol_error(status.code(), &stderr_output) {
+            warn!(
+                "Git upload-pack protocol error (returning ERR pkt-line): {}",
+                stderr_str
+            );
+            return Ok(build_git_protocol_error_response(
+                GitService::UploadPack,
+                &stderr_str,
+            ));
+        }
+
+        // Transport errors (spawn failures, signals, etc.) remain as HTTP 500
         error!("Git upload-pack failed: {}", stderr_str);
         return Err(GitError::GitFailed(status.code()));
     }
@@ -277,6 +328,21 @@ pub async fn handle_receive_pack(
 
     if !status.success() {
         let stderr_str = String::from_utf8_lossy(&stderr_output);
+
+        // Check if this is a git protocol error (exit code 128 with stderr)
+        // Protocol errors should be returned as HTTP 200 with ERR pkt-line
+        if is_git_protocol_error(status.code(), &stderr_output) {
+            warn!(
+                "Git receive-pack protocol error (returning ERR pkt-line): {}",
+                stderr_str
+            );
+            return Ok(build_git_protocol_error_response(
+                GitService::ReceivePack,
+                &stderr_str,
+            ));
+        }
+
+        // Transport errors (spawn failures, signals, etc.) remain as HTTP 500
         error!("Git receive-pack failed: {}", stderr_str);
         return Err(GitError::GitFailed(status.code()));
     }