1 files changed, 177 insertions, 0 deletions

diff --git a/docs/E2E_FIX_PLAN.md b/docs/E2E_FIX_PLAN.md
new file mode 100644
index 0000000..52f8305
--- /dev/null
+++ b/docs/E2E_FIX_PLAN.md
@@ -0,0 +1,177 @@
+# E2E Test Stability Fix Plan
+
+## Problem Statement
+
+E2E tests on physical boards are failing due to five root causes:
+1. **LWIP socket exhaustion** (RC-0) — `LWIP_MAX_SOCKETS=10` was too low for two httpd servers + DNS + DoT + wifistr WebSockets
+2. **Over-tuned httpd settings** (RC-1) — setting `max_open_sockets=2` and `keep_alive_enable=false` caused socket leaks by interfering with ESP-IDF's internal session management
+3. **Owner auto-grant** (RC-2) — makes "no internet before auth" tests non-deterministic
+4. **No boot-ready probe** (RC-3) — tests start before HTTP servers are up
+5. **Serial monitoring resets** (RC-4) — Python `serial.Serial()` toggles DTR/RTS on USB-Serial/JTAG boards, causing chip resets mid-operation
+
+### Baseline Test Results (Board A, before fixes)
+
+| Suite | Pass | Fail | Notes |
+|---|---|---|---|
+| Smoke | 2/6 | 4 | Port 80 unresponsive, cascading failures |
+| Network | 4/7 | 3 | DNS forward + ping after auth (timing) |
+| API | 16/20 | 4 | Portal port 80 slow/crashed, captive URIs |
+| DNS+Firewall | 15/16 | 1 | Ping after auth (timing) |
+| Reset-Auth | 12/15 | 3 | Allotment was 0 (fixed), 2nd payment |
+| Session | 14/14 | 0 | Perfect |
+| Phase 2 | 12/12 | 0 | Perfect |
+
+### Verified Test Results (Board ACM2, after all fixes, commit `144b48f`)
+
+All API endpoints verified working on AP IP `10.192.45.1` with 2-3s delays between requests:
+- `GET /usage` — returns session/client counts (50/50 sequential requests passed)
+- `GET /portal-config` — returns `{priceSats, stepMs, mintUrl, metric, stepBytes}`
+- `GET /whoami` — returns client IP
+- `GET /grant_access` — grants firewall access
+- `POST /` (payment) — accepts Cashu token, returns `kind:1022`
+- `GET /` (port 80 portal) — returns 3829 bytes HTML
+- `GET /reset_authentication` — clears all sessions and firewall rules
+
+Full payment flow verified: check → pay → verify → grant → portal → reset → verify clean state.
+
+---
+
+## Root Causes
+
+### RC-0: LWIP socket exhaustion (FIXED)
+
+`CONFIG_LWIP_MAX_SOCKETS=10` in sdkconfig. Socket budget at steady state:
+
+| Component | Sockets | Notes |
+|---|---|---|
+| Captive portal (port 80) | 5 | 1 listen + 4 workers (default `max_open_sockets`) |
+| API server (port 2121) | 5 | 1 listen + 4 workers |
+| DNS server (UDP 53) | 1 | |
+| DoT reject (TCP 853) | 1 | |
+| wifistr WebSocket x2 | 2 | relay.damus.io + nos.lol |
+| **Total** | **14** | **Exceeds LWIP_MAX_SOCKETS=10 by 4** |
+
+**Fix** (commit `144b48f`): Set `CONFIG_LWIP_MAX_SOCKETS=20` (matching standalone tollgate). Use default `max_open_sockets=4` on both servers. Previous fix tried `max_open_sockets=2` which caused worse problems (see RC-1).
+
+### RC-1: Over-tuned httpd settings (FIXED)
+
+Initial fix reduced `max_open_sockets` to 2 and added `keep_alive_enable=false`, `linger_timeout=0`. This caused socket leaks — ESP-IDF's httpd manages its own session pool internally, and overriding these settings interfered with socket lifecycle management.
+
+**Symptoms**: Board works for 10-20 requests, then all HTTP becomes unresponsive. Sockets accumulate in CLOSE_WAIT/TIME_WAIT and never get freed.
+
+**Fix** (commit `144b48f`): Reverted to ESP-IDF defaults for all httpd settings except `stack_size=16384` and `max_uri_handlers`. Default `max_open_sockets=4` and `keep_alive_enable=true` (default) work correctly.
+
+### RC-2: Owner auto-grant (FIXED)
+
+`tollgate_core_client_connected()` granted firewall access to the first WiFi client unconditionally. IP was passed as `0` (bug), creating nondeterministic behavior.
+
+**Fix** (commit `c89ab31`): Removed `tollgate_core_fw_grant()` call from `client_connected()`. Owner tracking kept for logging.
+
+### RC-3: No boot-ready probe (PENDING)
+
+Tests use fixed sleeps after flash. No polling for HTTP server readiness.
+
+**Fix**: Add `arch-wait-ready` Makefile target that polls `:2121/usage`.
+
+### RC-4: Serial monitoring resets boards (DISCOVERED)
+
+Python `serial.Serial()` on USB-Serial/JTAG ESP32-S3 boards toggles DTR/RTS during initialization, causing `rst:0x15 (USB_UART_CHIP_RESET)`. This resets the chip even if `dtr=False, rts=False` is set after construction.
+
+**Symptoms**:
+- Board boots successfully, services start, gets IP
+- Python serial read causes immediate `ESP-ROM: boot:0x0 (DOWNLOAD)` or `rst:0x15`
+- Board appears "dead" after testing — actually reset into download mode
+- Earlier sessions attributed this to "socket exhaustion" or "WiFi instability"
+
+**Fix**: Never use Python `serial.Serial()` for monitoring. Use `idf.py monitor` (which handles DTR/RTS correctly) or read-only tools. All hardware access must go through Makefile mutex targets.
+
+---
+
+## Fix Steps
+
+### Step 0: Fix LWIP socket exhaustion — DONE
+- [x] Set `CONFIG_LWIP_MAX_SOCKETS=20` via sdkconfig (commit `144b48f`)
+- [x] Use default `max_open_sockets` on both HTTP servers (removed override)
+- [x] Verified: 50/50 sequential API requests pass on Board ACM2
+
+**Files**: `sdkconfig`, `main/captive_portal.c`, `main/tollgate_api.c`
+
+### Step 1: Kill owner auto-grant — DONE
+- [x] Remove `tollgate_core_fw_grant()` from `tollgate_core_client_connected()` (commit `c89ab31`)
+- [x] Keep owner tracking for logging
+
+**Files**: `components/tollgate_core/src/tollgate_core.c`
+
+### Step 2: HTTP server robustness — DONE
+- [x] Add `Connection: close` header to port 80 responses (commit `c89ab31`)
+- [x] Increase captive portal stack to 16384 (commit `c89ab31`)
+- [x] Use ESP-IDF default socket management (commit `144b48f`)
+
+**Files**: `main/captive_portal.c`, `main/tollgate_api.c`
+
+### Step 3: Add API endpoints — DONE
+- [x] `GET /portal-config` on port 2121 returning `{priceSats, mintUrl, ...}` (commit `c89ab31`)
+- [x] `GET /grant_access` — manual firewall grant (commit `c89ab31`)
+- [x] `GET /reset_authentication` — clear all auth (commit `c89ab31`)
+- [x] CORS header on portal-config
+
+**Files**: `main/tollgate_api.c`
+
+### Step 4: Remove NAPT flush from `fw_revoke_all()` — DONE
+- [x] Remove `ip_napt_enable()` toggle that caused 30s hangs (commit `c89ab31`)
+
+**Files**: `components/tollgate_core/src/tollgate_core_firewall.c`
+
+### Step 5: Boot-ready probe — PENDING
+- [ ] Add `arch-wait-ready` Makefile target that polls `:2121/usage`
+- [ ] Update `arch-test-full` to call `arch-wait-ready` first
+- [ ] Add 2-3 second delays between test requests (burst rate mitigation)
+
+**Files**: `physical-router-test-automation/esp32/Makefile`
+
+### Step 6: Hardware testing — BLOCKED
+- [ ] Flash to working board via Makefile mutex targets
+- [ ] Run `make arch-test-full`
+- [ ] Document results
+- [ ] Board A stuck in download mode (GPIO0 strapping pin) — needs hardware fix
+
+---
+
+## Burst Rate Limitation
+
+On USB-Serial/JTAG ESP32-S3 boards, back-to-back HTTP requests with no delay can
+overwhelm the WiFi AP stack. With 2-3 second delays between requests, the board
+handles 50+ sequential requests reliably. Without delays, rapid bursts of 10+
+requests can cause the WiFi AP to become unresponsive.
+
+**Mitigation**: E2E tests should include a 2-3 second delay between HTTP requests.
+This is a WiFi AP throughput limitation, not a firmware bug.
+
+## Board Status
+
+| Board | Port | MAC | Status |
+|-------|------|-----|--------|
+| Board A | `/dev/ttyACM0` | `94:a9:90:2e:37:7c` | **BROKEN** — stuck in download mode (`boot:0x0`), GPIO0 strapping pin issue, needs hardware fix |
+| Board B | `/dev/ttyACM1` | `fc:01:2c:c5:50:50` | Unknown — newly discovered, needs firmware flash |
+| Board C | `/dev/ttyACM2` | `20:6e:f1:98:d7:08` | **WORKING** — all endpoints verified, payment flow tested |
+
+## Key Architecture Decisions
+
+- **Port 80**: Portal HTML + captive detection URIs only. No API, no state mutation.
+- **Port 2121**: All API operations (discovery, payment, grant, reset, whoami, usage, wallet, portal-config).
+- **Owner tracking**: Kept for logging/display, no longer grants free internet.
+- **Connection: close**: Set on ALL port 80 responses to hint clients.
+- **Default httpd settings**: ESP-IDF's built-in session management works correctly. Do not override `max_open_sockets`, `keep_alive_enable`, `linger_timeout`, or timeouts.
+
+## Execution Order
+
+Steps 0-4 are DONE (commits `c89ab31`, `144b48f`).
+Step 5 (boot-ready probe) is next — code only, no hardware needed.
+Step 6 (validation) requires working board via Makefile mutex targets.
+
+## Hardware Access Rules
+
+- **ALWAYS** use Makefile mutex targets (`make arch-flash-a`, etc.) for hardware access
+- **NEVER** call `esptool.py` directly — bypasses mutex and conflicts with other sessions
+- **NEVER** use Python `serial.Serial()` for monitoring — causes DTR/RTS resets on USB-Serial/JTAG
+- Multiple opencode sessions may be active — mutex prevents board conflicts