From 82f1fc0d5535eda3fc9eab799d81b3e220dbe4ef Mon Sep 17 00:00:00 2001 From: Your Name Date: Wed, 20 May 2026 02:10:01 +0530 Subject: feat: add tollgate_core component + market config wiring - Add tollgate_core ESP-IDF component (skeleton: cashu, dns, firewall, session) - Add tollgate_platform.c with SPIFFS config backend - Wire market_enabled, market_scan_interval_s, client_auto_switch in config.c - Add lwip_tollgate_hooks.h (updated from feature branch) - Add E2E fix plan, tollgate_core design doc, WPA autodetect plan - Add integration test network helpers - Add CONSOLIDATION.md plan Reverts the broken merge (be4788b) that gutted config.c/tollgate_main.c/tollgate_api.c and replaces it with a clean addition on top of intact master. --- docs/E2E_FIX_PLAN.md | 177 +++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 177 insertions(+) create mode 100644 docs/E2E_FIX_PLAN.md (limited to 'docs/E2E_FIX_PLAN.md') diff --git a/docs/E2E_FIX_PLAN.md b/docs/E2E_FIX_PLAN.md new file mode 100644 index 0000000..52f8305 --- /dev/null +++ b/docs/E2E_FIX_PLAN.md @@ -0,0 +1,177 @@ +# E2E Test Stability Fix Plan + +## Problem Statement + +E2E tests on physical boards are failing due to five root causes: +1. **LWIP socket exhaustion** (RC-0) — `LWIP_MAX_SOCKETS=10` was too low for two httpd servers + DNS + DoT + wifistr WebSockets +2. **Over-tuned httpd settings** (RC-1) — setting `max_open_sockets=2` and `keep_alive_enable=false` caused socket leaks by interfering with ESP-IDF's internal session management +3. **Owner auto-grant** (RC-2) — makes "no internet before auth" tests non-deterministic +4. **No boot-ready probe** (RC-3) — tests start before HTTP servers are up +5. **Serial monitoring resets** (RC-4) — Python `serial.Serial()` toggles DTR/RTS on USB-Serial/JTAG boards, causing chip resets mid-operation + +### Baseline Test Results (Board A, before fixes) + +| Suite | Pass | Fail | Notes | +|---|---|---|---| +| Smoke | 2/6 | 4 | Port 80 unresponsive, cascading failures | +| Network | 4/7 | 3 | DNS forward + ping after auth (timing) | +| API | 16/20 | 4 | Portal port 80 slow/crashed, captive URIs | +| DNS+Firewall | 15/16 | 1 | Ping after auth (timing) | +| Reset-Auth | 12/15 | 3 | Allotment was 0 (fixed), 2nd payment | +| Session | 14/14 | 0 | Perfect | +| Phase 2 | 12/12 | 0 | Perfect | + +### Verified Test Results (Board ACM2, after all fixes, commit `144b48f`) + +All API endpoints verified working on AP IP `10.192.45.1` with 2-3s delays between requests: +- `GET /usage` — returns session/client counts (50/50 sequential requests passed) +- `GET /portal-config` — returns `{priceSats, stepMs, mintUrl, metric, stepBytes}` +- `GET /whoami` — returns client IP +- `GET /grant_access` — grants firewall access +- `POST /` (payment) — accepts Cashu token, returns `kind:1022` +- `GET /` (port 80 portal) — returns 3829 bytes HTML +- `GET /reset_authentication` — clears all sessions and firewall rules + +Full payment flow verified: check → pay → verify → grant → portal → reset → verify clean state. + +--- + +## Root Causes + +### RC-0: LWIP socket exhaustion (FIXED) + +`CONFIG_LWIP_MAX_SOCKETS=10` in sdkconfig. Socket budget at steady state: + +| Component | Sockets | Notes | +|---|---|---| +| Captive portal (port 80) | 5 | 1 listen + 4 workers (default `max_open_sockets`) | +| API server (port 2121) | 5 | 1 listen + 4 workers | +| DNS server (UDP 53) | 1 | | +| DoT reject (TCP 853) | 1 | | +| wifistr WebSocket x2 | 2 | relay.damus.io + nos.lol | +| **Total** | **14** | **Exceeds LWIP_MAX_SOCKETS=10 by 4** | + +**Fix** (commit `144b48f`): Set `CONFIG_LWIP_MAX_SOCKETS=20` (matching standalone tollgate). Use default `max_open_sockets=4` on both servers. Previous fix tried `max_open_sockets=2` which caused worse problems (see RC-1). + +### RC-1: Over-tuned httpd settings (FIXED) + +Initial fix reduced `max_open_sockets` to 2 and added `keep_alive_enable=false`, `linger_timeout=0`. This caused socket leaks — ESP-IDF's httpd manages its own session pool internally, and overriding these settings interfered with socket lifecycle management. + +**Symptoms**: Board works for 10-20 requests, then all HTTP becomes unresponsive. Sockets accumulate in CLOSE_WAIT/TIME_WAIT and never get freed. + +**Fix** (commit `144b48f`): Reverted to ESP-IDF defaults for all httpd settings except `stack_size=16384` and `max_uri_handlers`. Default `max_open_sockets=4` and `keep_alive_enable=true` (default) work correctly. + +### RC-2: Owner auto-grant (FIXED) + +`tollgate_core_client_connected()` granted firewall access to the first WiFi client unconditionally. IP was passed as `0` (bug), creating nondeterministic behavior. + +**Fix** (commit `c89ab31`): Removed `tollgate_core_fw_grant()` call from `client_connected()`. Owner tracking kept for logging. + +### RC-3: No boot-ready probe (PENDING) + +Tests use fixed sleeps after flash. No polling for HTTP server readiness. + +**Fix**: Add `arch-wait-ready` Makefile target that polls `:2121/usage`. + +### RC-4: Serial monitoring resets boards (DISCOVERED) + +Python `serial.Serial()` on USB-Serial/JTAG ESP32-S3 boards toggles DTR/RTS during initialization, causing `rst:0x15 (USB_UART_CHIP_RESET)`. This resets the chip even if `dtr=False, rts=False` is set after construction. + +**Symptoms**: +- Board boots successfully, services start, gets IP +- Python serial read causes immediate `ESP-ROM: boot:0x0 (DOWNLOAD)` or `rst:0x15` +- Board appears "dead" after testing — actually reset into download mode +- Earlier sessions attributed this to "socket exhaustion" or "WiFi instability" + +**Fix**: Never use Python `serial.Serial()` for monitoring. Use `idf.py monitor` (which handles DTR/RTS correctly) or read-only tools. All hardware access must go through Makefile mutex targets. + +--- + +## Fix Steps + +### Step 0: Fix LWIP socket exhaustion — DONE +- [x] Set `CONFIG_LWIP_MAX_SOCKETS=20` via sdkconfig (commit `144b48f`) +- [x] Use default `max_open_sockets` on both HTTP servers (removed override) +- [x] Verified: 50/50 sequential API requests pass on Board ACM2 + +**Files**: `sdkconfig`, `main/captive_portal.c`, `main/tollgate_api.c` + +### Step 1: Kill owner auto-grant — DONE +- [x] Remove `tollgate_core_fw_grant()` from `tollgate_core_client_connected()` (commit `c89ab31`) +- [x] Keep owner tracking for logging + +**Files**: `components/tollgate_core/src/tollgate_core.c` + +### Step 2: HTTP server robustness — DONE +- [x] Add `Connection: close` header to port 80 responses (commit `c89ab31`) +- [x] Increase captive portal stack to 16384 (commit `c89ab31`) +- [x] Use ESP-IDF default socket management (commit `144b48f`) + +**Files**: `main/captive_portal.c`, `main/tollgate_api.c` + +### Step 3: Add API endpoints — DONE +- [x] `GET /portal-config` on port 2121 returning `{priceSats, mintUrl, ...}` (commit `c89ab31`) +- [x] `GET /grant_access` — manual firewall grant (commit `c89ab31`) +- [x] `GET /reset_authentication` — clear all auth (commit `c89ab31`) +- [x] CORS header on portal-config + +**Files**: `main/tollgate_api.c` + +### Step 4: Remove NAPT flush from `fw_revoke_all()` — DONE +- [x] Remove `ip_napt_enable()` toggle that caused 30s hangs (commit `c89ab31`) + +**Files**: `components/tollgate_core/src/tollgate_core_firewall.c` + +### Step 5: Boot-ready probe — PENDING +- [ ] Add `arch-wait-ready` Makefile target that polls `:2121/usage` +- [ ] Update `arch-test-full` to call `arch-wait-ready` first +- [ ] Add 2-3 second delays between test requests (burst rate mitigation) + +**Files**: `physical-router-test-automation/esp32/Makefile` + +### Step 6: Hardware testing — BLOCKED +- [ ] Flash to working board via Makefile mutex targets +- [ ] Run `make arch-test-full` +- [ ] Document results +- [ ] Board A stuck in download mode (GPIO0 strapping pin) — needs hardware fix + +--- + +## Burst Rate Limitation + +On USB-Serial/JTAG ESP32-S3 boards, back-to-back HTTP requests with no delay can +overwhelm the WiFi AP stack. With 2-3 second delays between requests, the board +handles 50+ sequential requests reliably. Without delays, rapid bursts of 10+ +requests can cause the WiFi AP to become unresponsive. + +**Mitigation**: E2E tests should include a 2-3 second delay between HTTP requests. +This is a WiFi AP throughput limitation, not a firmware bug. + +## Board Status + +| Board | Port | MAC | Status | +|-------|------|-----|--------| +| Board A | `/dev/ttyACM0` | `94:a9:90:2e:37:7c` | **BROKEN** — stuck in download mode (`boot:0x0`), GPIO0 strapping pin issue, needs hardware fix | +| Board B | `/dev/ttyACM1` | `fc:01:2c:c5:50:50` | Unknown — newly discovered, needs firmware flash | +| Board C | `/dev/ttyACM2` | `20:6e:f1:98:d7:08` | **WORKING** — all endpoints verified, payment flow tested | + +## Key Architecture Decisions + +- **Port 80**: Portal HTML + captive detection URIs only. No API, no state mutation. +- **Port 2121**: All API operations (discovery, payment, grant, reset, whoami, usage, wallet, portal-config). +- **Owner tracking**: Kept for logging/display, no longer grants free internet. +- **Connection: close**: Set on ALL port 80 responses to hint clients. +- **Default httpd settings**: ESP-IDF's built-in session management works correctly. Do not override `max_open_sockets`, `keep_alive_enable`, `linger_timeout`, or timeouts. + +## Execution Order + +Steps 0-4 are DONE (commits `c89ab31`, `144b48f`). +Step 5 (boot-ready probe) is next — code only, no hardware needed. +Step 6 (validation) requires working board via Makefile mutex targets. + +## Hardware Access Rules + +- **ALWAYS** use Makefile mutex targets (`make arch-flash-a`, etc.) for hardware access +- **NEVER** call `esptool.py` directly — bypasses mutex and conflicts with other sessions +- **NEVER** use Python `serial.Serial()` for monitoring — causes DTR/RTS resets on USB-Serial/JTAG +- Multiple opencode sessions may be active — mutex prevents board conflicts -- cgit v1.2.3