diff options
| author | Your Name <you@example.com> | 2026-05-19 19:55:33 +0530 |
|---|---|---|
| committer | Your Name <you@example.com> | 2026-05-19 20:33:03 +0530 |
| commit | c89ab319fd520c9914b015264a26b581e2103954 (patch) | |
| tree | 5fa6396c9d3f68b300339c28a0b5ead9c2019304 /docs | |
| parent | 08d7df158acf92399acdbb8a527620a6b1a94f16 (diff) | |
fix: E2E test stability — socket exhaustion, auto-grant, HTTP robustness
Root causes discovered:
- RC-0: LWIP socket exhaustion (CONFIG_LWIP_MAX_SOCKETS=10, need 14)
- Two HTTP servers (5 sockets each) + DNS + DoT + wifistr WS = 14 > 10
- Fix: increase to 16, reduce max_open_sockets to 2 on both servers
- RC-1: Port 80 captive portal crashes under load
- Fix: Connection: close on all handlers, stack 16384
- RC-2: Owner auto-grant makes tests non-deterministic
- Fix: remove tollgate_core_fw_grant() from client_connected()
Also adds:
- /grant_access and /reset_authentication on API server (port 2121)
- /portal-config endpoint for future JS-based portal config
- Error-checked URI handler registration
- Connection: close on captive portal handlers
E2E test fixes:
- dig +short instead of nslookup for DNS checks
- port 2121 for grant/reset/usage/whoami in all tests
- pre-mint tokens before blocking internet
- increased timeouts and sleeps for reliability
Diffstat (limited to 'docs')
| -rw-r--r-- | docs/E2E_FIX_PLAN.md | 153 |
1 files changed, 153 insertions, 0 deletions
diff --git a/docs/E2E_FIX_PLAN.md b/docs/E2E_FIX_PLAN.md new file mode 100644 index 0000000..480a2d3 --- /dev/null +++ b/docs/E2E_FIX_PLAN.md | |||
| @@ -0,0 +1,153 @@ | |||
| 1 | # E2E Test Stability Fix Plan | ||
| 2 | |||
| 3 | ## Problem Statement | ||
| 4 | |||
| 5 | E2E tests on physical Board A are failing due to four root causes: | ||
| 6 | 1. **LWIP socket exhaustion** (RC-0) — wifistr WebSockets consume all 10 LWIP sockets | ||
| 7 | 2. **Port 80 captive portal crashes** under load (RC-1) | ||
| 8 | 3. **Owner auto-grant** makes "no internet before auth" tests non-deterministic (RC-2) | ||
| 9 | 4. **No boot-ready probe** — tests start before HTTP servers are up (RC-3) | ||
| 10 | |||
| 11 | ### Baseline Test Results | ||
| 12 | |||
| 13 | | Suite | Pass | Fail | Notes | | ||
| 14 | |---|---|---|---| | ||
| 15 | | Smoke | 2/6 | 4 | Port 80 unresponsive, cascading failures | | ||
| 16 | | Network | 4/7 | 3 | DNS forward + ping after auth (timing) | | ||
| 17 | | API | 16/20 | 4 | Portal port 80 slow/crashed, captive URIs | | ||
| 18 | | DNS+Firewall | 15/16 | 1 | Ping after auth (timing) | | ||
| 19 | | Reset-Auth | 12/15 | 3 | Allotment was 0 (fixed), 2nd payment | | ||
| 20 | | Session | 14/14 | 0 | Perfect | | ||
| 21 | | Phase 2 | 12/12 | 0 | Perfect | | ||
| 22 | |||
| 23 | --- | ||
| 24 | |||
| 25 | ## Root Causes | ||
| 26 | |||
| 27 | ### RC-0: LWIP socket exhaustion (CRITICAL) | ||
| 28 | |||
| 29 | `CONFIG_LWIP_MAX_SOCKETS=10` in sdkconfig. Socket budget at steady state: | ||
| 30 | |||
| 31 | | Component | Sockets | Notes | | ||
| 32 | |---|---|---| | ||
| 33 | | Captive portal (port 80) | 5 | 1 listen + 4 workers (`max_open_sockets=4` default) | | ||
| 34 | | API server (port 2121) | 5 | 1 listen + 4 workers | | ||
| 35 | | DNS server (UDP 53) | 1 | | | ||
| 36 | | DoT reject (TCP 853) | 1 | | | ||
| 37 | | wifistr WebSocket x2 | 2 | relay.damus.io + nos.lol | | ||
| 38 | | **Total** | **14** | **Exceeds LWIP_MAX_SOCKETS=10 by 4** | | ||
| 39 | |||
| 40 | When wifistr opens WebSocket connections (~17s after boot), it exhausts all | ||
| 41 | available LWIP sockets. The httpd listening sockets are already bound but | ||
| 42 | worker sockets can't accept new connections. TCP SYN gets RST. ICMP (ping) | ||
| 43 | still works because it doesn't use LWIP sockets. | ||
| 44 | |||
| 45 | **Symptoms observed**: | ||
| 46 | - Serial log shows "Captive portal started on port 80" and "API started on port 2121" | ||
| 47 | - `ping 10.192.45.1` works but `curl` gets "Connection refused" | ||
| 48 | - `nmap -p 80,2121` shows both ports "closed" | ||
| 49 | - Board is alive (serial shows wifistr publishing) but HTTP servers are non-functional | ||
| 50 | - Even the original (unmodified) firmware exhibits this after erase+reflash | ||
| 51 | |||
| 52 | **Fix**: Increase `CONFIG_LWIP_MAX_SOCKETS` to 16. Reduce `max_open_sockets` to 2 | ||
| 53 | on both servers (saves 4 sockets: 2+2 instead of 4+4 workers). | ||
| 54 | |||
| 55 | ### RC-1: Port 80 captive portal crashes | ||
| 56 | |||
| 57 | The `portal_handler` does per-request `malloc` (~4KB) + two-pass `strstr()` | ||
| 58 | template substitution. No caching. OS captive detection probes flood 4 sockets | ||
| 59 | simultaneously. No `Connection: close` header means clients hold sockets open. | ||
| 60 | |||
| 61 | **Fix**: Add `Connection: close` to all handlers. Increase stack to 16384. | ||
| 62 | |||
| 63 | ### RC-2: Owner auto-grant | ||
| 64 | |||
| 65 | `tollgate_core_client_connected()` grants firewall access to the first WiFi | ||
| 66 | client unconditionally. IP is passed as `0` (bug), creating nondeterministic | ||
| 67 | behavior for "no internet before auth" tests. | ||
| 68 | |||
| 69 | **Fix**: Remove `tollgate_core_fw_grant()` call. Keep owner tracking for logging. | ||
| 70 | |||
| 71 | ### RC-3: No boot-ready probe | ||
| 72 | |||
| 73 | Tests use fixed sleeps after flash. No polling for HTTP server readiness. | ||
| 74 | |||
| 75 | **Fix**: Add `arch-wait-ready` Makefile target that polls `:2121/usage`. | ||
| 76 | |||
| 77 | --- | ||
| 78 | |||
| 79 | ## Fix Steps | ||
| 80 | |||
| 81 | ### Step 0: Fix LWIP socket exhaustion (PREREQUISITE) | ||
| 82 | - [x] Set `CONFIG_LWIP_MAX_SOCKETS=16` via sdkconfig | ||
| 83 | - [x] Set `max_open_sockets = 2` on both HTTP servers | ||
| 84 | - [ ] Verify TCP connections work after rebuild + flash | ||
| 85 | |||
| 86 | **Files**: `sdkconfig`, `main/captive_portal.c`, `main/tollgate_api.c` | ||
| 87 | |||
| 88 | ### Step 1: Kill owner auto-grant | ||
| 89 | - [x] Remove `tollgate_core_fw_grant()` from `tollgate_core_client_connected()` | ||
| 90 | - [x] Keep owner tracking for logging | ||
| 91 | - [ ] Verify tests pass without auto-grant | ||
| 92 | |||
| 93 | **Files**: `components/tollgate_core/src/tollgate_core.c` | ||
| 94 | |||
| 95 | ### Step 2: Add `Connection: close` to all port 80 handlers | ||
| 96 | - [x] Add `httpd_resp_set_hdr(req, "Connection", "close")` to every handler | ||
| 97 | - [x] Increase captive portal stack from 8192 to 16384 | ||
| 98 | - [ ] Verify port 80 stability under load | ||
| 99 | |||
| 100 | **Files**: `main/captive_portal.c` | ||
| 101 | |||
| 102 | ### Step 3: Add `/portal-config` API endpoint | ||
| 103 | - [x] Add `GET /portal-config` on port 2121 returning `{priceSats, mintUrl, ...}` | ||
| 104 | - [x] Returns CORS header `Access-Control-Allow-Origin: *` | ||
| 105 | - [ ] Verify endpoint returns correct JSON | ||
| 106 | |||
| 107 | **Files**: `main/tollgate_api.c` | ||
| 108 | |||
| 109 | ### Step 4: Remove NAPT flush from `fw_revoke_all()` | ||
| 110 | - [x] Remove `ip_napt_enable()` toggle that was causing 30s hangs | ||
| 111 | - [ ] Verify `/reset_authentication` responds instantly | ||
| 112 | |||
| 113 | **Files**: `components/tollgate_core/src/tollgate_core_firewall.c` | ||
| 114 | |||
| 115 | ### Step 5: Boot-ready probe in test infrastructure | ||
| 116 | - [ ] Add `arch-wait-ready` Makefile target that polls `:2121/usage` | ||
| 117 | - [ ] Update `arch-test-full` to call `arch-wait-ready` first | ||
| 118 | - [ ] Verify tests work immediately after flash | ||
| 119 | |||
| 120 | **Files**: `physical-router-test-automation/esp32/Makefile` | ||
| 121 | |||
| 122 | ### Step 6: Rebuild and validate | ||
| 123 | - [ ] Rebuild firmware with all fixes | ||
| 124 | - [ ] Flash to Board A | ||
| 125 | - [ ] Run `make arch-test-full` | ||
| 126 | - [ ] Document results in this file | ||
| 127 | |||
| 128 | --- | ||
| 129 | |||
| 130 | ## Key Architecture Decisions | ||
| 131 | |||
| 132 | - **Port 80**: Portal HTML + captive detection URIs only. No API, no state mutation. | ||
| 133 | - **Port 2121**: All API operations (discovery, payment, grant, reset, whoami, usage, wallet, portal-config). | ||
| 134 | - **Owner tracking**: Kept for logging/display, no longer grants free internet. | ||
| 135 | - **Connection: close**: Set on ALL port 80 responses to free sockets immediately. | ||
| 136 | - **max_open_sockets = 2**: Conservative to leave headroom for DNS, DoT, wifistr. | ||
| 137 | |||
| 138 | ## Target Test Results | ||
| 139 | |||
| 140 | | Suite | Target | Stretch | | ||
| 141 | |---|---|---| | ||
| 142 | | Smoke | 6/6 | 6/6 | | ||
| 143 | | Network | 6/7 | 7/7 | | ||
| 144 | | API | 18/20 | 20/20 | | ||
| 145 | | DNS+Firewall | 15/16 | 16/16 | | ||
| 146 | | Reset-Auth | 15/15 | 15/15 | | ||
| 147 | | Session | 14/14 | 14/14 | | ||
| 148 | | Phase 2 | 12/12 | 12/12 | | ||
| 149 | |||
| 150 | ## Execution Order | ||
| 151 | |||
| 152 | 0 -> 1 -> 2 -> 3 -> 4 -> 5 -> 6 | ||
| 153 | (Socket fix -> Owner fix -> Connection close -> Portal config API -> NAPT fix -> Boot probe -> Validate) | ||