upleb.uk

Public git repos — served from a NIP-34 GRASP relay at git.upleb.uk

summaryrefslogtreecommitdiff
path: root/docs
diff options
context:
space:
mode:
authorYour Name <you@example.com>2026-05-19 19:55:33 +0530
committerYour Name <you@example.com>2026-05-19 20:33:03 +0530
commitc89ab319fd520c9914b015264a26b581e2103954 (patch)
tree5fa6396c9d3f68b300339c28a0b5ead9c2019304 /docs
parent08d7df158acf92399acdbb8a527620a6b1a94f16 (diff)
fix: E2E test stability — socket exhaustion, auto-grant, HTTP robustness
Root causes discovered: - RC-0: LWIP socket exhaustion (CONFIG_LWIP_MAX_SOCKETS=10, need 14) - Two HTTP servers (5 sockets each) + DNS + DoT + wifistr WS = 14 > 10 - Fix: increase to 16, reduce max_open_sockets to 2 on both servers - RC-1: Port 80 captive portal crashes under load - Fix: Connection: close on all handlers, stack 16384 - RC-2: Owner auto-grant makes tests non-deterministic - Fix: remove tollgate_core_fw_grant() from client_connected() Also adds: - /grant_access and /reset_authentication on API server (port 2121) - /portal-config endpoint for future JS-based portal config - Error-checked URI handler registration - Connection: close on captive portal handlers E2E test fixes: - dig +short instead of nslookup for DNS checks - port 2121 for grant/reset/usage/whoami in all tests - pre-mint tokens before blocking internet - increased timeouts and sleeps for reliability
Diffstat (limited to 'docs')
-rw-r--r--docs/E2E_FIX_PLAN.md153
1 files changed, 153 insertions, 0 deletions
diff --git a/docs/E2E_FIX_PLAN.md b/docs/E2E_FIX_PLAN.md
new file mode 100644
index 0000000..480a2d3
--- /dev/null
+++ b/docs/E2E_FIX_PLAN.md
@@ -0,0 +1,153 @@
1# E2E Test Stability Fix Plan
2
3## Problem Statement
4
5E2E tests on physical Board A are failing due to four root causes:
61. **LWIP socket exhaustion** (RC-0) — wifistr WebSockets consume all 10 LWIP sockets
72. **Port 80 captive portal crashes** under load (RC-1)
83. **Owner auto-grant** makes "no internet before auth" tests non-deterministic (RC-2)
94. **No boot-ready probe** — tests start before HTTP servers are up (RC-3)
10
11### Baseline Test Results
12
13| Suite | Pass | Fail | Notes |
14|---|---|---|---|
15| Smoke | 2/6 | 4 | Port 80 unresponsive, cascading failures |
16| Network | 4/7 | 3 | DNS forward + ping after auth (timing) |
17| API | 16/20 | 4 | Portal port 80 slow/crashed, captive URIs |
18| DNS+Firewall | 15/16 | 1 | Ping after auth (timing) |
19| Reset-Auth | 12/15 | 3 | Allotment was 0 (fixed), 2nd payment |
20| Session | 14/14 | 0 | Perfect |
21| Phase 2 | 12/12 | 0 | Perfect |
22
23---
24
25## Root Causes
26
27### RC-0: LWIP socket exhaustion (CRITICAL)
28
29`CONFIG_LWIP_MAX_SOCKETS=10` in sdkconfig. Socket budget at steady state:
30
31| Component | Sockets | Notes |
32|---|---|---|
33| Captive portal (port 80) | 5 | 1 listen + 4 workers (`max_open_sockets=4` default) |
34| API server (port 2121) | 5 | 1 listen + 4 workers |
35| DNS server (UDP 53) | 1 | |
36| DoT reject (TCP 853) | 1 | |
37| wifistr WebSocket x2 | 2 | relay.damus.io + nos.lol |
38| **Total** | **14** | **Exceeds LWIP_MAX_SOCKETS=10 by 4** |
39
40When wifistr opens WebSocket connections (~17s after boot), it exhausts all
41available LWIP sockets. The httpd listening sockets are already bound but
42worker sockets can't accept new connections. TCP SYN gets RST. ICMP (ping)
43still works because it doesn't use LWIP sockets.
44
45**Symptoms observed**:
46- Serial log shows "Captive portal started on port 80" and "API started on port 2121"
47- `ping 10.192.45.1` works but `curl` gets "Connection refused"
48- `nmap -p 80,2121` shows both ports "closed"
49- Board is alive (serial shows wifistr publishing) but HTTP servers are non-functional
50- Even the original (unmodified) firmware exhibits this after erase+reflash
51
52**Fix**: Increase `CONFIG_LWIP_MAX_SOCKETS` to 16. Reduce `max_open_sockets` to 2
53on both servers (saves 4 sockets: 2+2 instead of 4+4 workers).
54
55### RC-1: Port 80 captive portal crashes
56
57The `portal_handler` does per-request `malloc` (~4KB) + two-pass `strstr()`
58template substitution. No caching. OS captive detection probes flood 4 sockets
59simultaneously. No `Connection: close` header means clients hold sockets open.
60
61**Fix**: Add `Connection: close` to all handlers. Increase stack to 16384.
62
63### RC-2: Owner auto-grant
64
65`tollgate_core_client_connected()` grants firewall access to the first WiFi
66client unconditionally. IP is passed as `0` (bug), creating nondeterministic
67behavior for "no internet before auth" tests.
68
69**Fix**: Remove `tollgate_core_fw_grant()` call. Keep owner tracking for logging.
70
71### RC-3: No boot-ready probe
72
73Tests use fixed sleeps after flash. No polling for HTTP server readiness.
74
75**Fix**: Add `arch-wait-ready` Makefile target that polls `:2121/usage`.
76
77---
78
79## Fix Steps
80
81### Step 0: Fix LWIP socket exhaustion (PREREQUISITE)
82- [x] Set `CONFIG_LWIP_MAX_SOCKETS=16` via sdkconfig
83- [x] Set `max_open_sockets = 2` on both HTTP servers
84- [ ] Verify TCP connections work after rebuild + flash
85
86**Files**: `sdkconfig`, `main/captive_portal.c`, `main/tollgate_api.c`
87
88### Step 1: Kill owner auto-grant
89- [x] Remove `tollgate_core_fw_grant()` from `tollgate_core_client_connected()`
90- [x] Keep owner tracking for logging
91- [ ] Verify tests pass without auto-grant
92
93**Files**: `components/tollgate_core/src/tollgate_core.c`
94
95### Step 2: Add `Connection: close` to all port 80 handlers
96- [x] Add `httpd_resp_set_hdr(req, "Connection", "close")` to every handler
97- [x] Increase captive portal stack from 8192 to 16384
98- [ ] Verify port 80 stability under load
99
100**Files**: `main/captive_portal.c`
101
102### Step 3: Add `/portal-config` API endpoint
103- [x] Add `GET /portal-config` on port 2121 returning `{priceSats, mintUrl, ...}`
104- [x] Returns CORS header `Access-Control-Allow-Origin: *`
105- [ ] Verify endpoint returns correct JSON
106
107**Files**: `main/tollgate_api.c`
108
109### Step 4: Remove NAPT flush from `fw_revoke_all()`
110- [x] Remove `ip_napt_enable()` toggle that was causing 30s hangs
111- [ ] Verify `/reset_authentication` responds instantly
112
113**Files**: `components/tollgate_core/src/tollgate_core_firewall.c`
114
115### Step 5: Boot-ready probe in test infrastructure
116- [ ] Add `arch-wait-ready` Makefile target that polls `:2121/usage`
117- [ ] Update `arch-test-full` to call `arch-wait-ready` first
118- [ ] Verify tests work immediately after flash
119
120**Files**: `physical-router-test-automation/esp32/Makefile`
121
122### Step 6: Rebuild and validate
123- [ ] Rebuild firmware with all fixes
124- [ ] Flash to Board A
125- [ ] Run `make arch-test-full`
126- [ ] Document results in this file
127
128---
129
130## Key Architecture Decisions
131
132- **Port 80**: Portal HTML + captive detection URIs only. No API, no state mutation.
133- **Port 2121**: All API operations (discovery, payment, grant, reset, whoami, usage, wallet, portal-config).
134- **Owner tracking**: Kept for logging/display, no longer grants free internet.
135- **Connection: close**: Set on ALL port 80 responses to free sockets immediately.
136- **max_open_sockets = 2**: Conservative to leave headroom for DNS, DoT, wifistr.
137
138## Target Test Results
139
140| Suite | Target | Stretch |
141|---|---|---|
142| Smoke | 6/6 | 6/6 |
143| Network | 6/7 | 7/7 |
144| API | 18/20 | 20/20 |
145| DNS+Firewall | 15/16 | 16/16 |
146| Reset-Auth | 15/15 | 15/15 |
147| Session | 14/14 | 14/14 |
148| Phase 2 | 12/12 | 12/12 |
149
150## Execution Order
151
1520 -> 1 -> 2 -> 3 -> 4 -> 5 -> 6
153(Socket fix -> Owner fix -> Connection close -> Portal config API -> NAPT fix -> Boot probe -> Validate)