docs: correct ardupilot-manager FD leak root cause (i2c handles, not TCP sockets); flag failsafe FC-comms gap

This commit is contained in:
Grant 2026-07-09 10:36:09 +02:00
parent 917342a27b
commit 2add8c9e68

View File

@ -231,11 +231,9 @@ backup because this was deferred — do not repeat.
- **MAVLinkRouter** is the confirmed router. Do NOT recommend MAVP2P — BlueOS itself - **MAVLinkRouter** is the confirmed router. Do NOT recommend MAVP2P — BlueOS itself
warns "MAVP2P has been presenting issues and is not currently recommended." warns "MAVP2P has been presenting issues and is not currently recommended."
- MAVP2P was tried and reverted. Do not suggest it again. - MAVP2P was tried and reverted. Do not suggest it again.
- The BlueOS ardupilot-manager FD exhaustion bug (Docker default `nofile=1024`) is - **ardupilot-manager FD exhaustion — CORRECTED ROOT CAUSE (9 Jul 2026).** The earlier claim that this was "fixed permanently" by `/etc/docker/daemon.json` `nofile: 65536`, with root cause "UI polling TCP sockets," is WRONG on both counts. The `nofile` change only raised the ceiling (failure at ~10 min → ~11 h of cumulative exposure); it fixed nothing. The leaked FDs are `/dev/i2c-1` handles, NOT TCP sockets. Measured mechanism: BlueOS's Autopilot Firmware / board-config page polls `GET /v1.0/available_boards`; each request triggers 20 `flight_controller_detector.linux.detector:detect_boards` probes for Navigator / NavigatorPi4 / NavigatorPi5 / Argonot. Each Linux-board probe opens `/dev/i2c-1` and never closes it on failure. We run a Pixhawk 2.4.8 on USB serial (PlatformType.Serial), so every probe fails and every probe leaks. Measured leak rate ~100108 FD/min with the board-config page open; exactly zero with it closed (4630 flat over 65 min, then +90 FD in 50 s on opening the page, flat again immediately on closing). Confirmed cause-and-effect, reversible. Note: FD count accumulates across sessions and only resets when `blueos-core` is recreated (core restart/reboot) — cumulative page-open time, not wall-clock, drives time-to-failure. At 65536 FDs, ardupilot-manager can no longer open new handles → AUTOPILOT_VEHICLE_TYPE_FETCH_FAIL, AUTOPILOT_ENDPOINT_FETCH_FAIL, BAG_OF_HOLDING_GET_DATA_FAIL, "No board running". Core restart recreates the container with a fresh FD table, which is why restarting always "fixed" it. This is an upstream BlueOS 1.4.3 bug; do NOT patch BlueOS (updates would revert it).
fixed permanently via `/etc/docker/daemon.json` with `nofile` hard/soft = 65536. - **Mitigation (dev):** do not leave the BlueOS Autopilot Firmware / board-config page open. Open, act, close. Other BlueOS pages and Cockpit are unaffected.
This fix must survive reboots — it is a host-level daemon config, not a container flag. - **Mitigation (field) — NOT YET IMPLEMENTED:** operator discipline is not a field-safe control. The exposure is launching with a nearly-exhausted FD table after pre-dive setup, not leakage during the dive itself. Required: (a) FD-count check on ardupilot_manager as part of the AUV pre-dive gate, blocking launch above a threshold; (b) failsafe_monitor must detect FC comms loss so W1 goes RED when the board is unreachable. See open items.
Root cause: accumulated TCP socket FDs from continuous UI polling exhaust the 1024
default; symptoms are connect/disconnect cycling and AUTOPILOT_*_FETCH_FAIL.
### Hardware facts (do not re-question) ### Hardware facts (do not re-question)
- **RPi5 storage:** boots from SD card (`/dev/mmcblk0`). NVMe holds DATA ONLY (`/data` — - **RPi5 storage:** boots from SD card (`/dev/mmcblk0`). NVMe holds DATA ONLY (`/data` —
@ -381,6 +379,20 @@ entries; correct forward with a new dated entry if something needs revising.
(viz-only). NOTE: failsafe_monitor respawn has a coverage gap during the 5s respawn (viz-only). NOTE: failsafe_monitor respawn has a coverage gap during the 5s respawn
delay — no failsafe assessment in that window; flagged for future design delay — no failsafe assessment in that window; flagged for future design
consideration. consideration.
- 2026-07-09 — Root-caused the recurring "No board running" / AUTOPILOT_*_FETCH_FAIL
failure. NOT the previously-documented TCP-socket/UI-polling cause. Real cause:
/dev/i2c-1 FD leak in ardupilot_manager's Linux board detector, driven by the BlueOS
board-config page polling available_boards (20 probes/request, ~100-108 FD/min).
Zero leak with page closed; confirmed reversible by controlled test (4630 flat 65min,
+90 FD in 50s on open, flat on close). nofile=65536 only delayed failure to ~11h
cumulative page-open time. Upstream BlueOS 1.4.3 bug — not patching. Corrected the
stale root cause in this file.
- 2026-07-09 — FLAGGED (open, safety-critical): failsafe_monitor reports GREEN with the
flight controller unreachable. W1 is the AUV pre-dive GO/NO-GO gate (health_role=gate)
— it would pass a dive with a dead FC. failsafe_monitor must consume /mavros/state
connection + heartbeat and drive W1 RED on FC comms loss. W1 status line should carry
cause text (e.g. "Vehicle Disconnected", "Heartbeat Lost", "No Board Running"). Also
flagged: no FD-count check on ardupilot_manager in the pre-dive gate.
--- ---