docs: correct ardupilot-manager FD leak root cause (i2c handles, not TCP sockets); flag failsafe FC-comms gap
This commit is contained in:
parent
917342a27b
commit
2add8c9e68
22
CLAUDE.md
22
CLAUDE.md
@ -231,11 +231,9 @@ backup because this was deferred — do not repeat.
|
||||
- **MAVLinkRouter** is the confirmed router. Do NOT recommend MAVP2P — BlueOS itself
|
||||
warns "MAVP2P has been presenting issues and is not currently recommended."
|
||||
- MAVP2P was tried and reverted. Do not suggest it again.
|
||||
- The BlueOS ardupilot-manager FD exhaustion bug (Docker default `nofile=1024`) is
|
||||
fixed permanently via `/etc/docker/daemon.json` with `nofile` hard/soft = 65536.
|
||||
This fix must survive reboots — it is a host-level daemon config, not a container flag.
|
||||
Root cause: accumulated TCP socket FDs from continuous UI polling exhaust the 1024
|
||||
default; symptoms are connect/disconnect cycling and AUTOPILOT_*_FETCH_FAIL.
|
||||
- **ardupilot-manager FD exhaustion — CORRECTED ROOT CAUSE (9 Jul 2026).** The earlier claim that this was "fixed permanently" by `/etc/docker/daemon.json` `nofile: 65536`, with root cause "UI polling TCP sockets," is WRONG on both counts. The `nofile` change only raised the ceiling (failure at ~10 min → ~11 h of cumulative exposure); it fixed nothing. The leaked FDs are `/dev/i2c-1` handles, NOT TCP sockets. Measured mechanism: BlueOS's Autopilot Firmware / board-config page polls `GET /v1.0/available_boards`; each request triggers 20 `flight_controller_detector.linux.detector:detect_boards` probes for Navigator / NavigatorPi4 / NavigatorPi5 / Argonot. Each Linux-board probe opens `/dev/i2c-1` and never closes it on failure. We run a Pixhawk 2.4.8 on USB serial (PlatformType.Serial), so every probe fails and every probe leaks. Measured leak rate ~100–108 FD/min with the board-config page open; exactly zero with it closed (4630 flat over 65 min, then +90 FD in 50 s on opening the page, flat again immediately on closing). Confirmed cause-and-effect, reversible. Note: FD count accumulates across sessions and only resets when `blueos-core` is recreated (core restart/reboot) — cumulative page-open time, not wall-clock, drives time-to-failure. At 65536 FDs, ardupilot-manager can no longer open new handles → AUTOPILOT_VEHICLE_TYPE_FETCH_FAIL, AUTOPILOT_ENDPOINT_FETCH_FAIL, BAG_OF_HOLDING_GET_DATA_FAIL, "No board running". Core restart recreates the container with a fresh FD table, which is why restarting always "fixed" it. This is an upstream BlueOS 1.4.3 bug; do NOT patch BlueOS (updates would revert it).
|
||||
- **Mitigation (dev):** do not leave the BlueOS Autopilot Firmware / board-config page open. Open, act, close. Other BlueOS pages and Cockpit are unaffected.
|
||||
- **Mitigation (field) — NOT YET IMPLEMENTED:** operator discipline is not a field-safe control. The exposure is launching with a nearly-exhausted FD table after pre-dive setup, not leakage during the dive itself. Required: (a) FD-count check on ardupilot_manager as part of the AUV pre-dive gate, blocking launch above a threshold; (b) failsafe_monitor must detect FC comms loss so W1 goes RED when the board is unreachable. See open items.
|
||||
|
||||
### Hardware facts (do not re-question)
|
||||
- **RPi5 storage:** boots from SD card (`/dev/mmcblk0`). NVMe holds DATA ONLY (`/data` —
|
||||
@ -381,6 +379,20 @@ entries; correct forward with a new dated entry if something needs revising.
|
||||
(viz-only). NOTE: failsafe_monitor respawn has a coverage gap during the 5s respawn
|
||||
delay — no failsafe assessment in that window; flagged for future design
|
||||
consideration.
|
||||
- 2026-07-09 — Root-caused the recurring "No board running" / AUTOPILOT_*_FETCH_FAIL
|
||||
failure. NOT the previously-documented TCP-socket/UI-polling cause. Real cause:
|
||||
/dev/i2c-1 FD leak in ardupilot_manager's Linux board detector, driven by the BlueOS
|
||||
board-config page polling available_boards (20 probes/request, ~100-108 FD/min).
|
||||
Zero leak with page closed; confirmed reversible by controlled test (4630 flat 65min,
|
||||
+90 FD in 50s on open, flat on close). nofile=65536 only delayed failure to ~11h
|
||||
cumulative page-open time. Upstream BlueOS 1.4.3 bug — not patching. Corrected the
|
||||
stale root cause in this file.
|
||||
- 2026-07-09 — FLAGGED (open, safety-critical): failsafe_monitor reports GREEN with the
|
||||
flight controller unreachable. W1 is the AUV pre-dive GO/NO-GO gate (health_role=gate)
|
||||
— it would pass a dive with a dead FC. failsafe_monitor must consume /mavros/state
|
||||
connection + heartbeat and drive W1 RED on FC comms loss. W1 status line should carry
|
||||
cause text (e.g. "Vehicle Disconnected", "Heartbeat Lost", "No Board Running"). Also
|
||||
flagged: no FD-count check on ardupilot_manager in the pre-dive gate.
|
||||
|
||||
---
|
||||
|
||||
|
||||
Loading…
Reference in New Issue
Block a user