Grant 5551173b7a docs: restructure into docs/, CLAUDE.md rules-only, DIR v1.2, handover v2.24

2026-07-11 18:46:58 +02:00

77 KiB

Raw Blame History

ROV Autonomous Inspection System — Project Handover

Project Name: Argonaut 3 Version: 2.24 Date: 11 July 2026 File: docs/handover.md (repo-tracked, no version number in filename — see Version History for versioning)

This document is the single source of truth for current project state — what's built, verified, broken, parked, and next. Rules for how Claude works are in CLAUDE.md (repo root). Design intent ("why") is in docs/design_intent_register.md. Domain specs (UI, failsafe, sim/ML, recording lifecycle) are in docs/, referenced from §0 and inline, not duplicated here.

Status: ⚠ CORRECTION (still open) — the BlueOS "FD exhaustion" root cause recorded in v2.10–v2.21 was WRONG. It is NOT TCP-socket/uvicorn exhaustion from UI polling, and nofile=65536 did NOT fix it — that change only raised the ceiling (failure at ~10 min → ~11 h of cumulative exposure). The real cause is a /dev/i2c-1 file-descriptor leak in ardupilot_manager's Linux board detector, driven by the BlueOS Autopilot Firmware / board-config page. See §15a. This bug is OPEN.

✅ Resolved this session — the vehicle-health/FC-comms-loss fault (was OPEN, SAFETY-CRITICAL as of v2.22) is fixed: failsafe_monitor now drives W1 RED on FC comms loss, verified live end-to-end. See §15d and Changelog, 11 Jul entries.

Full narrative for each prior version is in Version History below — not repeated here.

0. QUICK STATUS BOARD (read this first)

CONFIRMED WORKING (verified across sessions to date):

Full autonomy stack — 19 nodes, EKF ~11 Hz on /odometry/filtered, /tf valid. Runs as systemd argonaut.service (manual-start in dev — verify BlueOS heartbeat first).
Pixhawk 2.4.8 — confirmed active on /dev/ttyACM0. connected: true, system_status: 3 (STANDBY), mode: MANUAL. SITL is NOT used in production. BlueOS preferred_board: Pixhawk1.
MAVLink routing — mavlink-routerd running on ttyACM0:115200. MAVLinkRouter is the confirmed router. MAVP2P has known issues (BlueOS own warning) — do not use, do not suggest.
~~BlueOS FD exhaustion — permanently fixed~~ ✗ RETRACTED (9 Jul 2026). NOT FIXED. See §15a. The nofile: 65536 change raised the ceiling only. Real cause is a /dev/i2c-1 FD leak in ardupilot_manager's Linux board detector. Bug is OPEN. Dev mitigation: do not leave the BlueOS Autopilot Firmware / board-config page open.
cockpit_bridge node — ROS2 → WebSocket :9001 → Cockpit data lake. 19 variables live (8 telemetry + 11 mode-profile flags).
Operating-mode profile system — mode_profile_loader reads mode_profiles.yaml (ROV/AUV defaults + custom hybrid via Save As), publishes the resolved flag set latched on /rov/mode/profile, reloadable via /rov/mode/reload (std_srvs/Trigger). cockpit_bridge surfaces the flags to the data lake as external/rov-*. Widgets read flags, not modes; hybrid = any saved profile with overrides. Verified end-to-end: ROV/AUV resolve + live reload. Commit 4e48dc2.
Recording Manager — recording_manager.py node active in argonaut.service. Publishes /rov/recording/active (Bool). Confirmed data: true in full stack. Bags to /data/bags/. Commit 5d75a9d.
/data/bags/ — created on NVMe. Recording live at ~2.5 MB/s (nav/MAVLink topics, no camera). 512GB NVMe will last weeks at this rate.
W0 Data Lake Inspector — confirmed working. 706/706 vars live.
W1 System Health Indicator — failsafe_monitor now drives RED on FC comms loss (11 Jul 2026). Previously visual-confirmed only (GREEN, full test pending); now verified live end-to-end: kill mavros → RED + "Heartbeat Lost", restart → recover to GREEN. Four commits 86b2728/1b0050d/dbe9515/3409ae5 — full detail in Changelog below. §15d (was OPEN, SAFETY-CRITICAL) is resolved.
W6 Altitude Widget — deployed. NO SONAR state confirmed (expected — no Ping2 connected). Commit 0640353.
Backup system — push model, both Pis → server. RPi4: 00:00 daily. RPi5: 00:30 daily. Server monitor alerts on FAILED (always) and OVERDUE ≥2 days.
FastAPI backend (rov_api) — deployed and verified end-to-end on RPi5:8081. Runs as argonaut-api.service (manual-start in dev). Endpoints confirmed: GET /health (live mission/recording/failsafe), POST /abort (W3 → /rov/mission/abort → failsafe_monitor FSM NORMAL→HOLD_AND_RECOVER), POST /mission/start (MissionCommand START), POST /mission/stop (MissionCommand ABORT). Commit 66a18ef.
Deploy files version-controlled — repo deploy/ folder holds both services' units + wrappers + README. Commit 97decac.
Hostname resolution (DNS) — symbytech/blueos/rov-brain resolve on laptop (Windows hosts) and RPi5 (/etc/hosts). Git remotes on hostname form.
Gitea remote — hostname form: ssh://git@symbytech:222/administrator/rov-autonomy.git
All three (Gitea/laptop/RPi5) level at 4e48dc2.

UNTESTED (visual/UI confirmed, real data not flowing yet):

W2 Mission Status — UI confirmed (IDLE state + progress bar). Untested: needs a mission running. Test: POST /mission/start via FastAPI (now available) with a loaded plan, observe widget.
W4 Mission Setup Button — UI confirmed (cyan button, opens setup URL). Untested: needs a setup page served by FastAPI (not yet built — Stage 2b/Phase 6).
W5 Return Budget — UI confirmed (shows pending). Untested: bridge does not yet publish external/rov-return-budget. Backend deferred to Stage 2b (return budget approach under discussion — "shortest path to safety" concept to be defined).

BLOCKED / NOT BUILT:

W3 RETURN TO SAFE button — backend now LIVE (/abort verified end-to-end). Remaining: update widget code label from "ABORT MISSION" to "RETURN TO SAFE" and set FASTAPI_HOST to http://192.168.1.101:8081.
W4 setup page — the page FastAPI would serve is not built (Stage 2b/Phase 6).
W5 return budget — /return_budget endpoint deferred to Stage 2b.
W7 Backup Due Notice — not built. Depends on FastAPI /backup/* (Stage 2b, Option A design: server exposes read-only JSON status, RPi5 polls outbound).
external/rov-return-budget — not published by cockpit_bridge. W5 budget figure pending.

OPEN ITEMS / HOUSEKEEPING:

W3 widget code: update button label to "RETURN TO SAFE"; set FASTAPI_HOST constant.
W6 full test: hardware-gated on Ping2 sonar purchase (Phase 3, ~$280 USD).
RPi5 SSH password: weak dev password. Harden before field deployment (bundled with UFW).
UFW: inactive on RPi5. Deferred to pre-field/pre-remote hardening pass (no online exposure currently — Tailscale disabled, no proxy to Pis). Bundle with SSH password change.
Widget consolidation: Grant to review as development progresses — may consolidate W0–W7 into a single unified widget. Flagged thought, not yet designed.
W2, W4, W5: untested. Require system testing with live mission / backend pages.
Optional: RPi5 remote was on hostname form after DNS fix — consistent with laptop. Done.

PARKED (design decisions deferred, priority order — moved here from CLAUDE.md 11 Jul 2026):

~~Doc restructure~~ — DONE this session (11 Jul 2026): this document, CLAUDE.md, and docs/design_intent_register.md restructured/updated; all docs moved into repo docs/.
Mode-dependent failsafe recovery — AUV (health_role: gate) commits to recovery on condition-clear, does not auto-resume; ROV (advisory) holds and hands the continue-or-recover decision to the operator. Requires failsafe_monitor to read the mode profile (currently mode-blind). Intent now recorded as a DIR-7 addendum (11 Jul 2026, docs/design_intent_register.md) — implementation still open.
Auto-recovery — failsafe_monitor should attempt to re-establish the FC link itself, escalating/holding RED only if it cannot. Real-world dives have no operator to restore comms. Intent now recorded in DIR's Parked Design Items (11 Jul 2026) — implementation still open.
Dedicated string cause field in FailsafeStatus.msg, to replace the message-substring shortcut currently used in cockpit_bridge. Intent now recorded in DIR's Parked Design Items (11 Jul 2026) — implementation still open.
Operator control surface (start/pause/stop via rov_api) and one-command stack startup + healthcheck.
UFW / SSH pre-field hardening — see Open Items above.
In-mission AMBER runtime-response — AMBER means action-required once a mission is underway, not just informational; response branches by mode. Intent now recorded as a DIR-7 addendum (11 Jul 2026) — AUV's autonomous curtailment logic is unspecified, left for a dedicated design session.

NEXT (priority order, carried from §12):

W3 widget code: update label to "RETURN TO SAFE", set FASTAPI_HOST = http://192.168.1.101:8081, then test W3 end-to-end (button → /abort → FSM, already verified server-side).
Stage 2b design decisions (deferred pending Grant): return budget "shortest path to safety" concept; W7 backup notice (Option A confirmed: server read-only JSON, RPi5 polls outbound).
W2 testing: test with live mission via POST /mission/start (backend now available)
- loaded plan.
Ping2 sonar purchase (~$280 USD): required for W6 full end-to-end test.
Pre-field hardening pass: UFW + RPi5 SSH password (bundled, triggered before remote ops/field).

DATA LAKE VARIABLES (19 total, published by cockpit_bridge):

Variable	Source topic	Notes
`external/rov-failsafe`	`/rov/failsafe`	0=GREEN, 1=AMBER, 2=RED; -1=no data
`external/rov-depth`	`/rov/depth`	Depth in metres (3 dp)
`external/rov-voltage`	`/mavros/battery`	Battery volts (2 dp)
`external/rov-heading`	`/mavros/mavros/compass_hdg`	Compass degrees (1 dp)
`external/rov-ms`	`/rov/mission/status`	Mission state: 0=IDLE…4=ABORTED; -1=no data
`external/rov-mp`	`/rov/mission/status`	Mission progress 0-100%
`external/rov-recording`	`/rov/recording/active`	1=recording, 0=not recording
`external/rov-altitude`	`/ping2/altitude`	Ping2 altitude metres; -1.0=NO SONAR
`external/rov-mode`	`/rov/mode/profile`	Active profile name (string): ROV / AUV / hybrid
`external/rov-mode-base`	`/rov/mode/profile`	Base mode: 0=ROV, 1=AUV
`external/rov-hilt`	`/rov/mode/profile`	HILT link: 1=present, 0=absent
`external/rov-health-role`	`/rov/mode/profile`	W1 role: 0=advisory, 1=gate
`external/rov-rec-control`	`/rov/mode/profile`	Recording: 0=manual, 1=auto (ARM-bounded)
`external/rov-nogo-gate`	`/rov/mode/profile`	Recording no-go: 0=relaxed, 1=enforced
`external/rov-rec-reminder`	`/rov/mode/profile`	ARM reminder (manual): 1=on, 0=off
`external/rov-return-mode`	`/rov/mode/profile`	Return: 0=operator-selectable, 1=autonomous
`external/rov-live-telem`	`/rov/mode/profile`	Live telemetry: 0=on, 1=predive-only
`external/rov-nav-display`	`/rov/mode/profile`	Nav display: 1=shown to pilot, 0=autonomy only
`external/rov-camera-role`	`/rov/mode/profile`	Camera: 0=pilot-primary, 1=record-only

RECENT COMMITS (latest first):

Hash	Description
`3409ae5`	W1 widget: state word on #w1-message (no flicker), cause text red 11px on #w1-footer
`dbe9515`	cockpit_bridge: publish external/rov-failsafe-cause (substring stopgap per DIR-7)
`d854641`	Fix cockpit_bridge publishing failsafe_state (FSM) instead of assessment_state to rov-failsafe — real cause of that day's "no data" symptom
`1b0050d`	failsafe_monitor FSM fix: RETURN_TO_SAFE was a terminal trap, both states now recover to NORMAL on condition-clear
`86b2728`	failsafe_monitor: comms loss forces unconditional RED + fcu_ever_connected latch + cause text on FailsafeStatus.message
`4e48dc2`	feat(mode): operating-mode profile system — ModeProfile.msg, mode_profile_loader, mode_profiles.yaml, cockpit_bridge flags
`785ec56`	Widgets housekeeping: add W6 (`w6_altitude.json`), remove probe widget
`97decac`	Add deploy/ folder — systemd units and launch wrappers + README
`66a18ef`	Phase 3 Stage 2: rov_api FastAPI backend (core) — /health, /abort, /mission/start, /mission/stop
`0640353`	W6: Add rov-altitude to cockpit_bridge for Ping2 downward sonar
`5d75a9d`	Phase 3 Stage 1: Recording Manager (DIR-9)
`2cb7b74`	W0 layout fix (3-column, horizontal scroll)
`66f9bd7`	W0–W5 all confirmed working, committed
`c94412d`	Full stack zero-crash launch (MAVROS fcu_url, plugin_allowlist, QoS, battery guard)
`12b5161`	cockpit_bridge node — ROS2→WebSocket :9001

Table has historically lagged actual commit history (e.g. 9109439, 87000e6, c099c00, a5e1c99 from the 8–9 Jul session were never added) — not reconstructed retroactively here without measuring actual git log; flagged so it doesn't look more complete than it is.

Version History

Version	Date	Changes
2.24	11 Jul 2026	§15d resolved. Vehicle health / FC-comms-loss fault (discovered 9 Jul, flagged OPEN/SAFETY-CRITICAL in v2.22) is fixed as of four commits today: `86b2728` (unconditional RED on comms loss + `fcu_ever_connected` latch + cause text), `1b0050d` (FSM recovery fix — `RETURN_TO_SAFE` was a terminal trap), `dbe9515` (new `external/rov-failsafe-cause` variable), `3409ae5` (W1 widget shows state + cause text). Verified live end-to-end. Related fault found+fixed same session: `cockpit_bridge` was publishing `failsafe_state` (FSM, 3+) not `assessment_state` (0/1/2) to `rov-failsafe` — the actual cause of that day's "widgets show no data," fixed in `d854641`, compounded by a restart-storm wedging port 9001 (→ CLAUDE.md's one-restart-then-verify-9001 rule). §0 Confirmed Working and Recent Commits updated; PARKED list updated to reflect intent now recorded in the DIR (mode-dependent recovery, auto-recovery, dedicated cause field, AMBER runtime-response — all still open on implementation, closed on design intent). Companion DIR bump: `docs/design_intent_register.md` v1.2 adds the two DIR-7 addenda and a Parked Design Items section — see that changelog for detail, not repeated here.
2.23	11 Jul 2026	Doc restructure. This document is now the single source of truth for current state (Quick Status Board gains PARKED and NEXT subsections, folding in the list previously buried in CLAUDE.md's SESSION LOG and §12's Next Priorities). New Changelog section (below) replaces CLAUDE.md's SESSION LOG — same entries, same append-only format, moved here. The embedded Design Intent Register (formerly here, ~200 lines) is removed — full DIR now lives only in `docs/design_intent_register.md`; this document keeps a short pointer + summary of which DIRs govern current work. All docs moved into the repo under `docs/` (git-tracked, no version number in filenames — see CLAUDE.md Documentation Discipline). Fixed: duplicate `## 15d` heading (second instance renamed `15e`); §15's "Recording control" locked-decision row corrected from "DIR-9 — always-on" to DIR-9a (ARM-bounded); duplicate 8-variable data-lake table in §9 replaced with a pointer to §0's canonical 19-variable table; stale "Widget JSON not in Gitea" row in §16 marked resolved (W6 committed per `785ec56`, already reflected in §12). No new facts asserted beyond what was already recorded elsewhere in this document, CLAUDE.md, or the DIR — this pass is consolidation, not new findings.
2.22	9 Jul 2026	⚠ ROOT-CAUSE CORRECTION (supersedes v2.10 and v2.19 claims). The BlueOS "FD exhaustion" bug was NEVER fixed. `nofile=65536` was a ceiling raise, not a fix — it moved failure from ~10 min to ~11 h of cumulative exposure. Real cause, measured: `ardupilot_manager`'s Linux board detector opens `/dev/i2c-1` on each failed probe and never closes it. The BlueOS Autopilot Firmware / board-config page polls `GET /v1.0/available_boards`; each request fires 20 `detect_boards` probes (Navigator/NavigatorPi4/NavigatorPi5/Argonot). We run a Pixhawk on USB serial, so every probe fails and leaks. Measured ~100–108 FD/min with the page open; exactly zero with it closed (4630 flat over 65 min; +90 FD in 50 s on opening; flat immediately on closing). FD composition: 2740/2745 were `/dev/i2c-1`, 3 sockets. Probe:request ratio 20:1 (8920:445, from logs). Count accumulates across sessions; resets only when `blueos-core` is recreated. Upstream BlueOS 1.4.3 defect — do NOT patch BlueOS (updates revert). Dev mitigation: don't leave the board-config page open. Field mitigation NOT built. §15a rewritten. New §15d — OPEN, SAFETY-CRITICAL: `failsafe_monitor` reported GREEN while the FC was unreachable; W1 is the AUV pre-dive GO/NO-GO gate (`health_role: gate`) and would pass a dive with a dead flight controller. All widget indicators reflect the RPi5 stack, not the vehicle. Also this session: `rov_api` `/mode` endpoints (GET/POST, validate-before-write + `/rov/mode/reload` trigger) built, deployed, tested incl. rejection path (`9109439`). W8 mode indicator built and validated live — render + data (`87000e6`). Chronic "widgets show no data" root-caused: `cockpit_bridge` was in the launch file but had no respawn; a boot-time race left it permanently dead, taking the whole data-lake feed with it — fixed with `respawn=True` (`c099c00`), verified across a cold stack restart. Respawn extended to `failsafe_monitor`, `motion_controller`, `ekf_filter_node`, `depth_node`, `mission_executor` (`a5e1c99`); deliberately NOT applied to `mavros_node` (would mask FCU-link instability), `recording_manager` (pending DIR-9a rework), perception (not operational), `foxglove_bridge` (viz-only). FLAGGED OPEN: `recording_manager.py` still implements superseded DIR-9 (continuous-from-boot) rather than governing DIR-9a (ARM-bounded); and `argonaut-stack.sh` passes `record:=${RECORD}` to a launch file that declares no such arg, so `RECORD=false` is silently ignored. Claude Code (VS Code) adopted for repo edits; SESSION LOG added to `CLAUDE.md` to bridge Claude Code ↔ claude.ai.
2.21	7 Jul 2026	Operating-mode profile system built, verified end-to-end, committed (`4e48dc2`) and pushed. New `ModeProfile.msg` (rov_interfaces) — atomic resolved flag set. New `mode_profile_loader` node + `config/mode_profiles.yaml` (rov_mission): ROV/AUV default profiles plus custom hybrid via Save As; resolved flags published latched on `/rov/mode/profile`; reloadable via `/rov/mode/reload` (std_srvs/Trigger); fail-safe validation rejects bad flag values (caught a YAML 1.1 `on`→bool trap; fixed by quoting + a normaliser). `cockpit_bridge` now surfaces 11 mode flags to the data lake — *19 `external/rov-` variables total (was 8). Widgets read flags, not modes; hybrid = any saved profile with overrides. Verified: ROV and AUV both resolve + live reload. DIR reworked for consistency: DIR-9 collapsed to a stub; DIR-9a governing — single ARM-bounded black-box recording (AUV: ARM starts / DISARM stops; ROV: operator start/stop + ARM reminder; purpose black-box/debug/ML/fault-finding only; certified deliverable capture out of scope pending Class-approved devices). DIR-1 corrected (tether/HILT, not power source, determines mode; tethered may use battery or topside supply). DIR-7 gains AUV-mode health semantics (GREEN/AMBER/RED = within/approaching/terminate, parameter-driven). DIR-4/5/10 recording contradictions removed (always-on continuous "core" dropped; base set now recorded within the armed run). Backup:** overdue alert resolved (07-05 was a one-night hostname-resolution transient, not a config regression); segment `_9` size-mismatch re-synced (captured mid-rotation); RPi5 backup script hardened (`--exclude='*.mcap'` so in-progress bags are not captured mid-write; active-dive prune guard).
2.20	July 2026	Phase 3 Stage 2 core complete: `rov_api` FastAPI backend deployed on RPi5:8081 (commit `66a18ef`) — HTTP-to-ROS2 bridge, uvicorn main thread + embedded rclpy node in daemon thread. Endpoints verified end-to-end: GET /health (mission/recording/failsafe), POST /abort (W3 RETURN TO SAFE → /rov/mission/abort → failsafe_monitor FSM NORMAL→HOLD_AND_RECOVER), POST /mission/start (MissionCommand START), POST /mission/stop (MissionCommand ABORT). Runs as argonaut-api.service (manual-start, same policy as argonaut.service). fastapi/uvicorn installed system-wide (--ignore-installed typing_extensions). Deploy files version-controlled in repo deploy/ folder (commit `97decac`): both services' units + wrappers + README documenting locations, pip deps, install steps. Hostname resolution (DNS) fixed on laptop (Windows hosts) and RPi5 (/etc/hosts) — symbytech/blueos/rov-brain resolve; git remotes returned to hostname form. Widget housekeeping (commit `785ec56`): W6 committed as w6_altitude.json (underscore convention), probe widget removed from Gitea, 7 local Copy/working cruft files deleted, W0–W5 line-ending-only churn discarded. All three (Gitea/laptop/RPi5) level at `785ec56`. Stage 2b deferred (return budget, /backup/*, /mission/upload) — return budget approach pending "shortest path to safety" discussion; W7 backup design chosen (Option A: server read-only JSON, RPi5 polls outbound). UFW deferred to pre-field hardening pass (no online exposure).
2.19	July 2026	Phase 3 Stage 1 complete: `recording_manager` deployed — continuous rosbag2 MCAP recorder, zstd compression, 500MB bag splitting, 10GB retention policy, 1Hz health monitoring, hard no-go gate on `/rov/recording/active`. Pixhawk 2.4.8 confirmed active FC (SITL removed from BlueOS); BlueOS FD exhaustion claimed fixed (`/etc/docker/daemon.json`, nofile=65536) — ✗ this claim was WRONG, retracted in v2.22, see §15a. MAVLinkRouter confirmed; MAVP2P not recommended (BlueOS own warning). W6 Altitude Widget built and deployed; cockpit_bridge updated to 8 variables (`rov-recording`, `rov-altitude` added). W3 button label decided: "RETURN TO SAFE" (per DIR-5 — assessed vehicle return-to-safe, not emergency hard-stop). Tailscale stopped/disabled on RPi5. Gitea remote updated to IP-based URL. ardupilot_manager_monitor.sh deleted from RPi5. Probe widget deleted (dev tool, not part of W0–W7).
2.18	July 2026	Simulation & ML tooling scoped into new companion doc `Argonaut3_Simulation_ML_Tooling_Plan_v1_0.md`. Two tracks defined: Track 1 (Gazebo Harmonic + ArduSub SITL, CPU-led, runs on existing laptop) for nav/mission/failsafe bench; Track 2 (HoloOcean/UNav-Sim/Isaac, RTX GPU) for perception synthetic-data + model training; convergence = structure-relative visual localization (the real bottleneck). Dev-PC tiers costed (min $1.2–1.6k / rec $2.0–2.8k / high $3.5–6k+); cloud alternative costed (free: Kaggle/Colab/Lightning; paid RTX 4090 ~$0.33/hr Vast/RunPod). Training-asset sources listed (Fab/Megascans free; SubPipe/COU datasets; YOLO/SAM2 base models) — NO public jacket dataset exists, CAD+synthetic is the path. RECORD CORRECTED: past SITL failures were under-resourced hosts (server 2-vCPU VM; RPi4 + half-connected Pixhawk) — not intrinsic to SITL. R210 server cannot host GPU sim (1U). PCIe Gen 3 on RPi5 NVMe: option logged, DEFERRED to Phase 3 camera-to-MCAP.
2.17	June 2026	W0–W5 all committed to Gitea (W0 layout fix `2cb7b74`). Stale server clone `~/rov-autonomy` deleted. MAVROS recovery: confirmed data point — RPi4 BlueOS core restart ALONE recovered the system this session (RPi5 untouched). DESIGN INTENT REGISTER added as Section 1 — dual-capability AUV/ROV platform, HILT as override channel, mission lifecycle, abort/return-to-safe model (mission-bounded), Emergency Surface shelved. Re-scopes W3/W4 backend: near-term need is tethered HILT path. FastAPI backend NOT started — intent locked first.
2.16	June 2026	W1–W5 all confirmed working in Cockpit v1.18.2 with live data. W1 failsafe variable name corrected (`rov-failsafe` full word, not truncated `rov-failsa`). Outstanding widget work is backend-side. Flagged items consolidated.
2.15	June 2026	Cockpit v1.18.2 DIY widget data lake getter CONFIRMED via probe widget. `getDataLakeVariableData(id)` returns live values. W1–W5 widget blocker resolved. Full `window.cockpit` surface documented.
2.14	June 2026	COCKPIT DATA BRIDGE BUILT & WORKING (the P2 keystone). New `cockpit_bridge` ROS2 node serves a WebSocket on :9001; all 6 vars land in data lake as `external/rov-*` (commit `12b5161`). RPi4 SD replaced + backup system (pi4-backup.sh) in place & verified. Logging flood (NaN TF) mitigated. W0 working (scroll fixed).
2.13	June 2026	RPi5 storage RE-ARCHITECTED to OS-on-SD + data-on-NVMe. OS migrated SD via rsync; NVMe wiped to single ext4 data volume at /data; workspace moved to /data/ros2_ws. Boot fixed by switching root mount from LABEL to PARTUUID. Full stack now runs as systemd service argonaut.service.
2.12	June 2026	Full stack now launches end-to-end with ZERO crashes (commit `c94412d`). Fixed: MAVROS fcu_url, plugin_allowlist, timestamped MCAP bag dir, BEST_EFFORT QoS, battery<0=UNKNOWN guard.
2.11	June 2026	Full autonomy stack deployed and launched end-to-end on RPi5. THREE repo bugs found and fixed. foxglove_bridge :8765 + MCAP recording to /data/bags confirmed working.
2.10	June 2026	MAVROS link RPi5->RPi4 verified. BlueOS "too many open files" re-diagnosed as uvicorn FD exhaustion. ✗ WRONG — retracted in v2.22, see §15a.
2.9	June 2026	RPi4 fully reinstalled. BlueOS 1.4.3 on fresh SD. Static IP 192.168.1.100. ArduSub SITL 4.5.7 STABLE running. Both Pis operational.
2.8	June 2026	RPi5 base stack complete. Ubuntu 24.04.4 installed. Hostname rov-brain, static IP 192.168.1.101, Tailscale. ROS2 Jazzy, Docker, MAVROS installed.
2.7	May 2026	Cockpit version corrected to v1.18.0. Generic WebSocket confirmed as correct dev injection path. Data lake key format confirmed with external/ prefix.
2.6	May 2026	All 6 widgets built and confirmed working in Cockpit native desktop app. Widget files committed to Gitea.
2.5	May 2026	Widget approach confirmed (DIY JSON), Data Lake Inspector built.
2.4	May 2026	Architecture overhaul — three-prong system, Foxglove, server separation.
2.3	May 2026	Clean dev environment — BlueOS VM + internal SITL + Cockpit working.
2.2	May 2026	Dev environment attempt — standalone Docker SITL stack (superseded).
2.1	May 2026	Server dev environment plan, shopping list update.
2.0	May 2026	Workspace, build pipeline, failsafe design, UI design.
1.0	April 2026	Initial handover document.

Changelog

Moved here from CLAUDE.md's SESSION LOG (11 Jul 2026 restructure). This is the granular, session-by-session record that bridges Claude Code (VS Code) sessions and the claude.ai chat — finer-grained than the Version History table above, which narrates per-doc-version. Append-only; correct forward with a new dated entry, never edit past ones.

Claude Code appends an entry here at the end of any session where one of these happened — not for routine edits, refactors, or in-progress work: a design/architecture decision was made or changed; a hardware fact was confirmed or changed; a security-relevant change was made; a backup/recovery-relevant change was made; a bug was root-caused and fixed (not just patched); a previously-deferred item was completed or its trigger condition changed; anything Grant would reasonably expect this chat to already know next time he opens it. Skip typo fixes, comment-only changes, WIP work, or anything already fully captured by the git commit message with no extra context needed. Full trigger-rule text lives in CLAUDE.md.

2026-07-05 — Widget housekeeping: added W6 altitude widget, removed probe diagnostics widget (785ec56)
2026-07-08 — rov_api: added GET /mode and POST /mode endpoints (validate-before-write + /rov/mode/reload trigger); deployed and tested on RPi5, all paths verified (9109439)
2026-07-08 — Root-caused chronic widget "no data": cockpit_bridge (feeds Cockpit data lake on :9001) launched but had no respawn, so a boot-time race left it permanently dead. Added respawn=True to its launch Node. W8 mode indicator built + validated live (render+data). W1/W8 confirmed GREEN/AUV once bridge up.
2026-07-08 — FLAGGED (open): recording_manager.py still implements superseded DIR-9 (continuous-from-boot recording), NOT the governing DIR-9a (ARM-bounded black-box: AUV ARM-start/DISARM-stop, ROV operator manual + arm reminder). Design decision exists only in handover + CLAUDE.md; code never updated. foxglove_mcap.launch.py comments also still say DIR-9. Also: argonaut-stack.sh passes record:=${RECORD} but rov_full.launch.py declares no such arg, so RECORD=false is silently ignored and bags record regardless. Both deferred to a dedicated DIR-9a recorder-rework session — do NOT wire RECORD into the current DIR-9 model.
2026-07-08 — Added respawn=True/respawn_delay=5.0 to failsafe_monitor, motion_controller, ekf_filter_node, depth_node, mission_executor. Deliberately NOT applied to: mavros_node (respawn could mask/worsen the chronic FCU-link instability — belongs in that investigation), recording_manager (pending DIR-9a rework), camera_node/feature_detector (perception not operational yet), foxglove_bridge (viz-only). NOTE: failsafe_monitor respawn has a coverage gap during the 5s respawn delay — no failsafe assessment in that window; flagged for future design consideration.
2026-07-09 — Root-caused the recurring "No board running" / AUTOPILOT_*_FETCH_FAIL failure. NOT the previously-documented TCP-socket/UI-polling cause. Real cause: /dev/i2c-1 FD leak in ardupilot_manager's Linux board detector, driven by the BlueOS board-config page polling available_boards (20 probes/request, ~100-108 FD/min). Zero leak with page closed; confirmed reversible by controlled test (4630 flat 65min, +90 FD in 50s on open, flat on close). nofile=65536 only delayed failure to ~11h cumulative page-open time. Upstream BlueOS 1.4.3 bug — not patching. Corrected the stale root cause in this file.
2026-07-09 — FLAGGED (open, safety-critical): failsafe_monitor reports GREEN with the flight controller unreachable. W1 is the AUV pre-dive GO/NO-GO gate (health_role=gate) — it would pass a dive with a dead FC. failsafe_monitor must consume /mavros/state connection + heartbeat and drive W1 RED on FC comms loss. W1 status line should carry cause text (e.g. "Vehicle Disconnected", "Heartbeat Lost", "No Board Running"). Also flagged: no FD-count check on ardupilot_manager in the pre-dive gate.
2026-07-11 — RESOLVED the 2026-07-09 flagged safety-critical fault: failsafe_monitor now drives W1 RED on FC comms loss. Four commits: (1) 86b2728 — comms loss forces unconditional RED (was scored as one marginal parameter, only reached AMBER) plus fcu_ever_connected latch so the startup window before any heartbeat is not reported GREEN; cause text "Heartbeat Lost"/"Vehicle Disconnected" added to FailsafeStatus.message per DIR-7. (2) 1b0050d — FSM recovery fix: RETURN_TO_SAFE was a terminal trap (only HOLD_AND_RECOVER could return to NORMAL); now both recover when conditions clear. (3) dbe9515 — cockpit_bridge publishes new data lake variable rov-failsafe-cause (substring of message before first "|"; empty when nominal). (4) 3409ae5 — W1 shows state word on #w1-message (state-driven, no flicker) and cause text on #w1-footer, footer restyled red 11px, blank when nominal. All four verified live end-to-end: kill mavros -> RED + "Heartbeat Lost", restart -> recover to GREEN. Also corrected: cockpit_bridge was publishing failsafe_state (FSM, can be 3+) not assessment_state (0/1/2) to rov-failsafe — fixed in dbe9515's sibling commit d854641; this was the real cause of the day's "widgets show no data" (value 3 out of W1's range), compounded by a restart-storm wedging 9001.
2026-07-11 — Doc restructure (this pass): consolidated current-state into this document, moved this log here from CLAUDE.md, removed the embedded DIR copy in favour of a pointer to docs/design_intent_register.md, moved all project docs into the repo docs/ folder. See Version History v2.23 above for detail.

Design Intent Register — pointer

The full Design Intent Register (DIR-1 through DIR-12) lives in docs/design_intent_register.md, git-tracked in the repo, not duplicated here. That file is the source of truth for why — read it before any design/architecture decision.

DIRs governing current work, for quick orientation:

DIR-1/DIR-2 — tether/HILT link (not power source) determines AUV vs ROV mode; HILT is an override channel, not a confirmation gate.
DIR-7/DIR-12 — health has two distinct subjects (vehicle vs autonomy stack); AUV health_role: gate vs ROV advisory; health assessment must include FC connectivity and must carry cause text. (Implemented 11 Jul 2026 — see Changelog above.)
DIR-9a (governing, supersedes DIR-9) — recording is a single ARM-bounded black-box tier, keyed to the recording_control flag, not the mode name. recording_manager.py still implements the superseded DIR-9 model — known drift, open.
DIR-5/5a/5b — return-to-safe is mission-bounded, vehicle-layer, and mode-dependent (AUV automatic, ROV alert+override); operator override continues the mission, it does not pause it.

1. Project Overview

Goal

A fully autonomous underwater inspection system targeting hull and jacket surveys with no human operator during field operations. Built on a shoestring budget using commodity hardware, designed for field deployment from day one.

Target Structures

Jackets: Fixed offshore structures — predictable geometry
Hulls: In-port vessels — moored, requires online re-localisation

Key Design Values

Cost-conscious — use what is available, upgrade when a dead end is reached
Field-operational from day one — same codebase, different configs per environment
Full autonomy — no pilot required during operation
Cockpit as access hub — Argonaut 3 widgets only, not a container for everything
QGroundControl retained as dev/debug only

2. Platform

Vehicle: BlueROV2 Heavy (Blue Robotics) Operating mode: Vehicle-side compute only Tether: Single Ethernet cable — all topside/bottomside communication over one cable

3. Hardware Architecture

On the ROV

Board	Role	OS
Raspberry Pi 4 (8GB)	BlueOS gateway — MAVLink routing, Pixhawk FC host	BlueOS 1.4.3 (armv7)
Raspberry Pi 5 (8GB)	Autonomy brain — ROS2 stack, foxglove_bridge, MCAP recorder	Ubuntu 24.04 LTS + ROS2 Jazzy

RPi4: BlueOS 1.4.3 running. Static IP 192.168.1.100. Tailscale 100.100.119.125 (node renamed blueos-1). Pixhawk 2.4.8 active on /dev/ttyACM0 — SITL is NOT in production use. BlueOS preferred_board: Pixhawk1. Docker nofile ulimit = 65536 (ceiling raise only — NOT a fix; see §15a, ardupilot-manager /dev/i2c-1 FD leak is an OPEN bug). Pirate Mode enabled.

RPi5: Ubuntu 24.04.4, boots from SD card (/dev/mmcblk0). NVMe (/data) = data only (ros2_ws, bags, logs — NOT the OS). Static IP 192.168.1.101. Tailscale installed but STOPPED and DISABLED (re-enable: sudo systemctl enable --now tailscaled). ROS2 Jazzy, Docker, MAVROS installed. argonaut.service running (manual-start in dev). Weak SSH password — harden before field deployment.

Sensors

Sensor	Status
Xsens MTI-30-2A5G4 IMU	Have unit — CA-USB-MTI cable on order
IP cameras (2+)	Have units — not yet integrated
Ping2 Sonar (downward)	Phase 3 purchase (~$280 USD) — required for W6 full test
Ping2 Sonar (forward)	Phase 4 purchase
Ping360 Scanning Sonar	Phase 4-5 purchase
DVL (Doppler Velocity Log)	Phase 7 — critical for accurate X/Y positioning
Tritech Gemini 720im	Phase 6+

DVL is the most important missing sensor. Without it, X/Y position is dead-reckoned from IMU only and drifts over time. Z axis (Bar30) is accurate. All position displays must reflect this uncertainty honestly.

Power

Device	PSU	Status
RPi4	5.1V / 3A RPi4 official PSU	In use
RPi5	5V / 5A 27W USB-C official RPi5 PSU	In use

Storage

Device	Storage	Notes
RPi4	Fresh microSD (Patriot 64GB or equivalent)	BlueOS OS drive
RPi5 (OS)	SD card (`/dev/mmcblk0`)	Ubuntu 24.04, hardened (log2ram, tmpfs, journald capped)
RPi5 (Data)	RPi M.2 HAT+ + RPi SSD 512GB (2230 NVMe) at `/data`	ros2_ws, bags, logs

4. Software Stack

RPi4 (BlueOS Gateway)

BlueOS 1.4.3 (armv7) — running, static IP 192.168.1.100
Tailscale 100.100.119.125 (blueos-1)
Pixhawk 2.4.8 — active flight controller on /dev/ttyACM0:115200
MAVLinkRouter — routes Pixhawk to MAVROS on RPi5 and to Edge PC for QGC debug. MAVP2P has known issues (BlueOS own warning) — do not use.
/etc/docker/daemon.json — nofile: 65536 (ceiling raise only, NOT a fix — see §15a)
Pirate Mode — enabled

RPi5 (Autonomy Brain)

Ubuntu Server 24.04.4 LTS (Noble, arm64) — boots from SD, NVMe = /data
ROS2 Jazzy Jalisco — auto-sourced in .bashrc
MAVROS + GeographicLib datasets
Docker 29.5.3 — runs without sudo
Argonaut 3 autonomy stack (7 packages) — deployed, running as argonaut.service
foxglove_bridge — confirmed running on :8765
recording_manager — confirmed active, bags to /data/bags/
cockpit_bridge — WebSocket :9001, 8 variables
Tailscale — installed, service stopped and disabled
Xsens MTi ROS2 driver — pending CA-USB-MTI cable

Edge PC / Field Laptop BB8 (Topside)

Cockpit native desktop app v1.18.2 — primary GCS, Argonaut 3 widgets
Foxglove Studio — live monitoring (Prong 2) and post-dive replay (Prong 3)
Mission Planner (Phase 6+) — standalone 3D web app, locally installed
Local model + mission cache — glTF twins and YAMLs pre-synced, offline capable

Cockpit native app is required. The BlueOS browser extension version is "Lite" and does not support DIY widgets.

5. System Architecture — Three Zones

ONSHORE                    OFFSHORE TOPSIDE              OFFSHORE BOTTOMSIDE
------------------         ----------------------        -----------------------------
Dev Server                 Edge PC / Field Laptop        BlueROV2 Heavy
  Harbor registry            Cockpit native v1.18.2        RPi4 (BlueOS)
  Gitea source control       Foxglove Studio                 MAVLink Router
  ARM64 build pipeline*      Mission Planner*                Pixhawk 2.4.8 (ArduSub)
                             Local cache                 RPi5 (ROS2 Jazzy)
Ops Server (separate)                                      Autonomy stack (7 packages)
  Digital twin storage     <-- Ethernet Tether -->         foxglove_bridge :8765
  Mission archive                                           recording_manager
  Post-dive data                                            cockpit_bridge :9001
  Model conversion                                        Sensors
  Tailscale endpoint                                        Xsens, Bar30, cameras, sonar

*ARM64 builds: laptop only (server overheats under compilation load) *Mission Planner: Phase 6+ standalone app

Ethernet Tether — Single Cable

Flow	Direction
BlueOS web UI :80	Tether to Edge PC browser
MAVLink UDP :14550	Tether to QGC on Edge PC (debug)
foxglove WebSocket :8765	Tether to Foxglove Studio on Edge PC
cockpit_bridge WebSocket :9001	Tether to Cockpit on Edge PC
Mission YAML upload	Edge PC to RPi5 via SSH/SCP
MCAP sync post-dive	RPi5 to Edge PC via SSH/SCP
Container deploy	Edge PC to RPi5 via SSH

6. Three-Prong Mission System

Prong 1 — Mission Creation + Editing

glTF model synced to Edge PC local cache pre-dive
Mission Planner on Edge PC: Three.js 3D scene with structure twin
Hull mode: parametric lawnmower path generator
Jacket mode: 3D waypoint editor
Output: Mission YAML (rich format, structure-relative 3D coordinates)
Status: Phase 6+ — standalone web app, not in Cockpit

Prong 2 — Mission Execution + Monitoring

rov_mission on RPi5 executes YAML via state machine
MAVROS translates ROS2 setpoints to MAVLink to RPi4 to ArduSub to thrusters
All ROS2 topics streamed via foxglove_bridge WebSocket to Foxglove Studio
Cockpit widgets on Edge PC: Health / Mission Status / Return To Safe
Action path: Cockpit widget fetch() -> FastAPI on RPi5 -> /rov/mission/abort

Prong 3 — Data Collection + Replay

MCAP recorder logs ALL ROS2 topics to NVMe SSD during dive
Post-dive: MCAP transferred to Edge PC via SCP over tether
Foxglove Studio on Edge PC: immediate replay, no server needed
When server reachable: MCAP archived to Ops Server via Tailscale

7. Digital Twin

Format	Origin	Status
EBM / EBMX	CADMATIC — hull model	Have file
NWD	Autodesk Navisworks — platform models	Have files (x2 platforms)
IFC	Open BIM standard — intermediate	Conversion target
glTF / GLB	Khronos Group — web render target	Final format for all tools

Conversion pipeline: EBM / NWD -> CAD Exchanger SDK -> IFC -> glTF / GLB Currently manual. Automated pipeline is a later infrastructure task.

8. ROS2 Workspace

Location on RPi5: /data/ros2_ws/src/rov-autonomy/ Gitea repo: administrator/rov-autonomy Gitea SSH remote: ssh://git@symbytech:222/administrator/rov-autonomy.git (hostname form — resolves via hosts entries on laptop and RPi5)

Package Structure

Package	Purpose	Key nodes
`rov_interfaces`	Custom messages and services	— (msgs/srvs only)
`rov_navigation`	State estimation, EKF, depth	`state_estimator`, `depth_node`
`rov_perception`	Camera nodes, feature detection	`camera_node`, `feature_detector`
`rov_control`	Motion controller, failsafe monitor	`motion_controller`, `failsafe_monitor`
`rov_mission`	Mission executor, cockpit bridge, recorder	`mission_executor`, `cockpit_bridge`, `recording_manager`
`rov_bringup`	Top-level launch files	— (launch files only)
`rov_api`	FastAPI backend — HTTP→ROS2 bridge on :8081	`api_node`
`rov_simulation`	Simulation nodes (dev only)	Do NOT deploy in production launch files

Key Topics

Topic	Type	Publisher
`/imu/data`	sensor_msgs/Imu	Xsens driver
`/mavros/imu/data`	sensor_msgs/Imu	MAVROS
`/odometry/filtered`	nav_msgs/Odometry	EKF node
`/rov/failsafe`	rov_interfaces/FailsafeStatus	failsafe_monitor
`/rov/mission/status`	rov_interfaces/MissionStatus	mission_executor
`/rov/recording/active`	std_msgs/Bool	recording_manager
`/rov/recording/bag_dir`	std_msgs/String	recording_manager
`/ping2/altitude`	sensor_msgs/Range	ping2 driver (hardware-gated)

Key Commits

Hash	Description
`0640353`	W6: Add rov-altitude to cockpit_bridge for Ping2 downward sonar
`5d75a9d`	Phase 3 Stage 1: Recording Manager (DIR-9)
`2cb7b74`	W0 layout fix (3-column, horizontal scroll)
`66f9bd7`	W0–W5 all confirmed working, committed
`12b5161`	cockpit_bridge node — ROS2→WebSocket :9001 data-lake injection
`c94412d`	Full stack zero-crash launch
`45b3301`	Fix full-stack launch: fcu_url, plugin_allowlist, QoS, battery guard
`7590faf`	Missing setup.cfg in all 5 Python packages
`f1c6710`	Missing config/ dir in rov_bringup

Build/Deploy Procedure

# On RPi5 — build (symlink-install works correctly with setup.cfg files in place)
cd /data/ros2_ws
colcon build --packages-select <pkg> --symlink-install
source /data/ros2_ws/install/setup.bash

9. UI Architecture — Argonaut 3 / Cockpit

Cockpit Setup

App: Cockpit native desktop v1.18.2 — installed on field laptop BB8 (Windows) Vehicle connection: Enter RPi4 IP (192.168.1.100) on first launch Generic WebSocket: ws://192.168.1.101:9001 (cockpit_bridge on RPi5)

Cockpit native desktop app is mandatory for DIY widget work. The BlueOS browser extension version is "Lite" and does NOT support DIY widgets. Generic WebSocket data injection requires v1.18.0 minimum.

Widgets are Cockpit DIY widgets: plain HTML/CSS/JS stored as JSON files and imported via the gear icon on a placed DIY widget.

Import format: JSON file with three string fields:

{
  "html": "...",
  "css":  "...",
  "js":   "...",
  "inheritCockpitStyles": false
}

CSS rules for DIY widgets (hard-won — do not break these):

No body {} or :root {} blocks — bleed into Cockpit's own styles and crash the renderer
No @keyframes animations — cause renderer instability
No unicode characters in JS strings — use plain ASCII equivalents
No position: fixed overlays — use display: block/none toggling instead
No async/await — use .then()/.catch() for fetch calls
Use element ID/class selectors only — no global tag selectors (e.g. div {})

Data Injection Path — Confirmed Production Path

cockpit_bridge (ROS2 node on RPi5, rov_mission pkg)
  WebSocket server :9001
    <-- Cockpit connects as client
      --> Sends 'variable-name=value' messages at 2Hz
        --> Data lake populated as external/{variable-name}
          --> DIY widget reads via getDataLakeVariableData('external/{variable-name}')

Confirmed data lake read method: window.cockpit.getDataLakeVariableData(variableId) Does NOT exist: getDataLakeValue(), listenToDatalakeVariable() (not in DIY widget scope) Metadata only: getAllDataLakeVariablesInfo() — enumeration only, not value reads

Data Lake Variables

Full current list (19 total: 8 telemetry + 11 mode-profile flags) is in §0 QUICK STATUS BOARD — not duplicated here to avoid the two tables drifting apart, which is exactly what happened between v2.21 (8 vars, this table) and v2.22 (19 vars, §0's table) before this restructure.

Gitea repo: rov-autonomy/widgets/ Laptop local: C:\Users\grant\Desktop\Autonomous ROV System\rov-autonomy\widgets\

#	File	Widget	Test Status	Variable
0	`w0_data_lake_inspector.json`	Data Lake Inspector	Confirmed working (706 vars live)	None — reads all
1	`w1_system_health_indicator.json`	System Health Indicator	Visual confirmed — full test pending	`external/rov-failsafe`
2	`w2_mission_status.json`	Mission Status	Visual confirmed — untested (needs mission running)	`external/rov-ms`, `external/rov-mp`
3	`w3_abort_button.json`	Return To Safe Button	UI confirmed — label rename pending ("ABORT MISSION" → "RETURN TO SAFE"); backend not built	FastAPI `/abort` POST
4	`w4_mission_setup_button.json`	Mission Setup Button	Visual confirmed — untested (needs FastAPI backend)	None — opens URL
5	`w5_battery_return_budget.json`	Return Budget	Visual confirmed — untested (`rov-return-budget` not published)	`external/rov-return-budget`
6	`w6_altitude.json`	Altitude	NO SONAR confirmed — hardware-gated for full test	`external/rov-altitude`
7	`w7_backup_due_notice.json`	Backup Due Notice	Not built — depends on FastAPI backend	FastAPI `/backup/status`

Testing note: W0 is the only widget with confirmed live data end-to-end. W1–W5 display correctly but have not been tested with actual operational data. Full testing requires a live mission, FastAPI backend, and/or Ping2 hardware depending on the widget. W6 full test requires Ping2 sonar hardware. W7 not started.

Open Cockpit native desktop app
Connect to vehicle (192.168.1.100)
Enable Pirate Mode (Settings → General → Enable Pirate Mode) — resets each session
Enter edit mode (pencil icon, top right)
Click Add Widget at the bottom — scroll right to find the </> DIY widget
Drag it onto the main view area and resize
Click the gear icon on the placed widget
Click Import — select the .json file from the widgets folder
Widget loads immediately

10. Failsafe Design Summary

Full design in docs/failsafe_design.md.

Three States

State	Meaning
GREEN	All systems nominal
AMBER	One or more parameters degraded — mission continues with reduced confidence
RED	Critical failure — safe action triggered automatically

Trigger Conditions

Trigger	Threshold	State
Comms loss	>2s (configurable)	RED → SURFACE
Battery low	<15% above return budget	AMBER
Battery critical	<5% above return budget	RED → SURFACE
Depth exceeded	Configurable per site	RED → SURFACE
IMU fault	Data gap >500ms	AMBER
DVL fault (Phase 7)	Data gap >1s	AMBER

11. Infrastructure — SymbyTech Server

Item	Value
Hardware	Dell PowerEdge R210
OS	Ubuntu Server 24.04 LTS
Static IP	192.168.1.175 (on eno2 — NOT eno1)
RAID Mount	/mnt/raid-data/
Tailscale IP	100.104.236.104
CPU cores	4 (host)

WARNING: eno2 is the primary NIC, not eno1. Never change this in netplan.

Gitea

Item	Value
Public URL	https://git.symbytech.com
SSH remote	`ssh://git@192.168.1.175:222/administrator/REPO.git`
ROV repo	`administrator/rov-autonomy`
Data location	/mnt/raid-data/forgejo/

Use the server IP directly (192.168.1.175:222) — the hostname symbytech may not resolve on all networks. Do NOT use git.symbytech.com for SSH port 222.

Harbor Registry

Item	Value
Public URL	https://registry.symbytech.com
Local URL	http://192.168.1.175:8090
Storage	/mnt/raid-data/harbor/

Backup System (built 30 Jun 2026)

Push model — each Pi self-backs-up via root cron, pushing to server over SSH.

Device	Script	Schedule	Log
RPi4	`/usr/local/bin/rov-brain-pi4-backup.sh`	00:00 daily	`/var/log/rov-brain-pi4-backup.log`
RPi5	`/usr/local/bin/rov-brain-pi5-backup.sh`	00:30 daily (staggered)	`/var/log/rov-brain-pi5-backup.log`

Server monitor (/usr/local/bin/symbytech-monitor.sh → check_rov_backup) reads */backup-status.txt under /mnt/raid-data/backup/. Alerts: FAILED (always), OVERDUE ≥2 days (throttled to once/7 days per device, regardless of online/offline).

RPi5 backup images /dev/mmcblk0 (SD card OS) and rsyncs /data (NVMe data). RPi5 boots from SD — /dev/mmcblk0 is the OS drive. NVMe is data only.

Development VM — DECOMMISSIONED

The BlueOS KVM VM (blueos-vm, NAT IP 192.168.122.89, Tailscale 100.84.141.120) was used during early development before real RPi4/RPi5 hardware was operational. It is no longer active (virsh list --all returns empty). The NPM proxy rule arov.symbytech.com → 192.168.122.89:80 is a leftover dead rule with no active target. All [VM] commands in this document are historical only — do not use.

12. Phase Task Status

Phase 1 — Foundation

Task	Status
Flash BlueOS onto RPi4	DONE — BlueOS 1.4.3, static IP 192.168.1.100
Install Ubuntu 24.04 on RPi5	DONE — Ubuntu 24.04.4, hostname rov-brain, IP 192.168.1.101
Install ROS2 Jazzy on RPi5	DONE — Jazzy Jalisco, auto-sourced in .bashrc
Confirm network comms between Pis	DONE — MAVROS `/mavros/state connected:true` against Pixhawk
Install Docker on RPi5	DONE — Docker 29.5.3, no sudo required
Install MAVROS	DONE — running in argonaut.service
Install Tailscale on RPi5	DONE — installed; service stopped and disabled
Install Tailscale on RPi4	DONE — Tailscale IP 100.100.119.125
Harbor project and user setup	DONE
Connect Xsens IMU	PENDING — CA-USB-MTI cable on order
foxglove_bridge install on RPi5	DONE — ros-jazzy-foxglove-bridge, :8765 confirmed
Create /data/bags on NVMe	DONE — recording live ~2.5 MB/s
Deploy ROS2 workspace from Gitea	DONE — all 7 packages built, argonaut.service running
Automated backup system	DONE — both Pis → server, monitoring active
BlueOS FD exhaustion fix	✗ NOT DONE — root cause was wrong. `nofile=65536` is a ceiling raise. Real cause: `/dev/i2c-1` FD leak in ardupilot_manager board detector. OPEN — see §15a
Pixhawk 2.4.8 as active FC	DONE — confirmed ttyACM0, MAVROS connected:true

Phase 2 — Widgets and Data Path

Task	Status
Confirm widget approach	DONE — Cockpit DIY widgets, JSON format
Confirm Cockpit version required	DONE — native desktop v1.18.2
Establish CSS rules for DIY widgets	DONE
W0 Data Lake Inspector	DONE — confirmed working (706 vars live)
W1 System Health Indicator	DONE — visual confirmed
W2 Mission Status	DONE — visual confirmed; full test pending
W3 Return To Safe Button	DONE (UI); label rename pending; backend not built
W4 Mission Setup Button	DONE — visual confirmed; full test pending
W5 Return Budget	DONE — visual confirmed; full test pending
W6 Altitude Widget	DONE — NO SONAR state confirmed (hardware-gated)
W7 Backup Due Notice	NOT STARTED — depends on FastAPI backend
All W0–W6 committed to Gitea	W0–W5: committed. W6 widget JSON: pending re-import from laptop
cockpit_bridge node	DONE — 8 variables, commits `5d75a9d` + `0640353`
Probe widget	DELETED — dev tool only, not part of W0–W7

Phase 3 — Recording and Mission Control (Active)

Task	Status
Recording Manager (DIR-9)	DONE — Stage 1 complete, commit `5d75a9d`
FastAPI backend core (RPi5:8081)	DONE — Stage 2 core, commit `66a18ef`, verified end-to-end
Deploy files version-controlled	DONE — deploy/ folder, commit `97decac`
Hostname/DNS resolution fix	DONE — laptop + RPi5 hosts entries, git remotes on hostname
Widget housekeeping (W6 add, probe remove)	DONE — commit `785ec56`
W3 backend `/abort` POST end-to-end	DONE — verified: FSM NORMAL→HOLD_AND_RECOVER
W3 label update to "RETURN TO SAFE"	PENDING — widget code + FASTAPI_HOST constant
W5 return budget publishing	DEFERRED — Stage 2b (return budget approach TBD)
W7 Backup Due Notice + /backup/*	DEFERRED — Stage 2b (Option A design)
/mission/upload endpoint	DEFERRED — Stage 2b
Ping2 sonar purchase (W6 full test)	PENDING — Phase 3, ~$280 USD

Next Priorities

See §0 QUICK STATUS BOARD — NEXT (moved there 11 Jul 2026 restructure to avoid two lists drifting apart).

13. Session Lessons Learned — Data Injection Investigation (Task 11)

This section documents what was learned during the Task 11 data injection investigation so it is not repeated. These are permanent historical records.

What was attempted and why it failed

Approach	Why it failed
UDP to VM port 14550 from server host	KVM NAT silently drops packets
UDP to VM port 14001 from server	Same KVM NAT issue
HTTP POST to mavlink2rest /mavlink	Stored in REST, not broadcast via WebSocket — not in data lake
UDP with sysid=255, compid=190	mavlink2rest only processes sysid 1 messages for data lake
UDP with sysid=1, compid=1 from Python	Caused ArduSub to detect heartbeat conflict — do not impersonate compid=1

What works

Approach	Result
Generic WebSocket (Cockpit v1.18.0+)	Confirmed working. WebSocket server on RPi5, Cockpit connects as client. Variables appear in data lake as `external/{name}` immediately. This is the correct and supported path.
Native ArduSub NAMED_VALUE messages	Confirmed working via W0 — flow: Pixhawk → ttyACM0 → mavlink-routerd → mavlink2rest UDP → mavlink2rest REST + WebSocket → Cockpit.

Key architectural insight

The ONLY path that gets external data into the Cockpit data lake is:

Messages from the Pixhawk via ttyACM0 → mavlink-routerd → mavlink2rest — for production ArduSub/MAVROS data
Generic WebSocket connections (Cockpit v1.18.0+) — for custom external data via cockpit_bridge

14. Roadmap Summary

Phase	Title	Status
1	Foundation	COMPLETE
2	Widgets and Data Path	COMPLETE (W6 deployed; W7 not started; backend pending)
3	Recording and Mission Control	IN PROGRESS — Stage 1 complete
4	Surface Following and Pattern Flight	Not started
5	Perception and Feature Detection	Not started
6	Full Autonomy + Mission Planner	Not started
7	Field Trials and Hardware Evolution	DVL integration
8	Commercial Product	Not started

15. Key Architecture Decisions (Locked)

Decision	Choice	Rationale
Autonomy compute	Vehicle-side RPi5	Lowest latency to sensors and thrusters
RPi4 role	Thin gateway only	Separation of concerns
OS on RPi5	Ubuntu 24.04 LTS	Only clean ROS2 Jazzy support
RPi5 storage	OS-on-SD, data-on-NVMe	Physical separation; SD hardened for low writes
ROS2 version	Jazzy Jalisco	Current LTS
Container registry	Harbor (self-hosted)	No rate limits, private
Cockpit version	Native desktop v1.18.2	Browser extension is Lite; v1.18.2 required for DIY widgets
Widget approach	Cockpit DIY widgets (JSON format)	Confirmed working, version-controlled in Gitea
MAVLink router	MAVLinkRouter	MAVP2P has known issues per BlueOS own warning
Flight controller	Pixhawk 2.4.8 (real hardware)	SITL retired from production
Data injection	Generic WebSocket via cockpit_bridge	Confirmed working; MAVLink injection paths do not populate data lake
WebSocket variable naming	Use hyphens not underscores	Cockpit normalises underscores to hyphens in data lake IDs
Recording format	MCAP (Foxglove native)	Seekable replay, no conversion needed
Recording control	recording_manager ROS2 node	DIR-9a (governing) — single ARM-bounded black-box tier, keyed to `recording_control` flag, not mission state. `recording_manager.py` implementation still lags on superseded DIR-9 (continuous-from-boot) — known drift, open.
foxglove_bridge role	Visualisation ONLY	Does NOT record MCAP (DIR-9a)
ARM64 builds	Laptop only	Server overheats under compilation load
Acoustic positioning	Excluded	Unreliable near steel
DVL	Deferred to Phase 7	Budget constraint — highest priority future sensor

15a. ardupilot-manager FD Exhaustion — OPEN BUG (root cause corrected 9 Jul 2026)

⚠ This section replaces the previous "RESOLVED — Permanent Fix Applied" entry, which was wrong on both diagnosis and fix. The stale root cause survived from v2.10 to v2.21 and actively prevented this being solved, because it made every recurrence look inexplicable.

What the old entry claimed (WRONG — do not reinstate)

✗ "Root cause: accumulated TCP socket FDs from continuous UI polling exhausting nofile=1024."
✗ "Permanently fixed via /etc/docker/daemon.json nofile: 65536."
✗ (v2.10) "re-diagnosed as uvicorn FD exhaustion."

None of these are correct. The leaked FDs are not sockets. nofile: 65536 fixed nothing — it raised the ceiling, converting a failure at ~10 minutes into a failure at ~11 hours, which disguised a hard leak as an intermittent annoyance.

Actual root cause (measured 9 Jul 2026)

ardupilot_manager leaks /dev/i2c-1 file descriptors, one per failed Linux-board probe.

Mechanism, end to end:

The BlueOS Autopilot Firmware / board-config page polls GET /v1.0/available_boards.
Each request triggers 20 flight_controller_detector.linux.detector:detect_boards probes — for Navigator, NavigatorPi4, NavigatorPi5, and Argonot.
Each Linux-board probe opens /dev/i2c-1 and never closes it when detection fails.
We run a Pixhawk 2.4.8 on USB serial (PlatformType.Serial), not a Navigator. Every probe fails. Every probe leaks.
At 65536 FDs, ardupilot_manager can no longer open new handles. Everything requiring a fresh handle fails: AUTOPILOT_VEHICLE_TYPE_FETCH_FAIL, AUTOPILOT_ENDPOINT_FETCH_FAIL, BAG_OF_HOLDING_GET_DATA_FAIL, "No board running".
Restarting the core recreates the blueos-core container with a fresh FD table — which is why a core restart has always appeared to "fix" it.

Evidence (controlled measurement, RPi4, 9 Jul 2026)

Time	Board-config page	FD count on PID 122 (`ardupilot_manager/main.py`)
07:51 → 08:56 (65 min)	CLOSED	4630 → 4630 — absolutely flat
09:03:35 → 09:04:24 (50 s)	OPEN	4630 → 4720 — +90 FDs (~108/min)
09:04:24 → 09:04:47	CLOSED	4720 → 4720 — flat immediately

FD composition at 2740 total: 2740 × /dev/i2c-1, 3 sockets, 2 pipes. Not a socket leak. Probe-to-request ratio confirmed from logs: 8920 detect_boards ÷ 445 available_boards = 20:1.

Cause and effect: isolated, reproducible, reversible.

Critical property — the count does NOT reset at launch

FD count accumulates across sessions and only resets when blueos-core is recreated (core restart / reboot). Cumulative page-open time, not wall-clock time, drives time-to-failure (~11 h at ~100 FD/min). Leave that page open two hours a day and the wall is hit inside a week without a single long session.

Why this matters for field operations

The exposure is not leakage during a dive (topside is disconnected; nothing polls). The exposure is launching with a nearly-exhausted FD table after pre-dive setup — precisely when the board-config page is most likely to have been open for hours. Mid-dive, with no operator, ardupilot_manager then fails on the next handle it needs.

Mitigation

Dev (effective immediately, zero code): Do not leave the BlueOS Autopilot Firmware / board-config page open. Open it, do the job, close it. Other BlueOS pages and Cockpit are unaffected and do not leak.

Field — NOT YET IMPLEMENTED (both are open items):

(a) FD-count check on ardupilot_manager as part of the AUV pre-dive gate, blocking launch above a threshold. Operator discipline is not a field-safe control.
(b) failsafe_monitor must detect FC comms loss so W1 goes RED when the board is unreachable — see §15d.

Do NOT patch BlueOS or ArduPilot. Any upstream update reverts local changes. This is an upstream BlueOS 1.4.3 defect; the correct route is an upstream report. All countermeasures must live on our side of the boundary.

The nofile: 65536 setting in /etc/docker/daemon.json remains in place and is harmless — but it is a ceiling raise, not a fix, and must never again be described as one.

15d. Vehicle health in the failsafe assessment — RESOLVED 11 Jul 2026

Previously titled "Vehicle health is NOT in the failsafe assessment — OPEN, SAFETY-CRITICAL." Discovered 9 Jul 2026 (below), fixed 11 Jul 2026 via four commits. Corrected root cause of a second, related fault (cockpit_bridge publishing the wrong variable) found and fixed in the same session — see the last commit below.

Original fault (9 Jul 2026)

Discovered while the flight controller was unreachable ("No board running"):

W1 showed GREEN / "Systems nominal".

Every widget indicator reflected the RPi5 autonomy stack, not the vehicle:

W1 read external/rov-failsafe from failsafe_monitor — which did not gate on FC connectivity.
W8 reads external/rov-mode, sourced from a YAML file. It would show AUV with the Pixhawk in a drawer.
Foxglove "Connected: true" means connected to foxglove_bridge on the RPi5 — not to the vehicle.

Why this was critical: in AUV mode, health_role: gate makes W1 the pre-dive GO/NO-GO gate. A gate that cannot see the flight controller is not a gate — it would pass a dive with a dead FC.

Fix (11 Jul 2026) — four commits, verified live end-to-end

86b2728 — comms loss now forces unconditional RED (previously scored as one marginal parameter among several, only reaching AMBER). Added an fcu_ever_connected latch so the startup window before any heartbeat has ever been received is not reported GREEN by default. Cause text ("Heartbeat Lost", "Vehicle Disconnected") added to FailsafeStatus.message, per DIR-7's "state must carry its cause" MUST.
1b0050d — FSM recovery fix: RETURN_TO_SAFE was a terminal trap (only HOLD_AND_RECOVER could transition back to NORMAL); both states now recover on condition-clear. (Mode-blind — see DIR-7's 11 Jul in-mission-recovery addendum for the still-open mode-dependent design intent this mechanism needs to grow into.)
dbe9515 — cockpit_bridge publishes a new data lake variable external/rov-failsafe-cause (substring of message before the first |; empty when nominal). Stopgap per DIR-7's cause MUST — the DIR's Parked Design Items records the proper fix (a dedicated string cause field on FailsafeStatus.msg).
3409ae5 — W1 widget shows the state word on #w1-message (state-driven, no flicker) and cause text on #w1-footer (red, 11px, blank when nominal).

Verified live: kill mavros → RED + "Heartbeat Lost"; restart mavros → recovers to GREEN.

cockpit_bridge was publishing failsafe_state (the FSM state, which can be 3 or higher) to external/rov-failsafe, not assessment_state (0/1/2) as W1 expects — fixed in commit d854641 (sibling of dbe9515). This was the actual cause of that day's "widgets show no data" symptom (a value of 3 falling outside W1's expected 0–2 range), compounded by a restart-storm that wedged port 9001 while diagnosing it (see CLAUDE.md's one-restart-then-verify-9001 rule, added as a direct result).

Still open (not resolved by this fix — see DIR-7 addenda and Parked Design Items)

Mode-dependent in-mission recovery (AUV commit-to-recovery vs ROV operator handoff)
Auto-recovery (failsafe_monitor re-establishing the FC link itself)
In-mission AMBER runtime-response, branched by mode
Dedicated string cause field on FailsafeStatus.msg (current fix is a substring stopgap)
FD-headroom check on ardupilot_manager as part of the AUV pre-dive gate (§15a mitigation (a), still not built)

MAVLinkRouter vs MAVP2P

BlueOS itself warns "MAVP2P has been presenting issues and is not currently recommended." MAVLinkRouter is the confirmed stable router. Do not recommend or use MAVP2P.

Symptoms (for reference — should not recur):

Frontend connected/disconnected cycling
"No board running" / "No endpoints available" in BlueOS UI
AUTOPILOT_*_FETCH_FAIL (timeout 30000ms) in notifications
OSError: [Errno 24] Too many open files in ardupilot-manager logs

15b. Session 6 Lessons Learned — RPi5 Base Stack Install

SSH on Ubuntu 24.04 Server (cloud-init image)

Ubuntu 24.04 preinstalled server image disables password authentication by default. To enable persistently:

echo "PasswordAuthentication yes" | sudo tee /etc/ssh/sshd_config.d/99-password-auth.conf
sudo systemctl restart ssh

Do NOT add KbdInteractiveAuthentication — removed in OpenSSH 9.x, causes sshd to fail on Ubuntu 24.04.

NVMe clone via dd — PARTUUID handling

dd copies the full disk bit-for-bit including MBR disk identifier, resulting in identical PARTUUIDs on both SD and NVMe. Fix with fdisk. However Ubuntu 24.04 preinstalled image uses LABEL= in both cmdline.txt and fstab, so the duplicate PARTUUID issue does not cause boot failure on this image.

rpi-clone does not support NVMe partition naming

rpi-clone fails on NVMe devices (expects sda1 style, not nvme0n1p1). Use dd for full disk clone.

initramfs rebuild required after NVMe clone on Ubuntu

After cloning to NVMe, rebuild initramfs before booting:

sudo chroot /mnt/nvme-root update-initramfs -u -k all

Ethernet cable quality matters

Two cables failed silently (no LEDs, interface DOWN) before a third worked. Always test with a known-good cable before debugging the OS.

15c. Session 7 Lessons Learned — Workspace Deploy & First Full Launch

The committed Gitea workspace was never build-tested — three latent bugs

Missing config/ dir in rov_bringup (commit f1c6710)
Missing setup.cfg in all 5 Python packages (commit 7590faf) — without them, setuptools installs executables to bin/ instead of lib/<pkg>/, so ros2 launch / ros2 run cannot find them.
ekf.yaml mixed int/float (commit 45b3301) — ROS2 YAML parser requires uniform sequence types.

Process lesson: future handovers must distinguish "written" from "built and run".

colcon: `--symlink-install` works correctly with setup.cfg files in place

With setup.cfg present in each Python package, --symlink-install correctly places executables in lib/<pkg>/. This is now the standard build command.

MAVROS full-launch fixes (commit `c94412d`) — reusable knowledge

fcu_url format: udp://@<remote_ip>:14550 is correct (bind any local, connect remote).
MAVROS plugin collisions on Jazzy: fix with plugin_allowlist: ['sys_status','sys_time','command','imu','global_position','setpoint_velocity'].
QoS on MAVROS topics: use qos_profile_sensor_data (BEST_EFFORT) on MAVROS-sourced subs.
SITL/real hardware battery: treat battery_percent < 0 as UNKNOWN, not critically low.
Rebuild discipline: colcon build --packages-select <pkg> required after editing installed launch/config files.

15e. Session — RPi5 Storage Re-Architecture

RPi5 boot facts:

BOOT_ORDER is a hex value in EEPROM (nibbles, read right-to-left: 1=SD, 6=NVMe, 4=USB, f=restart-loop).
FIX for wrong-root boot: change /boot/firmware/cmdline.txt and /etc/fstab from LABEL= to PARTUUID= (PARTUUIDs differ between drives; labels collided).
End-state BOOT_ORDER: 0xf41 (SD-first). NVMe is data-only — nothing to compete.

SD hardening:

log2ram — /var/log is a RAM disk (128M).
/tmp → tmpfs (256M) via fstab.
journald capped: SystemMaxUse=20M in /etc/systemd/journald.conf.

systemd service:

/etc/argonaut/rov.env — EnvironmentFile: ENV=dev, BLUEOS_IP=192.168.1.100.
/usr/local/bin/argonaut-stack.sh — wrapper: sources ROS2 setup, execs launch.
/etc/systemd/system/argonaut.service — User=ubuntu, EnvironmentFile, Restart=on-failure, RestartSec=5.
DEV POLICY: installed but NOT enabled-on-boot. Start manually: sudo systemctl start argonaut. Field: sudo systemctl enable argonaut.

16. Known Gaps and Risks

Gap	Notes	Priority
DVL absent	X/Y position dead-reckoned from IMU — drifts	High — Phase 7
Ping2 sonar not purchased	W6 full test blocked; downward altitude not flowing	Phase 3 (~$280 USD)
FastAPI backend not built	W3 abort, W4 setup, W5 budget, W7 backup all blocked	High — Phase 3 Stage 2
W3 button label not updated	Widget still shows "ABORT MISSION"; should be "RETURN TO SAFE"	Next session
~~Widget JSON not in Gitea~~	RESOLVED — W6 committed as `w6_altitude.json`, probe widget removed (`785ec56`), see §12	Closed
RPi5 weak SSH password	Password auth enabled, Tailscale service disabled but installed	Pre-field
RPi5 thermal in sealed enclosure	Conduction cooling path unresolved	Phase 3
ML inference compute	RPi5 has no GPU. ONNX/TFLite on-device or Jetson Orin Nano.	Phase 5
Sim/ML dev compute	Track 2 (perception) needs RTX GPU dev PC or cloud. Track 1 (Gazebo) runs on laptop.	Decision pending
Jacket training data	No public dataset. Path = synthetic-from-CAD + SubPipe/COU + field captures.	Phase 5+
PCIe Gen 3 (RPi5 NVMe)	`dtparam=pciex1_gen=3` → ~2× throughput but UNCERTIFIED. Benefit nil until camera→MCAP.	Phase 3 revisit
Ops Server separation	Currently same machine as Dev Server	Near-term
Model conversion pipeline	Manual for now	Phase 6
Widget consolidation	Grant to review — may consolidate W0–W7 into single unified widget	Future

17. Shopping List

Item	Supplier	Approx Cost	Priority
CA-USB-MTI (Xsens cable)	Newark Electronics	—	High — long lead time
Ping2 Sonar (downward)	Blue Robotics	~$280 USD	Phase 3 — unblocks W6 full test
Ops Server (cloud VPS or local machine)	TBD	—	Medium — before field ops

18. Reference Commands

Terminal label convention and shell prompt mapping: [RPi4] → pi@blueos:~$ (192.168.1.100) [RPi5] → ubuntu@rov-brain:~$ (192.168.1.101) [SERVER] → grant@symbytech:~$ (192.168.1.175) [LAPTOP — Git Bash] → grant@BB8 MINGW64 ... Label appears as plain text ABOVE the code block, never inside it.

RPi4 — SSH Access

[LAPTOP — Git Bash]

ssh pi@192.168.1.100

RPi4 — BlueOS UI

[BROWSER]

http://192.168.1.100

RPi4 — BlueOS Container

[RPi4]

sudo docker ps --format "table {{.Names}}\t{{.Status}}"

[RPi4]

sudo docker restart blueos-core

[RPi4]

sudo docker logs blueos-core --tail 30 2>&1

RPi5 — SSH Access

[LAPTOP — Git Bash]

ssh ubuntu@192.168.1.101

RPi5 — argonaut.service

[RPi5]

sudo systemctl start argonaut.service

[RPi5]

sudo systemctl status argonaut.service

[RPi5]

journalctl -u argonaut.service -f

RPi5 — argonaut-api.service (FastAPI backend :8081)

[RPi5]

sudo systemctl start argonaut-api.service

[RPi5]

systemctl is-active argonaut-api.service && curl -s http://localhost:8081/health

[RPi5]

curl -s -X POST http://localhost:8081/abort

[RPi5]

curl -s -X POST http://localhost:8081/mission/start -H "Content-Type: application/json" -d '{"mission_id":"test01","parameters":[]}'

RPi5 — ROS2 Stack Health

[RPi5]

source /data/ros2_ws/install/setup.bash && ros2 node list | wc -l

[RPi5]

ros2 topic echo /mavros/state --once

[RPi5]

ros2 topic echo /rov/recording/active --once

[RPi5]

ros2 topic echo /rov/failsafe --once

RPi5 — Bags

[RPi5]

ls -lh /data/bags/

[RPi5]

du -sh /data/bags/*

RPi5 — Build

[RPi5]

cd /data/ros2_ws && colcon build --packages-select rov_mission --symlink-install 2>&1 | tail -10

Gitea

[LAPTOP — Git Bash]

cd "C:\Users\grant\Desktop\Autonomous ROV System\rov-autonomy"

[LAPTOP — Git Bash]

git pull origin master

[RPi5]

cd /data/ros2_ws/src/rov-autonomy && git pull origin master && git push origin master

Server — General

[SERVER]

df -h && docker ps

[SERVER]

cd ~/harbor && sudo docker compose ps

Server — Harbor

[SERVER]

docker login registry.symbytech.com

Browser — Dev URLs

[BROWSER]

http://192.168.1.100           BlueOS (RPi4)
http://192.168.1.101:8765      Foxglove bridge WebSocket (connect via Foxglove Studio)
https://registry.symbytech.com Harbor
https://git.symbytech.com      Gitea
http://192.168.1.175:81        NPM admin
http://192.168.1.175:9000      Portainer

[COCKPIT]

1. Connect to vehicle (192.168.1.100)
2. Settings -> General -> Enable Pirate Mode (required for DIY widgets, resets each session)
3. Enter edit mode (pencil icon, top right)
4. Add Widget at bottom — scroll right to find </> DIY widget
5. Drag onto main view area and resize
6. Click gear icon on widget -> Import -> select .json from rov-autonomy/widgets/

[COCKPIT]

Generic WebSocket (one-time setup, persists):
Settings -> Generic WebSocket Connections
Add: ws://192.168.1.101:9001

19. Document Index

All project docs live in the repo under docs/ (git-tracked, no version number in the filename — the Version: field inside each file is authoritative; project knowledge holds a mirror copy for chat context, re-uploaded after any commit that changes these). Do not cite a version number from memory — check the file itself.

Document	Purpose
`docs/handover.md`	This document — master reference, single source of truth for current state
`CLAUDE.md` (repo root)	Rules for Claude — behaviour, conventions, safety, research requirements
`docs/design_intent_register.md`	Design Intent Register — authoritative design decisions, DIR-1…DIR-12
`docs/failsafe_design.md`	Failsafe state machine and sensor roadmap
`docs/ui_design.md`	Cockpit widget specs, data paths, and build status
`docs/simulation_ml_plan.md`	Simulation + ML tooling: dev-PC tiers, cloud options, training assets
`docs/mission_recording_lifecycle.md`	Lifecycle research backing DIR-9/9a/10

Server/infra docs (SymbyTech server, Harbor registry) and task/shopping lists stay project-knowledge-only — not part of the rov-autonomy repo docs/ folder.

Update this document at the end of each working session. Increment version number and add rows to Version History and Changelog as appropriate. Commit to the repo (docs/handover.md); re-upload to Claude project knowledge for chat context.

77 KiB Raw Blame History Unescape Escape

ROV Autonomous Inspection System — Project Handover

0. QUICK STATUS BOARD (read this first)

Version History

Changelog

Design Intent Register — pointer

1. Project Overview

Goal

Target Structures

Key Design Values

2. Platform

3. Hardware Architecture

On the ROV

Sensors

Power

Storage

4. Software Stack

RPi4 (BlueOS Gateway)

RPi5 (Autonomy Brain)

Edge PC / Field Laptop BB8 (Topside)

5. System Architecture — Three Zones

Ethernet Tether — Single Cable

6. Three-Prong Mission System

Prong 1 — Mission Creation + Editing

Prong 2 — Mission Execution + Monitoring

Prong 3 — Data Collection + Replay

7. Digital Twin

8. ROS2 Workspace

Package Structure

Key Topics

Key Commits

Build/Deploy Procedure

9. UI Architecture — Argonaut 3 / Cockpit

Cockpit Setup

Widget Approach — DIY Widgets (Confirmed)

Data Injection Path — Confirmed Production Path

Data Lake Variables

Widget File Locations

Widget Build Status

How to Install a Widget

10. Failsafe Design Summary

Three States

Trigger Conditions

11. Infrastructure — SymbyTech Server

Gitea

Harbor Registry

Backup System (built 30 Jun 2026)

Development VM — DECOMMISSIONED

12. Phase Task Status

Phase 1 — Foundation

Phase 2 — Widgets and Data Path

Phase 3 — Recording and Mission Control (Active)

Next Priorities

13. Session Lessons Learned — Data Injection Investigation (Task 11)

What was attempted and why it failed

What works

Key architectural insight

14. Roadmap Summary

15. Key Architecture Decisions (Locked)

15a. ardupilot-manager FD Exhaustion — OPEN BUG (root cause corrected 9 Jul 2026)

What the old entry claimed (WRONG — do not reinstate)

Actual root cause (measured 9 Jul 2026)

Evidence (controlled measurement, RPi4, 9 Jul 2026)

Critical property — the count does NOT reset at launch

Why this matters for field operations

Mitigation

15d. Vehicle health in the failsafe assessment — RESOLVED 11 Jul 2026

Original fault (9 Jul 2026)

Fix (11 Jul 2026) — four commits, verified live end-to-end

Related fault found and fixed in the same session

Still open (not resolved by this fix — see DIR-7 addenda and Parked Design Items)

MAVLinkRouter vs MAVP2P

15b. Session 6 Lessons Learned — RPi5 Base Stack Install

SSH on Ubuntu 24.04 Server (cloud-init image)

NVMe clone via dd — PARTUUID handling

rpi-clone does not support NVMe partition naming

initramfs rebuild required after NVMe clone on Ubuntu

Ethernet cable quality matters

15c. Session 7 Lessons Learned — Workspace Deploy & First Full Launch

The committed Gitea workspace was never build-tested — three latent bugs

77 KiB

Raw Blame History

colcon: `--symlink-install` works correctly with setup.cfg files in place

MAVROS full-launch fixes (commit `c94412d`) — reusable knowledge