rov-autonomy/docs/handover.md

ROV Autonomous Inspection System — Project Handover

Project Name: Argonaut 3 Version: 2.25 Date: 13 July 2026 File: docs/handover.md (repo-tracked, no version number in filename — see Version History for versioning)

This document is the single source of truth for current project state — what's built, verified, broken, parked, and next. Rules for how Claude works are in CLAUDE.md (repo root). Design intent ("why") is in docs/design_intent_register.md. Domain specs (UI, failsafe, sim/ML, recording lifecycle) are in docs/, referenced from §0 and inline, not duplicated here.

Status: ⚠ CORRECTION (still open) — the BlueOS "FD exhaustion" root cause recorded in v2.10–v2.21 was WRONG. It is NOT TCP-socket/uvicorn exhaustion from UI polling, and nofile=65536 did NOT fix it — that change only raised the ceiling (failure at ~10 min → ~11 h of cumulative exposure). The real cause is a /dev/i2c-1 file-descriptor leak in ardupilot_manager's Linux board detector, driven by the BlueOS Autopilot Firmware / board-config page. See §15a. This bug is OPEN.

✅ Resolved this session (13 Jul 2026) — the operating-mode profile system was found to have been dark at runtime since it was built: mode_profile_loader was never wired into the stack launch file, so /rov/mode/profile had zero publishers. Fixed and verified live (464e17e). With the mode system now actually publishing, failsafe_monitor's mode-aware recovery (2a3e577) surfaced a second gap — gate-mode committed recovery entered RETURN_TO_SAFE and could never leave it. Fixed with a new permanent arrival interface, verified live end-to-end on the bench (bf815cc). See §15f and Changelog, 13 Jul entries.

Full narrative for each prior version is in Version History below — not repeated here.

---

0. QUICK STATUS BOARD (read this first)

CONFIRMED WORKING (verified across sessions to date):

• Full autonomy stack — 19 nodes, EKF ~11 Hz on /odometry/filtered, /tf valid. Runs as systemd argonaut.service (manual-start in dev — verify BlueOS heartbeat first).
• Pixhawk 2.4.8 — confirmed active on /dev/ttyACM0. connected: true, system_status: 3 (STANDBY), mode: MANUAL. SITL is NOT used in production. BlueOS preferred_board: Pixhawk1.
• MAVLink routing — mavlink-routerd running on ttyACM0:115200. MAVLinkRouter is the confirmed router. MAVP2P has known issues (BlueOS own warning) — do not use, do not suggest.
• BlueOS FD exhaustion — permanently fixed ✗ RETRACTED (9 Jul 2026). NOT FIXED. See §15a. The nofile: 65536 change raised the ceiling only. Real cause is a /dev/i2c-1 FD leak in ardupilot_manager's Linux board detector. Bug is OPEN. Dev mitigation: do not leave the BlueOS Autopilot Firmware / board-config page open.
• cockpit_bridge node — ROS2 → WebSocket :9001 → Cockpit data lake. 19 variables live (8 telemetry + 11 mode-profile flags).
• Operating-mode profile system — mode_profile_loader reads mode_profiles.yaml (ROV/AUV defaults + custom hybrid via Save As), publishes the resolved flag set latched on /rov/mode/profile, reloadable via /rov/mode/reload (std_srvs/Trigger). cockpit_bridge surfaces the flags to the data lake as external/rov-*. Widgets read flags, not modes; hybrid = any saved profile with overrides. Verified end-to-end: ROV/AUV resolve + live reload. Commit 4e48dc2. Found never wired into the stack launch file (13 Jul 2026) — built and committed 7 Jul but never added to rov_full.launch.py, so /rov/mode/profile had zero publishers at runtime the whole time. Fixed with respawn, commit 464e17e, verified live (publisher count 0→1). See §15f.
• Recording Manager — recording_manager.py node active in argonaut.service. Publishes /rov/recording/active (Bool). Confirmed data: true in full stack. Bags to /data/bags/. Commit 5d75a9d.
• /data/bags/ — created on NVMe. Recording live at ~2.5 MB/s (nav/MAVLink topics, no camera). 512GB NVMe will last weeks at this rate.
• W0 Data Lake Inspector — confirmed working. 706/706 vars live.
• W1 System Health Indicator — failsafe_monitor now drives RED on FC comms loss (11 Jul 2026). Previously visual-confirmed only (GREEN, full test pending); now verified live end-to-end: kill mavros → RED + "Heartbeat Lost", restart → recover to GREEN. Four commits 86b2728/1b0050d/dbe9515/3409ae5 — full detail in Changelog below. §15d (was OPEN, SAFETY-CRITICAL) is resolved.
• Mode-aware failsafe recovery (13 Jul 2026) — failsafe_monitor now subscribes to /rov/mode/profile and branches condition-clear recovery on health_role (2a3e577): gate (AUV) commits HOLD_AND_RECOVER→RETURN_TO_SAFE on condition-clear instead of returning to NORMAL; advisory (ROV) holds and waits for an explicit operator resume (placeholder topic). This surfaced a new gap once the mode system was actually live — gate-mode RETURN_TO_SAFE had no exit, a terminal trap. Closed by a new PERMANENT interface, /rov/nav/safe_zone_reached (std_msgs/Bool), that transitions the FSM to NORMAL on arrival (bf815cc). Verified live end-to-end on the bench: published the event by hand, watched the FSM release RETURN_TO_SAFE→NORMAL and the motion_controller recovery broadcast stop. The topic/subscription are permanent; today's publisher is a manual bench command only — the real navigation-raised event and a temporary W1-contextual operator button are both still to come. See §15f, DIR-7 addendum, and the new Parked Design Item in docs/design_intent_register.md.
• W6 Altitude Widget — deployed. NO SONAR state confirmed (expected — no Ping2 connected). Commit 0640353.
• Backup system — push model, both Pis → server. RPi4: 00:00 daily. RPi5: 00:30 daily. Server monitor alerts on FAILED (always) and OVERDUE ≥2 days.
• FastAPI backend (rov_api) — deployed and verified end-to-end on RPi5:8081. Runs as argonaut-api.service (manual-start in dev). Endpoints confirmed: GET /health (live mission/recording/failsafe), POST /abort (W3 → /rov/mission/abort → failsafe_monitor FSM NORMAL→HOLD_AND_RECOVER), POST /mission/start (MissionCommand START), POST /mission/stop (MissionCommand ABORT). Commit 66a18ef.
• Deploy files version-controlled — repo deploy/ folder holds both services' units + wrappers + README. Commit 97decac.
• Hostname resolution (DNS) — symbytech/blueos/rov-brain resolve on laptop (Windows hosts) and RPi5 (/etc/hosts). Git remotes on hostname form.
• Gitea remote — hostname form: ssh://git@symbytech:222/administrator/rov-autonomy.git
• All three (Gitea/laptop/RPi5) level at 4e48dc2.

UNTESTED (visual/UI confirmed, real data not flowing yet):

• W2 Mission Status — UI confirmed (IDLE state + progress bar). Untested: needs a mission running. Test: POST /mission/start via FastAPI (now available) with a loaded plan, observe widget.
• W4 Mission Setup Button — UI confirmed (cyan button, opens setup URL). Untested: needs a setup page served by FastAPI (not yet built — Stage 2b/Phase 6).
• W5 Return Budget — UI confirmed (shows pending). Untested: bridge does not yet publish external/rov-return-budget. Backend deferred to Stage 2b (return budget approach under discussion — "shortest path to safety" concept to be defined).

BLOCKED / NOT BUILT:

• W3 RETURN TO SAFE button — backend now LIVE (/abort verified end-to-end). Remaining: update widget code label from "ABORT MISSION" to "RETURN TO SAFE" and set FASTAPI_HOST to http://192.168.1.101:8081.
• W4 setup page — the page FastAPI would serve is not built (Stage 2b/Phase 6).
• W5 return budget — /return_budget endpoint deferred to Stage 2b.
• W7 Backup Due Notice — not built. Depends on FastAPI /backup/* (Stage 2b, Option A design: server exposes read-only JSON status, RPi5 polls outbound).
• external/rov-return-budget — not published by cockpit_bridge. W5 budget figure pending.

OPEN ITEMS / HOUSEKEEPING:

• W3 widget code: update button label to "RETURN TO SAFE"; set FASTAPI_HOST constant.
• W6 full test: hardware-gated on Ping2 sonar purchase (Phase 3, ~$280 USD).
• RPi5 SSH password: weak dev password. Harden before field deployment (bundled with UFW).
• UFW: inactive on RPi5. Deferred to pre-field/pre-remote hardening pass (no online exposure currently — Tailscale disabled, no proxy to Pis). Bundle with SSH password change.
• Widget consolidation: Grant to review as development progresses — may consolidate W0–W7 into a single unified widget. Flagged thought, not yet designed.
• W2, W4, W5: untested. Require system testing with live mission / backend pages.
• Optional: RPi5 remote was on hostname form after DNS fix — consistent with laptop. Done.
• flag_manual_abort in failsafe_monitor is a one-way latch, never reset to False (found 13 Jul 2026, pre-existing, not caused by that session's work). Once any manual abort fires, the priority-6 handler returns early on every subsequent evaluation cycle for the rest of the process lifetime — permanently shadowing all lower-priority handling, including the new mode-aware recovery and arrival-event logic. Needs a reset condition (candidates: on reaching NORMAL, on an explicit operator reset, or process-restart only — undecided, needs its own session). Field-deployment concern: one manual abort currently disables all future automatic recovery for the rest of that dive. See §15f.
• failsafe_monitor produces no log output in journalctl (found 13 Jul 2026). State had to be read directly off /rov/failsafe every time rather than watched in the service log. Should log state transitions and cause on each one. Minor, but cost real diagnostic time. See §15f.

PARKED (design decisions deferred, priority order — moved here from CLAUDE.md 11 Jul 2026):

1. Doc restructure — DONE this session (11 Jul 2026): this document, CLAUDE.md, and docs/design_intent_register.md restructured/updated; all docs moved into repo docs/.
2. Mode-dependent failsafe recovery — AUV (health_role: gate) commits to recovery on condition-clear, does not auto-resume; ROV (advisory) holds and hands the continue-or-recover decision to the operator. Essentially complete (13 Jul 2026): failsafe_monitor now reads the mode profile and branches on health_role (2a3e577), and gate-mode RETURN_TO_SAFE now has a real exit via the new permanent /rov/nav/safe_zone_reached interface (bf815cc), verified live end-to-end on the bench. Remaining: the event's publisher — navigation cannot yet raise it, so the immediate next step is disposable bench scaffolding, a temporary W1-contextual operator button (shown only while W1 reads the recovering/RETURN_TO_SAFE state, backed by a rov_api endpoint), to be deleted outright once navigation publishes the real GPS/EKF-derived event. Advisory mode's operator resume is also still a placeholder topic pending the proper rov_api endpoint. See DIR-7 addendum + new Parked Design Item, docs/design_intent_register.md.
3. Auto-recovery — failsafe_monitor should attempt to re-establish the FC link itself, escalating/holding RED only if it cannot. Real-world dives have no operator to restore comms. Intent now recorded in DIR's Parked Design Items (11 Jul 2026) — implementation still open.
4. Dedicated string cause field in FailsafeStatus.msg, to replace the message-substring shortcut currently used in cockpit_bridge. Intent now recorded in DIR's Parked Design Items (11 Jul 2026) — implementation still open.
5. Operator control surface (start/pause/stop via rov_api) and one-command stack startup + healthcheck.
6. UFW / SSH pre-field hardening — see Open Items above.
7. In-mission AMBER runtime-response — AMBER means action-required once a mission is underway, not just informational; response branches by mode. Intent now recorded as a DIR-7 addendum (11 Jul 2026) — AUV's autonomous curtailment logic is unspecified, left for a dedicated design session.

NEXT (priority order, carried from §12):

1. Safe-zone-reached arrival button (bench scaffolding) — temporary W1-contextual operator button, shown only while W1 reads the recovering/RETURN_TO_SAFE state, backed by a new rov_api endpoint that publishes True to /rov/nav/safe_zone_reached. Disposable — deleted outright once navigation can raise the real GPS/EKF-derived arrival event on the same (permanent) topic. See §15f, DIR-7 addendum, and the new Parked Design Item.
2. W3 widget code: update label to "RETURN TO SAFE", set FASTAPI_HOST = http://192.168.1.101:8081, then test W3 end-to-end (button → /abort → FSM, already verified server-side).
3. Stage 2b design decisions (deferred pending Grant): return budget "shortest path to safety" concept; W7 backup notice (Option A confirmed: server read-only JSON, RPi5 polls outbound).
4. W2 testing: test with live mission via POST /mission/start (backend now available)
   • loaded plan.
5. Ping2 sonar purchase (~$280 USD): required for W6 full end-to-end test.
6. Pre-field hardening pass: UFW + RPi5 SSH password (bundled, triggered before remote ops/field).

DATA LAKE VARIABLES (19 total, published by cockpit_bridge):

Variable	Source topic	Notes
external/rov-failsafe	/rov/failsafe	0=GREEN, 1=AMBER, 2=RED; -1=no data
external/rov-depth	/rov/depth	Depth in metres (3 dp)
external/rov-voltage	/mavros/battery	Battery volts (2 dp)
external/rov-heading	/mavros/mavros/compass_hdg	Compass degrees (1 dp)
external/rov-ms	/rov/mission/status	Mission state: 0=IDLE…4=ABORTED; -1=no data
external/rov-mp	/rov/mission/status	Mission progress 0-100%
external/rov-recording	/rov/recording/active	1=recording, 0=not recording
external/rov-altitude	/ping2/altitude	Ping2 altitude metres; -1.0=NO SONAR
external/rov-mode	/rov/mode/profile	Active profile name (string): ROV / AUV / hybrid
external/rov-mode-base	/rov/mode/profile	Base mode: 0=ROV, 1=AUV
external/rov-hilt	/rov/mode/profile	HILT link: 1=present, 0=absent
external/rov-health-role	/rov/mode/profile	W1 role: 0=advisory, 1=gate
external/rov-rec-control	/rov/mode/profile	Recording: 0=manual, 1=auto (ARM-bounded)
external/rov-nogo-gate	/rov/mode/profile	Recording no-go: 0=relaxed, 1=enforced
external/rov-rec-reminder	/rov/mode/profile	ARM reminder (manual): 1=on, 0=off
external/rov-return-mode	/rov/mode/profile	Return: 0=operator-selectable, 1=autonomous
external/rov-live-telem	/rov/mode/profile	Live telemetry: 0=on, 1=predive-only
external/rov-nav-display	/rov/mode/profile	Nav display: 1=shown to pilot, 0=autonomy only
external/rov-camera-role	/rov/mode/profile	Camera: 0=pilot-primary, 1=record-only

RECENT COMMITS (latest first):

Hash	Description
bf815cc	failsafe_monitor: /rov/nav/safe_zone_reached permanent interface closes gate-mode RETURN_TO_SAFE terminal trap
464e17e	rov_full.launch.py: wire mode_profile_loader in (was built 4e48dc2, never launched) + respawn
2a3e577	failsafe_monitor: mode-aware recovery (health_role gate/advisory) per DIR-7 addendum
3409ae5	W1 widget: state word on #w1-message (no flicker), cause text red 11px on #w1-footer
dbe9515	cockpit_bridge: publish external/rov-failsafe-cause (substring stopgap per DIR-7)
d854641	Fix cockpit_bridge publishing failsafe_state (FSM) instead of assessment_state to rov-failsafe — real cause of that day's "no data" symptom
1b0050d	failsafe_monitor FSM fix: RETURN_TO_SAFE was a terminal trap, both states now recover to NORMAL on condition-clear
86b2728	failsafe_monitor: comms loss forces unconditional RED + fcu_ever_connected latch + cause text on FailsafeStatus.message
4e48dc2	feat(mode): operating-mode profile system — ModeProfile.msg, mode_profile_loader, mode_profiles.yaml, cockpit_bridge flags
785ec56	Widgets housekeeping: add W6 (w6_altitude.json), remove probe widget
97decac	Add deploy/ folder — systemd units and launch wrappers + README
66a18ef	Phase 3 Stage 2: rov_api FastAPI backend (core) — /health, /abort, /mission/start, /mission/stop
0640353	W6: Add rov-altitude to cockpit_bridge for Ping2 downward sonar
5d75a9d	Phase 3 Stage 1: Recording Manager (DIR-9)
2cb7b74	W0 layout fix (3-column, horizontal scroll)
66f9bd7	W0–W5 all confirmed working, committed
c94412d	Full stack zero-crash launch (MAVROS fcu_url, plugin_allowlist, QoS, battery guard)
12b5161	cockpit_bridge node — ROS2→WebSocket :9001

Table has historically lagged actual commit history (e.g. 9109439, 87000e6, c099c00, a5e1c99 from the 8–9 Jul session were never added) — not reconstructed retroactively here without measuring actual git log; flagged so it doesn't look more complete than it is.

---

Version History

Version	Date	Changes
2.25	13 Jul 2026	Two-part fix session, both surfaced by putting the 11 Jul mode-aware work under load. (1) mode_profile_loader was found never to have been wired into rov_full.launch.py despite being committed 7 Jul (4e48dc2) — /rov/mode/profile had zero publishers at runtime since then, so the entire operating-mode profile system had been dark and failsafe_monitor's health_role had been silently running on its gate default the whole time. Fixed by adding the node to the launch file with respawn (464e17e), verified live (publisher count 0→1). (2) With the mode system now actually publishing, failsafe_monitor's mode-aware recovery (2a3e577, same session — subscribes to the resolved profile, branches condition-clear recovery on health_role) surfaced a second gap: gate-mode committed recovery entered RETURN_TO_SAFE and could never leave it, because that branch deliberately ignores further condition-clear so recovery runs to completion, and nothing signalled completion. Fixed by a new PERMANENT interface, /rov/nav/safe_zone_reached (std_msgs/Bool), that transitions the FSM RETURN_TO_SAFE→NORMAL on arrival when gate-mode (bf815cc), verified live end-to-end on the bench (published the event by hand, watched the FSM release and the motion_controller recovery broadcast stop). Today's publisher is temporary bench scaffolding only — real navigation-raised arrival and a temporary W1-contextual operator button are both still to come; see the new Parked Design Item in docs/design_intent_register.md v1.3. §0 Confirmed Working, PARKED, and NEXT updated; new §15f narrative section. Two pre-existing failsafe_monitor defects found this session (not caused by it): flag_manual_abort is a one-way latch never reset, permanently shadowing all lower-priority handling (including this session's own new logic) after any single manual abort — field-deployment concern, needs its own session; and failsafe_monitor produced no log output in journalctl this session, costing diagnostic time. Both added to §0 Open Items and §15f. Companion DIR bump: docs/design_intent_register.md v1.3 adds the mode-aware safe-zone-reached Parked Design Item — see that changelog for detail.
2.24	11 Jul 2026	§15d resolved. Vehicle health / FC-comms-loss fault (discovered 9 Jul, flagged OPEN/SAFETY-CRITICAL in v2.22) is fixed as of four commits today: 86b2728 (unconditional RED on comms loss + fcu_ever_connected latch + cause text), 1b0050d (FSM recovery fix — RETURN_TO_SAFE was a terminal trap), dbe9515 (new external/rov-failsafe-cause variable), 3409ae5 (W1 widget shows state + cause text). Verified live end-to-end. Related fault found+fixed same session: cockpit_bridge was publishing failsafe_state (FSM, 3+) not assessment_state (0/1/2) to rov-failsafe — the actual cause of that day's "widgets show no data," fixed in d854641, compounded by a restart-storm wedging port 9001 (→ CLAUDE.md's one-restart-then-verify-9001 rule). §0 Confirmed Working and Recent Commits updated; PARKED list updated to reflect intent now recorded in the DIR (mode-dependent recovery, auto-recovery, dedicated cause field, AMBER runtime-response — all still open on implementation, closed on design intent). Companion DIR bump: docs/design_intent_register.md v1.2 adds the two DIR-7 addenda and a Parked Design Items section — see that changelog for detail, not repeated here.
2.23	11 Jul 2026	Doc restructure. This document is now the single source of truth for current state (Quick Status Board gains PARKED and NEXT subsections, folding in the list previously buried in CLAUDE.md's SESSION LOG and §12's Next Priorities). New Changelog section (below) replaces CLAUDE.md's SESSION LOG — same entries, same append-only format, moved here. The embedded Design Intent Register (formerly here, ~200 lines) is removed — full DIR now lives only in docs/design_intent_register.md; this document keeps a short pointer + summary of which DIRs govern current work. All docs moved into the repo under docs/ (git-tracked, no version number in filenames — see CLAUDE.md Documentation Discipline). Fixed: duplicate ## 15d heading (second instance renamed 15e); §15's "Recording control" locked-decision row corrected from "DIR-9 — always-on" to DIR-9a (ARM-bounded); duplicate 8-variable data-lake table in §9 replaced with a pointer to §0's canonical 19-variable table; stale "Widget JSON not in Gitea" row in §16 marked resolved (W6 committed per 785ec56, already reflected in §12). No new facts asserted beyond what was already recorded elsewhere in this document, CLAUDE.md, or the DIR — this pass is consolidation, not new findings.
2.22	9 Jul 2026	⚠ ROOT-CAUSE CORRECTION (supersedes v2.10 and v2.19 claims). The BlueOS "FD exhaustion" bug was NEVER fixed. nofile=65536 was a ceiling raise, not a fix — it moved failure from ~10 min to ~11 h of cumulative exposure. Real cause, measured: ardupilot_manager's Linux board detector opens /dev/i2c-1 on each failed probe and never closes it. The BlueOS Autopilot Firmware / board-config page polls GET /v1.0/available_boards; each request fires 20 detect_boards probes (Navigator/NavigatorPi4/NavigatorPi5/Argonot). We run a Pixhawk on USB serial, so every probe fails and leaks. Measured ~100–108 FD/min with the page open; exactly zero with it closed (4630 flat over 65 min; +90 FD in 50 s on opening; flat immediately on closing). FD composition: 2740/2745 were /dev/i2c-1, 3 sockets. Probe:request ratio 20:1 (8920:445, from logs). Count accumulates across sessions; resets only when blueos-core is recreated. Upstream BlueOS 1.4.3 defect — do NOT patch BlueOS (updates revert). Dev mitigation: don't leave the board-config page open. Field mitigation NOT built. §15a rewritten. New §15d — OPEN, SAFETY-CRITICAL: failsafe_monitor reported GREEN while the FC was unreachable; W1 is the AUV pre-dive GO/NO-GO gate (health_role: gate) and would pass a dive with a dead flight controller. All widget indicators reflect the RPi5 stack, not the vehicle. Also this session: rov_api /mode endpoints (GET/POST, validate-before-write + /rov/mode/reload trigger) built, deployed, tested incl. rejection path (9109439). W8 mode indicator built and validated live — render + data (87000e6). Chronic "widgets show no data" root-caused: cockpit_bridge was in the launch file but had no respawn; a boot-time race left it permanently dead, taking the whole data-lake feed with it — fixed with respawn=True (c099c00), verified across a cold stack restart. Respawn extended to failsafe_monitor, motion_controller, ekf_filter_node, depth_node, mission_executor (a5e1c99); deliberately NOT applied to mavros_node (would mask FCU-link instability), recording_manager (pending DIR-9a rework), perception (not operational), foxglove_bridge (viz-only). FLAGGED OPEN: recording_manager.py still implements superseded DIR-9 (continuous-from-boot) rather than governing DIR-9a (ARM-bounded); and argonaut-stack.sh passes record:=${RECORD} to a launch file that declares no such arg, so RECORD=false is silently ignored. Claude Code (VS Code) adopted for repo edits; SESSION LOG added to CLAUDE.md to bridge Claude Code ↔ claude.ai.
2.21	7 Jul 2026	Operating-mode profile system built, verified end-to-end, committed (4e48dc2) and pushed. New ModeProfile.msg (rov_interfaces) — atomic resolved flag set. New mode_profile_loader node + config/mode_profiles.yaml (rov_mission): ROV/AUV default profiles plus custom hybrid via Save As; resolved flags published latched on /rov/mode/profile; reloadable via /rov/mode/reload (std_srvs/Trigger); fail-safe validation rejects bad flag values (caught a YAML 1.1 on→bool trap; fixed by quoting + a normaliser). cockpit_bridge now surfaces 11 mode flags to the data lake — 19 external/rov-* variables total (was 8). Widgets read flags, not modes; hybrid = any saved profile with overrides. Verified: ROV and AUV both resolve + live reload. DIR reworked for consistency: DIR-9 collapsed to a stub; DIR-9a governing — single ARM-bounded black-box recording (AUV: ARM starts / DISARM stops; ROV: operator start/stop + ARM reminder; purpose black-box/debug/ML/fault-finding only; certified deliverable capture out of scope pending Class-approved devices). DIR-1 corrected (tether/HILT, not power source, determines mode; tethered may use battery or topside supply). DIR-7 gains AUV-mode health semantics (GREEN/AMBER/RED = within/approaching/terminate, parameter-driven). DIR-4/5/10 recording contradictions removed (always-on continuous "core" dropped; base set now recorded within the armed run). Backup: overdue alert resolved (07-05 was a one-night hostname-resolution transient, not a config regression); segment _9 size-mismatch re-synced (captured mid-rotation); RPi5 backup script hardened (--exclude='*.mcap' so in-progress bags are not captured mid-write; active-dive prune guard).
2.20	July 2026	Phase 3 Stage 2 core complete: rov_api FastAPI backend deployed on RPi5:8081 (commit 66a18ef) — HTTP-to-ROS2 bridge, uvicorn main thread + embedded rclpy node in daemon thread. Endpoints verified end-to-end: GET /health (mission/recording/failsafe), POST /abort (W3 RETURN TO SAFE → /rov/mission/abort → failsafe_monitor FSM NORMAL→HOLD_AND_RECOVER), POST /mission/start (MissionCommand START), POST /mission/stop (MissionCommand ABORT). Runs as argonaut-api.service (manual-start, same policy as argonaut.service). fastapi/uvicorn installed system-wide (--ignore-installed typing_extensions). Deploy files version-controlled in repo deploy/ folder (commit 97decac): both services' units + wrappers + README documenting locations, pip deps, install steps. Hostname resolution (DNS) fixed on laptop (Windows hosts) and RPi5 (/etc/hosts) — symbytech/blueos/rov-brain resolve; git remotes returned to hostname form. Widget housekeeping (commit 785ec56): W6 committed as w6_altitude.json (underscore convention), probe widget removed from Gitea, 7 local Copy/working cruft files deleted, W0–W5 line-ending-only churn discarded. All three (Gitea/laptop/RPi5) level at 785ec56. Stage 2b deferred (return budget, /backup/*, /mission/upload) — return budget approach pending "shortest path to safety" discussion; W7 backup design chosen (Option A: server read-only JSON, RPi5 polls outbound). UFW deferred to pre-field hardening pass (no online exposure).
2.19	July 2026	Phase 3 Stage 1 complete: recording_manager deployed — continuous rosbag2 MCAP recorder, zstd compression, 500MB bag splitting, 10GB retention policy, 1Hz health monitoring, hard no-go gate on /rov/recording/active. Pixhawk 2.4.8 confirmed active FC (SITL removed from BlueOS); BlueOS FD exhaustion claimed fixed (/etc/docker/daemon.json, nofile=65536) — ✗ this claim was WRONG, retracted in v2.22, see §15a. MAVLinkRouter confirmed; MAVP2P not recommended (BlueOS own warning). W6 Altitude Widget built and deployed; cockpit_bridge updated to 8 variables (rov-recording, rov-altitude added). W3 button label decided: "RETURN TO SAFE" (per DIR-5 — assessed vehicle return-to-safe, not emergency hard-stop). Tailscale stopped/disabled on RPi5. Gitea remote updated to IP-based URL. ardupilot_manager_monitor.sh deleted from RPi5. Probe widget deleted (dev tool, not part of W0–W7).
2.18	July 2026	Simulation & ML tooling scoped into new companion doc Argonaut3_Simulation_ML_Tooling_Plan_v1_0.md. Two tracks defined: Track 1 (Gazebo Harmonic + ArduSub SITL, CPU-led, runs on existing laptop) for nav/mission/failsafe bench; Track 2 (HoloOcean/UNav-Sim/Isaac, RTX GPU) for perception synthetic-data + model training; convergence = structure-relative visual localization (the real bottleneck). Dev-PC tiers costed (min $1.2–1.6k / rec $2.0–2.8k / high $3.5–6k+); cloud alternative costed (free: Kaggle/Colab/Lightning; paid RTX 4090 ~$0.33/hr Vast/RunPod). Training-asset sources listed (Fab/Megascans free; SubPipe/COU datasets; YOLO/SAM2 base models) — NO public jacket dataset exists, CAD+synthetic is the path. RECORD CORRECTED: past SITL failures were under-resourced hosts (server 2-vCPU VM; RPi4 + half-connected Pixhawk) — not intrinsic to SITL. R210 server cannot host GPU sim (1U). PCIe Gen 3 on RPi5 NVMe: option logged, DEFERRED to Phase 3 camera-to-MCAP.
2.17	June 2026	W0–W5 all committed to Gitea (W0 layout fix 2cb7b74). Stale server clone ~/rov-autonomy deleted. MAVROS recovery: confirmed data point — RPi4 BlueOS core restart ALONE recovered the system this session (RPi5 untouched). DESIGN INTENT REGISTER added as Section 1 — dual-capability AUV/ROV platform, HILT as override channel, mission lifecycle, abort/return-to-safe model (mission-bounded), Emergency Surface shelved. Re-scopes W3/W4 backend: near-term need is tethered HILT path. FastAPI backend NOT started — intent locked first.
2.16	June 2026	W1–W5 all confirmed working in Cockpit v1.18.2 with live data. W1 failsafe variable name corrected (rov-failsafe full word, not truncated rov-failsa). Outstanding widget work is backend-side. Flagged items consolidated.
2.15	June 2026	Cockpit v1.18.2 DIY widget data lake getter CONFIRMED via probe widget. getDataLakeVariableData(id) returns live values. W1–W5 widget blocker resolved. Full window.cockpit surface documented.
2.14	June 2026	COCKPIT DATA BRIDGE BUILT & WORKING (the P2 keystone). New cockpit_bridge ROS2 node serves a WebSocket on :9001; all 6 vars land in data lake as external/rov-* (commit 12b5161). RPi4 SD replaced + backup system (pi4-backup.sh) in place & verified. Logging flood (NaN TF) mitigated. W0 working (scroll fixed).
2.13	June 2026	RPi5 storage RE-ARCHITECTED to OS-on-SD + data-on-NVMe. OS migrated SD via rsync; NVMe wiped to single ext4 data volume at /data; workspace moved to /data/ros2_ws. Boot fixed by switching root mount from LABEL to PARTUUID. Full stack now runs as systemd service argonaut.service.
2.12	June 2026	Full stack now launches end-to-end with ZERO crashes (commit c94412d). Fixed: MAVROS fcu_url, plugin_allowlist, timestamped MCAP bag dir, BEST_EFFORT QoS, battery<0=UNKNOWN guard.
2.11	June 2026	Full autonomy stack deployed and launched end-to-end on RPi5. THREE repo bugs found and fixed. foxglove_bridge :8765 + MCAP recording to /data/bags confirmed working.
2.10	June 2026	MAVROS link RPi5->RPi4 verified. BlueOS "too many open files" re-diagnosed as uvicorn FD exhaustion. ✗ WRONG — retracted in v2.22, see §15a.
2.9	June 2026	RPi4 fully reinstalled. BlueOS 1.4.3 on fresh SD. Static IP 192.168.1.100. ArduSub SITL 4.5.7 STABLE running. Both Pis operational.
2.8	June 2026	RPi5 base stack complete. Ubuntu 24.04.4 installed. Hostname rov-brain, static IP 192.168.1.101, Tailscale. ROS2 Jazzy, Docker, MAVROS installed.
2.7	May 2026	Cockpit version corrected to v1.18.0. Generic WebSocket confirmed as correct dev injection path. Data lake key format confirmed with external/ prefix.
2.6	May 2026	All 6 widgets built and confirmed working in Cockpit native desktop app. Widget files committed to Gitea.
2.5	May 2026	Widget approach confirmed (DIY JSON), Data Lake Inspector built.
2.4	May 2026	Architecture overhaul — three-prong system, Foxglove, server separation.
2.3	May 2026	Clean dev environment — BlueOS VM + internal SITL + Cockpit working.
2.2	May 2026	Dev environment attempt — standalone Docker SITL stack (superseded).
2.1	May 2026	Server dev environment plan, shopping list update.
2.0	May 2026	Workspace, build pipeline, failsafe design, UI design.
1.0	April 2026	Initial handover document.

---

Changelog

Moved here from CLAUDE.md's SESSION LOG (11 Jul 2026 restructure). This is the granular, session-by-session record that bridges Claude Code (VS Code) sessions and the claude.ai chat — finer-grained than the Version History table above, which narrates per-doc-version. Append-only; correct forward with a new dated entry, never edit past ones.

Claude Code appends an entry here at the end of any session where one of these happened — not for routine edits, refactors, or in-progress work: a design/architecture decision was made or changed; a hardware fact was confirmed or changed; a security-relevant change was made; a backup/recovery-relevant change was made; a bug was root-caused and fixed (not just patched); a previously-deferred item was completed or its trigger condition changed; anything Grant would reasonably expect this chat to already know next time he opens it. Skip typo fixes, comment-only changes, WIP work, or anything already fully captured by the git commit message with no extra context needed. Full trigger-rule text lives in CLAUDE.md.

• 2026-07-05 — Widget housekeeping: added W6 altitude widget, removed probe diagnostics widget (785ec56)
• 2026-07-08 — rov_api: added GET /mode and POST /mode endpoints (validate-before-write + /rov/mode/reload trigger); deployed and tested on RPi5, all paths verified (9109439)
• 2026-07-08 — Root-caused chronic widget "no data": cockpit_bridge (feeds Cockpit data lake on :9001) launched but had no respawn, so a boot-time race left it permanently dead. Added respawn=True to its launch Node. W8 mode indicator built + validated live (render+data). W1/W8 confirmed GREEN/AUV once bridge up.
• 2026-07-08 — FLAGGED (open): recording_manager.py still implements superseded DIR-9 (continuous-from-boot recording), NOT the governing DIR-9a (ARM-bounded black-box: AUV ARM-start/DISARM-stop, ROV operator manual + arm reminder). Design decision exists only in handover + CLAUDE.md; code never updated. foxglove_mcap.launch.py comments also still say DIR-9. Also: argonaut-stack.sh passes record:=${RECORD} but rov_full.launch.py declares no such arg, so RECORD=false is silently ignored and bags record regardless. Both deferred to a dedicated DIR-9a recorder-rework session — do NOT wire RECORD into the current DIR-9 model.
• 2026-07-08 — Added respawn=True/respawn_delay=5.0 to failsafe_monitor, motion_controller, ekf_filter_node, depth_node, mission_executor. Deliberately NOT applied to: mavros_node (respawn could mask/worsen the chronic FCU-link instability — belongs in that investigation), recording_manager (pending DIR-9a rework), camera_node/feature_detector (perception not operational yet), foxglove_bridge (viz-only). NOTE: failsafe_monitor respawn has a coverage gap during the 5s respawn delay — no failsafe assessment in that window; flagged for future design consideration.
• 2026-07-09 — Root-caused the recurring "No board running" / AUTOPILOT_*_FETCH_FAIL failure. NOT the previously-documented TCP-socket/UI-polling cause. Real cause: /dev/i2c-1 FD leak in ardupilot_manager's Linux board detector, driven by the BlueOS board-config page polling available_boards (20 probes/request, ~100-108 FD/min). Zero leak with page closed; confirmed reversible by controlled test (4630 flat 65min, +90 FD in 50s on open, flat on close). nofile=65536 only delayed failure to ~11h cumulative page-open time. Upstream BlueOS 1.4.3 bug — not patching. Corrected the stale root cause in this file.
• 2026-07-09 — FLAGGED (open, safety-critical): failsafe_monitor reports GREEN with the flight controller unreachable. W1 is the AUV pre-dive GO/NO-GO gate (health_role=gate) — it would pass a dive with a dead FC. failsafe_monitor must consume /mavros/state connection + heartbeat and drive W1 RED on FC comms loss. W1 status line should carry cause text (e.g. "Vehicle Disconnected", "Heartbeat Lost", "No Board Running"). Also flagged: no FD-count check on ardupilot_manager in the pre-dive gate.
• 2026-07-11 — RESOLVED the 2026-07-09 flagged safety-critical fault: failsafe_monitor now drives W1 RED on FC comms loss. Four commits: (1) 86b2728 — comms loss forces unconditional RED (was scored as one marginal parameter, only reached AMBER) plus fcu_ever_connected latch so the startup window before any heartbeat is not reported GREEN; cause text "Heartbeat Lost"/"Vehicle Disconnected" added to FailsafeStatus.message per DIR-7. (2) 1b0050d — FSM recovery fix: RETURN_TO_SAFE was a terminal trap (only HOLD_AND_RECOVER could return to NORMAL); now both recover when conditions clear. (3) dbe9515 — cockpit_bridge publishes new data lake variable rov-failsafe-cause (substring of message before first "|"; empty when nominal). (4) 3409ae5 — W1 shows state word on #w1-message (state-driven, no flicker) and cause text on #w1-footer, footer restyled red 11px, blank when nominal. All four verified live end-to-end: kill mavros -> RED + "Heartbeat Lost", restart -> recover to GREEN. Also corrected: cockpit_bridge was publishing failsafe_state (FSM, can be 3+) not assessment_state (0/1/2) to rov-failsafe — fixed in dbe9515's sibling commit d854641; this was the real cause of the day's "widgets show no data" (value 3 out of W1's range), compounded by a restart-storm wedging 9001.
• 2026-07-11 — Doc restructure (this pass): consolidated current-state into this document, moved this log here from CLAUDE.md, removed the embedded DIR copy in favour of a pointer to docs/design_intent_register.md, moved all project docs into the repo docs/ folder. See Version History v2.23 above for detail.
• 2026-07-13 — Found mode_profile_loader (rov_mission) was built and committed 7 Jul in 4e48dc2 but never added to rov_full.launch.py, so /rov/mode/profile had zero publishers at runtime since then — the whole operating-mode profile system had been dark, and failsafe_monitor's health_role had been silently running on its gate default the whole time. Fixed by adding the Node to the launch file (with the other core nodes, before Perception) with respawn=True/respawn_delay=5.0, matching the other core nodes (c099c00, a5e1c99). Verified live: publisher count 0 -> 1 (464e17e).
• 2026-07-13 — failsafe_monitor made mode-aware (2a3e577, same session): subscribes to the resolved /rov/mode/profile and branches condition-clear recovery on health_role per the DIR-7 addendum — gate (AUV) commits HOLD_AND_RECOVER -> RETURN_TO_SAFE instead of returning to NORMAL; advisory (ROV) holds for an explicit operator resume (placeholder topic). This surfaced a new gap once the mode system was actually live (previous entry): gate-mode RETURN_TO_SAFE had no exit, a terminal trap, because that branch deliberately ignores further condition-clear so committed recovery runs to completion, and nothing signalled completion. Fixed (bf815cc) with a new subscription, /rov/nav/safe_zone_reached (std_msgs/Bool) — THIS TOPIC AND SUBSCRIPTION ARE THE PERMANENT nav-arrival interface; today's publisher is temporary bench scaffolding only. On gate + RETURN_TO_SAFE + True, transitions to NORMAL with cause text "Safe zone reached — recovery complete" (DIR-7 cause MUST, via the existing message-prefix mechanism, no FailsafeStatus.msg change). Verified live end-to-end on the bench: published the event by hand, watched the FSM release RETURN_TO_SAFE -> NORMAL and the motion_controller recovery broadcast stop. Real navigation-raised arrival event and a temporary W1-contextual operator button (rov_api- backed bench scaffolding, next step) are both still to come — see new Parked Design Item, docs/design_intent_register.md.
• 2026-07-13 — FLAGGED (open, pre-existing, found not caused by this session's work): (1) flag_manual_abort in failsafe_monitor is a one-way latch, never reset to False anywhere — once any manual abort fires, the priority-6 handler returns early every subsequent cycle for the rest of the process lifetime, permanently shadowing all lower-priority handling including this session's new mode-aware recovery and arrival-event logic. Needs a reset condition (candidates: on reaching NORMAL, on explicit operator reset, or process-restart only — undecided, needs its own session). Field-deployment concern: one manual abort disables all future automatic recovery for the rest of that dive. (2) failsafe_monitor produced no log output at all in journalctl this session — state had to be read directly off /rov/failsafe every time. Should log state transitions and cause. Minor but cost real diagnostic time this session.

---

Design Intent Register — pointer

The full Design Intent Register (DIR-1 through DIR-12) lives in docs/design_intent_register.md, git-tracked in the repo, not duplicated here. That file is the source of truth for why — read it before any design/architecture decision.

DIRs governing current work, for quick orientation:

• DIR-1/DIR-2 — tether/HILT link (not power source) determines AUV vs ROV mode; HILT is an override channel, not a confirmation gate.
• DIR-7/DIR-12 — health has two distinct subjects (vehicle vs autonomy stack); AUV health_role: gate vs ROV advisory; health assessment must include FC connectivity and must carry cause text. (Implemented 11 Jul 2026 — see Changelog above.)
• DIR-9a (governing, supersedes DIR-9) — recording is a single ARM-bounded black-box tier, keyed to the recording_control flag, not the mode name. recording_manager.py still implements the superseded DIR-9 model — known drift, open.
• DIR-5/5a/5b — return-to-safe is mission-bounded, vehicle-layer, and mode-dependent (AUV automatic, ROV alert+override); operator override continues the mission, it does not pause it.

---

1. Project Overview

Goal

A fully autonomous underwater inspection system targeting hull and jacket surveys with no human operator during field operations. Built on a shoestring budget using commodity hardware, designed for field deployment from day one.

Target Structures

• Jackets: Fixed offshore structures — predictable geometry
• Hulls: In-port vessels — moored, requires online re-localisation

Key Design Values

• Cost-conscious — use what is available, upgrade when a dead end is reached
• Field-operational from day one — same codebase, different configs per environment
• Full autonomy — no pilot required during operation
• Cockpit as access hub — Argonaut 3 widgets only, not a container for everything
• QGroundControl retained as dev/debug only

---

2. Platform

Vehicle: BlueROV2 Heavy (Blue Robotics) Operating mode: Vehicle-side compute only Tether: Single Ethernet cable — all topside/bottomside communication over one cable

---

3. Hardware Architecture

On the ROV

Board	Role	OS
Raspberry Pi 4 (8GB)	BlueOS gateway — MAVLink routing, Pixhawk FC host	BlueOS 1.4.3 (armv7)
Raspberry Pi 5 (8GB)	Autonomy brain — ROS2 stack, foxglove_bridge, MCAP recorder	Ubuntu 24.04 LTS + ROS2 Jazzy

RPi4: BlueOS 1.4.3 running. Static IP 192.168.1.100. Tailscale 100.100.119.125 (node renamed blueos-1). Pixhawk 2.4.8 active on /dev/ttyACM0 — SITL is NOT in production use. BlueOS preferred_board: Pixhawk1. Docker nofile ulimit = 65536 (ceiling raise only — NOT a fix; see §15a, ardupilot-manager /dev/i2c-1 FD leak is an OPEN bug). Pirate Mode enabled.

RPi5: Ubuntu 24.04.4, boots from SD card (/dev/mmcblk0). NVMe (/data) = data only (ros2_ws, bags, logs — NOT the OS). Static IP 192.168.1.101. Tailscale installed but STOPPED and DISABLED (re-enable: sudo systemctl enable --now tailscaled). ROS2 Jazzy, Docker, MAVROS installed. argonaut.service running (manual-start in dev). Weak SSH password — harden before field deployment.

Sensors

Sensor	Status
Xsens MTI-30-2A5G4 IMU	Have unit — CA-USB-MTI cable on order
IP cameras (2+)	Have units — not yet integrated
Ping2 Sonar (downward)	Phase 3 purchase (~$280 USD) — required for W6 full test
Ping2 Sonar (forward)	Phase 4 purchase
Ping360 Scanning Sonar	Phase 4-5 purchase
DVL (Doppler Velocity Log)	Phase 7 — critical for accurate X/Y positioning
Tritech Gemini 720im	Phase 6+

DVL is the most important missing sensor. Without it, X/Y position is dead-reckoned from IMU only and drifts over time. Z axis (Bar30) is accurate. All position displays must reflect this uncertainty honestly.

Power

Device	PSU	Status
RPi4	5.1V / 3A RPi4 official PSU	In use
RPi5	5V / 5A 27W USB-C official RPi5 PSU	In use

Storage

Device	Storage	Notes
RPi4	Fresh microSD (Patriot 64GB or equivalent)	BlueOS OS drive
RPi5 (OS)	SD card (/dev/mmcblk0)	Ubuntu 24.04, hardened (log2ram, tmpfs, journald capped)
RPi5 (Data)	RPi M.2 HAT+ + RPi SSD 512GB (2230 NVMe) at /data	ros2_ws, bags, logs

---

4. Software Stack

RPi4 (BlueOS Gateway)

• BlueOS 1.4.3 (armv7) — running, static IP 192.168.1.100
• Tailscale 100.100.119.125 (blueos-1)
• Pixhawk 2.4.8 — active flight controller on /dev/ttyACM0:115200
• MAVLinkRouter — routes Pixhawk to MAVROS on RPi5 and to Edge PC for QGC debug. MAVP2P has known issues (BlueOS own warning) — do not use.
• /etc/docker/daemon.json — nofile: 65536 (ceiling raise only, NOT a fix — see §15a)
• Pirate Mode — enabled

RPi5 (Autonomy Brain)

• Ubuntu Server 24.04.4 LTS (Noble, arm64) — boots from SD, NVMe = /data
• ROS2 Jazzy Jalisco — auto-sourced in .bashrc
• MAVROS + GeographicLib datasets
• Docker 29.5.3 — runs without sudo
• Argonaut 3 autonomy stack (7 packages) — deployed, running as argonaut.service
• foxglove_bridge — confirmed running on :8765
• recording_manager — confirmed active, bags to /data/bags/
• cockpit_bridge — WebSocket :9001, 8 variables
• Tailscale — installed, service stopped and disabled
• Xsens MTi ROS2 driver — pending CA-USB-MTI cable

Edge PC / Field Laptop BB8 (Topside)

• Cockpit native desktop app v1.18.2 — primary GCS, Argonaut 3 widgets
• Foxglove Studio — live monitoring (Prong 2) and post-dive replay (Prong 3)
• Mission Planner (Phase 6+) — standalone 3D web app, locally installed
• Local model + mission cache — glTF twins and YAMLs pre-synced, offline capable

Cockpit native app is required. The BlueOS browser extension version is "Lite" and does not support DIY widgets.

---

5. System Architecture — Three Zones

ONSHORE                    OFFSHORE TOPSIDE              OFFSHORE BOTTOMSIDE
------------------         ----------------------        -----------------------------
Dev Server                 Edge PC / Field Laptop        BlueROV2 Heavy
  Harbor registry            Cockpit native v1.18.2        RPi4 (BlueOS)
  Gitea source control       Foxglove Studio                 MAVLink Router
  ARM64 build pipeline*      Mission Planner*                Pixhawk 2.4.8 (ArduSub)
                             Local cache                 RPi5 (ROS2 Jazzy)
Ops Server (separate)                                      Autonomy stack (7 packages)
  Digital twin storage     <-- Ethernet Tether -->         foxglove_bridge :8765
  Mission archive                                           recording_manager
  Post-dive data                                            cockpit_bridge :9001
  Model conversion                                        Sensors
  Tailscale endpoint                                        Xsens, Bar30, cameras, sonar

*ARM64 builds: laptop only (server overheats under compilation load) *Mission Planner: Phase 6+ standalone app

Ethernet Tether — Single Cable

Flow	Direction
BlueOS web UI :80	Tether to Edge PC browser
MAVLink UDP :14550	Tether to QGC on Edge PC (debug)
foxglove WebSocket :8765	Tether to Foxglove Studio on Edge PC
cockpit_bridge WebSocket :9001	Tether to Cockpit on Edge PC
Mission YAML upload	Edge PC to RPi5 via SSH/SCP
MCAP sync post-dive	RPi5 to Edge PC via SSH/SCP
Container deploy	Edge PC to RPi5 via SSH

---

6. Three-Prong Mission System

Prong 1 — Mission Creation + Editing

• glTF model synced to Edge PC local cache pre-dive
• Mission Planner on Edge PC: Three.js 3D scene with structure twin
• Hull mode: parametric lawnmower path generator
• Jacket mode: 3D waypoint editor
• Output: Mission YAML (rich format, structure-relative 3D coordinates)
• Status: Phase 6+ — standalone web app, not in Cockpit

Prong 2 — Mission Execution + Monitoring

• rov_mission on RPi5 executes YAML via state machine
• MAVROS translates ROS2 setpoints to MAVLink to RPi4 to ArduSub to thrusters
• All ROS2 topics streamed via foxglove_bridge WebSocket to Foxglove Studio
• Cockpit widgets on Edge PC: Health / Mission Status / Return To Safe
• Action path: Cockpit widget fetch() -> FastAPI on RPi5 -> /rov/mission/abort

Prong 3 — Data Collection + Replay

• MCAP recorder logs ALL ROS2 topics to NVMe SSD during dive
• Post-dive: MCAP transferred to Edge PC via SCP over tether
• Foxglove Studio on Edge PC: immediate replay, no server needed
• When server reachable: MCAP archived to Ops Server via Tailscale

---

7. Digital Twin

Format	Origin	Status
EBM / EBMX	CADMATIC — hull model	Have file
NWD	Autodesk Navisworks — platform models	Have files (x2 platforms)
IFC	Open BIM standard — intermediate	Conversion target
glTF / GLB	Khronos Group — web render target	Final format for all tools

Conversion pipeline: EBM / NWD -> CAD Exchanger SDK -> IFC -> glTF / GLB Currently manual. Automated pipeline is a later infrastructure task.

---

8. ROS2 Workspace

Location on RPi5: /data/ros2_ws/src/rov-autonomy/ Gitea repo: administrator/rov-autonomy Gitea SSH remote: ssh://git@symbytech:222/administrator/rov-autonomy.git (hostname form — resolves via hosts entries on laptop and RPi5)

Package Structure

Package	Purpose	Key nodes
rov_interfaces	Custom messages and services	— (msgs/srvs only)
rov_navigation	State estimation, EKF, depth	state_estimator, depth_node
rov_perception	Camera nodes, feature detection	camera_node, feature_detector
rov_control	Motion controller, failsafe monitor	motion_controller, failsafe_monitor
rov_mission	Mission executor, cockpit bridge, recorder	mission_executor, cockpit_bridge, recording_manager
rov_bringup	Top-level launch files	— (launch files only)
rov_api	FastAPI backend — HTTP→ROS2 bridge on :8081	api_node
rov_simulation	Simulation nodes (dev only)	Do NOT deploy in production launch files

Key Topics

Topic	Type	Publisher
/imu/data	sensor_msgs/Imu	Xsens driver
/mavros/imu/data	sensor_msgs/Imu	MAVROS
/odometry/filtered	nav_msgs/Odometry	EKF node
/rov/failsafe	rov_interfaces/FailsafeStatus	failsafe_monitor
/rov/mission/status	rov_interfaces/MissionStatus	mission_executor
/rov/recording/active	std_msgs/Bool	recording_manager
/rov/recording/bag_dir	std_msgs/String	recording_manager
/ping2/altitude	sensor_msgs/Range	ping2 driver (hardware-gated)

Key Commits

Hash	Description
0640353	W6: Add rov-altitude to cockpit_bridge for Ping2 downward sonar
5d75a9d	Phase 3 Stage 1: Recording Manager (DIR-9)
2cb7b74	W0 layout fix (3-column, horizontal scroll)
66f9bd7	W0–W5 all confirmed working, committed
12b5161	cockpit_bridge node — ROS2→WebSocket :9001 data-lake injection
c94412d	Full stack zero-crash launch
45b3301	Fix full-stack launch: fcu_url, plugin_allowlist, QoS, battery guard
7590faf	Missing setup.cfg in all 5 Python packages
f1c6710	Missing config/ dir in rov_bringup

Build/Deploy Procedure

# On RPi5 — build (symlink-install works correctly with setup.cfg files in place)
cd /data/ros2_ws
colcon build --packages-select <pkg> --symlink-install
source /data/ros2_ws/install/setup.bash

---

9. UI Architecture — Argonaut 3 / Cockpit

Cockpit Setup

App: Cockpit native desktop v1.18.2 — installed on field laptop BB8 (Windows) Vehicle connection: Enter RPi4 IP (192.168.1.100) on first launch Generic WebSocket: ws://192.168.1.101:9001 (cockpit_bridge on RPi5)

Cockpit native desktop app is mandatory for DIY widget work. The BlueOS browser extension version is "Lite" and does NOT support DIY widgets. Generic WebSocket data injection requires v1.18.0 minimum.

Widget Approach — DIY Widgets (Confirmed)

Widgets are Cockpit DIY widgets: plain HTML/CSS/JS stored as JSON files and imported via the gear icon on a placed DIY widget.

Import format: JSON file with three string fields:

{
  "html": "...",
  "css":  "...",
  "js":   "...",
  "inheritCockpitStyles": false
}

CSS rules for DIY widgets (hard-won — do not break these):

• No body {} or :root {} blocks — bleed into Cockpit's own styles and crash the renderer
• No @keyframes animations — cause renderer instability
• No unicode characters in JS strings — use plain ASCII equivalents
• No position: fixed overlays — use display: block/none toggling instead
• No async/await — use .then()/.catch() for fetch calls
• Use element ID/class selectors only — no global tag selectors (e.g. div {})

Data Injection Path — Confirmed Production Path

cockpit_bridge (ROS2 node on RPi5, rov_mission pkg)
  WebSocket server :9001
    <-- Cockpit connects as client
      --> Sends 'variable-name=value' messages at 2Hz
        --> Data lake populated as external/{variable-name}
          --> DIY widget reads via getDataLakeVariableData('external/{variable-name}')

Confirmed data lake read method: window.cockpit.getDataLakeVariableData(variableId) Does NOT exist: getDataLakeValue(), listenToDatalakeVariable() (not in DIY widget scope) Metadata only: getAllDataLakeVariablesInfo() — enumeration only, not value reads

Data Lake Variables

Full current list (19 total: 8 telemetry + 11 mode-profile flags) is in §0 QUICK STATUS BOARD — not duplicated here to avoid the two tables drifting apart, which is exactly what happened between v2.21 (8 vars, this table) and v2.22 (19 vars, §0's table) before this restructure.

Widget File Locations

Gitea repo: rov-autonomy/widgets/ Laptop local: C:\Users\grant\Desktop\Autonomous ROV System\rov-autonomy\widgets\

Widget Build Status

#	File	Widget	Test Status	Variable
0	w0_data_lake_inspector.json	Data Lake Inspector	Confirmed working (706 vars live)	None — reads all
1	w1_system_health_indicator.json	System Health Indicator	Visual confirmed — full test pending	external/rov-failsafe
2	w2_mission_status.json	Mission Status	Visual confirmed — untested (needs mission running)	external/rov-ms, external/rov-mp
3	w3_abort_button.json	Return To Safe Button	UI confirmed — label rename pending ("ABORT MISSION" → "RETURN TO SAFE"); backend not built	FastAPI /abort POST
4	w4_mission_setup_button.json	Mission Setup Button	Visual confirmed — untested (needs FastAPI backend)	None — opens URL
5	w5_battery_return_budget.json	Return Budget	Visual confirmed — untested (rov-return-budget not published)	external/rov-return-budget
6	w6_altitude.json	Altitude	NO SONAR confirmed — hardware-gated for full test	external/rov-altitude
7	w7_backup_due_notice.json	Backup Due Notice	Not built — depends on FastAPI backend	FastAPI /backup/status

Testing note: W0 is the only widget with confirmed live data end-to-end. W1–W5 display correctly but have not been tested with actual operational data. Full testing requires a live mission, FastAPI backend, and/or Ping2 hardware depending on the widget. W6 full test requires Ping2 sonar hardware. W7 not started.

How to Install a Widget

1. Open Cockpit native desktop app
2. Connect to vehicle (192.168.1.100)
3. Enable Pirate Mode (Settings → General → Enable Pirate Mode) — resets each session
4. Enter edit mode (pencil icon, top right)
5. Click Add Widget at the bottom — scroll right to find the </> DIY widget
6. Drag it onto the main view area and resize
7. Click the gear icon on the placed widget
8. Click Import — select the .json file from the widgets folder
9. Widget loads immediately

---

10. Failsafe Design Summary

Full design in docs/failsafe_design.md.

Three States

State	Meaning
GREEN	All systems nominal
AMBER	One or more parameters degraded — mission continues with reduced confidence
RED	Critical failure — safe action triggered automatically

Trigger Conditions

Trigger	Threshold	State
Comms loss	>2s (configurable)	RED → SURFACE
Battery low	<15% above return budget	AMBER
Battery critical	<5% above return budget	RED → SURFACE
Depth exceeded	Configurable per site	RED → SURFACE
IMU fault	Data gap >500ms	AMBER
DVL fault (Phase 7)	Data gap >1s	AMBER

---

11. Infrastructure — SymbyTech Server

Item	Value
Hardware	Dell PowerEdge R210
OS	Ubuntu Server 24.04 LTS
Static IP	192.168.1.175 (on eno2 — NOT eno1)
RAID Mount	/mnt/raid-data/
Tailscale IP	100.104.236.104
CPU cores	4 (host)

WARNING: eno2 is the primary NIC, not eno1. Never change this in netplan.

Gitea

Item	Value
Public URL	https://git.symbytech.com
SSH remote	ssh://git@192.168.1.175:222/administrator/REPO.git
ROV repo	administrator/rov-autonomy
Data location	/mnt/raid-data/forgejo/

Use the server IP directly (192.168.1.175:222) — the hostname symbytech may not resolve on all networks. Do NOT use git.symbytech.com for SSH port 222.

Harbor Registry

Item	Value
Public URL	https://registry.symbytech.com
Local URL	http://192.168.1.175:8090
Storage	/mnt/raid-data/harbor/

Backup System (built 30 Jun 2026)

Push model — each Pi self-backs-up via root cron, pushing to server over SSH.

Device	Script	Schedule	Log
RPi4	/usr/local/bin/rov-brain-pi4-backup.sh	00:00 daily	/var/log/rov-brain-pi4-backup.log
RPi5	/usr/local/bin/rov-brain-pi5-backup.sh	00:30 daily (staggered)	/var/log/rov-brain-pi5-backup.log

Server monitor (/usr/local/bin/symbytech-monitor.sh → check_rov_backup) reads */backup-status.txt under /mnt/raid-data/backup/. Alerts: FAILED (always), OVERDUE ≥2 days (throttled to once/7 days per device, regardless of online/offline).

RPi5 backup images /dev/mmcblk0 (SD card OS) and rsyncs /data (NVMe data). RPi5 boots from SD — /dev/mmcblk0 is the OS drive. NVMe is data only.

Development VM — DECOMMISSIONED

The BlueOS KVM VM (blueos-vm, NAT IP 192.168.122.89, Tailscale 100.84.141.120) was used during early development before real RPi4/RPi5 hardware was operational. It is no longer active (virsh list --all returns empty). The NPM proxy rule arov.symbytech.com → 192.168.122.89:80 is a leftover dead rule with no active target. All [VM] commands in this document are historical only — do not use.

---

12. Phase Task Status

Phase 1 — Foundation

Task	Status
Flash BlueOS onto RPi4	DONE — BlueOS 1.4.3, static IP 192.168.1.100
Install Ubuntu 24.04 on RPi5	DONE — Ubuntu 24.04.4, hostname rov-brain, IP 192.168.1.101
Install ROS2 Jazzy on RPi5	DONE — Jazzy Jalisco, auto-sourced in .bashrc
Confirm network comms between Pis	DONE — MAVROS /mavros/state connected:true against Pixhawk
Install Docker on RPi5	DONE — Docker 29.5.3, no sudo required
Install MAVROS	DONE — running in argonaut.service
Install Tailscale on RPi5	DONE — installed; service stopped and disabled
Install Tailscale on RPi4	DONE — Tailscale IP 100.100.119.125
Harbor project and user setup	DONE
Connect Xsens IMU	PENDING — CA-USB-MTI cable on order
foxglove_bridge install on RPi5	DONE — ros-jazzy-foxglove-bridge, :8765 confirmed
Create /data/bags on NVMe	DONE — recording live ~2.5 MB/s
Deploy ROS2 workspace from Gitea	DONE — all 7 packages built, argonaut.service running
Automated backup system	DONE — both Pis → server, monitoring active
BlueOS FD exhaustion fix	✗ NOT DONE — root cause was wrong. nofile=65536 is a ceiling raise. Real cause: /dev/i2c-1 FD leak in ardupilot_manager board detector. OPEN — see §15a
Pixhawk 2.4.8 as active FC	DONE — confirmed ttyACM0, MAVROS connected:true

Phase 2 — Widgets and Data Path

Task	Status
Confirm widget approach	DONE — Cockpit DIY widgets, JSON format
Confirm Cockpit version required	DONE — native desktop v1.18.2
Establish CSS rules for DIY widgets	DONE
W0 Data Lake Inspector	DONE — confirmed working (706 vars live)
W1 System Health Indicator	DONE — visual confirmed
W2 Mission Status	DONE — visual confirmed; full test pending
W3 Return To Safe Button	DONE (UI); label rename pending; backend not built
W4 Mission Setup Button	DONE — visual confirmed; full test pending
W5 Return Budget	DONE — visual confirmed; full test pending
W6 Altitude Widget	DONE — NO SONAR state confirmed (hardware-gated)
W7 Backup Due Notice	NOT STARTED — depends on FastAPI backend
All W0–W6 committed to Gitea	W0–W5: committed. W6 widget JSON: pending re-import from laptop
cockpit_bridge node	DONE — 8 variables, commits 5d75a9d + 0640353
Probe widget	DELETED — dev tool only, not part of W0–W7

Phase 3 — Recording and Mission Control (Active)

Task	Status
Recording Manager (DIR-9)	DONE — Stage 1 complete, commit 5d75a9d
FastAPI backend core (RPi5:8081)	DONE — Stage 2 core, commit 66a18ef, verified end-to-end
Deploy files version-controlled	DONE — deploy/ folder, commit 97decac
Hostname/DNS resolution fix	DONE — laptop + RPi5 hosts entries, git remotes on hostname
Widget housekeeping (W6 add, probe remove)	DONE — commit 785ec56
W3 backend /abort POST end-to-end	DONE — verified: FSM NORMAL→HOLD_AND_RECOVER
W3 label update to "RETURN TO SAFE"	PENDING — widget code + FASTAPI_HOST constant
W5 return budget publishing	DEFERRED — Stage 2b (return budget approach TBD)
W7 Backup Due Notice + /backup/*	DEFERRED — Stage 2b (Option A design)
/mission/upload endpoint	DEFERRED — Stage 2b
Ping2 sonar purchase (W6 full test)	PENDING — Phase 3, ~$280 USD

Next Priorities

See §0 QUICK STATUS BOARD — NEXT (moved there 11 Jul 2026 restructure to avoid two lists drifting apart).

---

13. Session Lessons Learned — Data Injection Investigation (Task 11)

This section documents what was learned during the Task 11 data injection investigation so it is not repeated. These are permanent historical records.

What was attempted and why it failed

Approach	Why it failed
UDP to VM port 14550 from server host	KVM NAT silently drops packets
UDP to VM port 14001 from server	Same KVM NAT issue
HTTP POST to mavlink2rest /mavlink	Stored in REST, not broadcast via WebSocket — not in data lake
UDP with sysid=255, compid=190	mavlink2rest only processes sysid 1 messages for data lake
UDP with sysid=1, compid=1 from Python	Caused ArduSub to detect heartbeat conflict — do not impersonate compid=1

What works

Approach	Result
Generic WebSocket (Cockpit v1.18.0+)	Confirmed working. WebSocket server on RPi5, Cockpit connects as client. Variables appear in data lake as external/{name} immediately. This is the correct and supported path.
Native ArduSub NAMED_VALUE messages	Confirmed working via W0 — flow: Pixhawk → ttyACM0 → mavlink-routerd → mavlink2rest UDP → mavlink2rest REST + WebSocket → Cockpit.

Key architectural insight

The ONLY path that gets external data into the Cockpit data lake is:

1. Messages from the Pixhawk via ttyACM0 → mavlink-routerd → mavlink2rest — for production ArduSub/MAVROS data
2. Generic WebSocket connections (Cockpit v1.18.0+) — for custom external data via cockpit_bridge

---

14. Roadmap Summary

Phase	Title	Status
1	Foundation	COMPLETE
2	Widgets and Data Path	COMPLETE (W6 deployed; W7 not started; backend pending)
3	Recording and Mission Control	IN PROGRESS — Stage 1 complete
4	Surface Following and Pattern Flight	Not started
5	Perception and Feature Detection	Not started
6	Full Autonomy + Mission Planner	Not started
7	Field Trials and Hardware Evolution	DVL integration
8	Commercial Product	Not started

---

15. Key Architecture Decisions (Locked)

Decision	Choice	Rationale
Autonomy compute	Vehicle-side RPi5	Lowest latency to sensors and thrusters
RPi4 role	Thin gateway only	Separation of concerns
OS on RPi5	Ubuntu 24.04 LTS	Only clean ROS2 Jazzy support
RPi5 storage	OS-on-SD, data-on-NVMe	Physical separation; SD hardened for low writes
ROS2 version	Jazzy Jalisco	Current LTS
Container registry	Harbor (self-hosted)	No rate limits, private
Cockpit version	Native desktop v1.18.2	Browser extension is Lite; v1.18.2 required for DIY widgets
Widget approach	Cockpit DIY widgets (JSON format)	Confirmed working, version-controlled in Gitea
MAVLink router	MAVLinkRouter	MAVP2P has known issues per BlueOS own warning
Flight controller	Pixhawk 2.4.8 (real hardware)	SITL retired from production
Data injection	Generic WebSocket via cockpit_bridge	Confirmed working; MAVLink injection paths do not populate data lake
WebSocket variable naming	Use hyphens not underscores	Cockpit normalises underscores to hyphens in data lake IDs
Recording format	MCAP (Foxglove native)	Seekable replay, no conversion needed
Recording control	recording_manager ROS2 node	DIR-9a (governing) — single ARM-bounded black-box tier, keyed to recording_control flag, not mission state. recording_manager.py implementation still lags on superseded DIR-9 (continuous-from-boot) — known drift, open.
foxglove_bridge role	Visualisation ONLY	Does NOT record MCAP (DIR-9a)
ARM64 builds	Laptop only	Server overheats under compilation load
Acoustic positioning	Excluded	Unreliable near steel
DVL	Deferred to Phase 7	Budget constraint — highest priority future sensor

---

15a. ardupilot-manager FD Exhaustion — OPEN BUG (root cause corrected 9 Jul 2026)

⚠ This section replaces the previous "RESOLVED — Permanent Fix Applied" entry, which was wrong on both diagnosis and fix. The stale root cause survived from v2.10 to v2.21 and actively prevented this being solved, because it made every recurrence look inexplicable.

What the old entry claimed (WRONG — do not reinstate)

• ✗ "Root cause: accumulated TCP socket FDs from continuous UI polling exhausting nofile=1024."
• ✗ "Permanently fixed via /etc/docker/daemon.json nofile: 65536."
• ✗ (v2.10) "re-diagnosed as uvicorn FD exhaustion."

None of these are correct. The leaked FDs are not sockets. nofile: 65536 fixed nothing — it raised the ceiling, converting a failure at ~10 minutes into a failure at ~11 hours, which disguised a hard leak as an intermittent annoyance.

Actual root cause (measured 9 Jul 2026)

ardupilot_manager leaks /dev/i2c-1 file descriptors, one per failed Linux-board probe.

Mechanism, end to end:

1. The BlueOS Autopilot Firmware / board-config page polls GET /v1.0/available_boards.
2. Each request triggers 20 flight_controller_detector.linux.detector:detect_boards probes — for Navigator, NavigatorPi4, NavigatorPi5, and Argonot.
3. Each Linux-board probe opens /dev/i2c-1 and never closes it when detection fails.
4. We run a Pixhawk 2.4.8 on USB serial (PlatformType.Serial), not a Navigator. Every probe fails. Every probe leaks.
5. At 65536 FDs, ardupilot_manager can no longer open new handles. Everything requiring a fresh handle fails: AUTOPILOT_VEHICLE_TYPE_FETCH_FAIL, AUTOPILOT_ENDPOINT_FETCH_FAIL, BAG_OF_HOLDING_GET_DATA_FAIL, "No board running".
6. Restarting the core recreates the blueos-core container with a fresh FD table — which is why a core restart has always appeared to "fix" it.

Evidence (controlled measurement, RPi4, 9 Jul 2026)

Time	Board-config page	FD count on PID 122 (ardupilot_manager/main.py)
07:51 → 08:56 (65 min)	CLOSED	4630 → 4630 — absolutely flat
09:03:35 → 09:04:24 (50 s)	OPEN	4630 → 4720 — +90 FDs (~108/min)
09:04:24 → 09:04:47	CLOSED	4720 → 4720 — flat immediately

FD composition at 2740 total: 2740 × /dev/i2c-1, 3 sockets, 2 pipes. Not a socket leak. Probe-to-request ratio confirmed from logs: 8920 detect_boards ÷ 445 available_boards = 20:1.

Cause and effect: isolated, reproducible, reversible.

Critical property — the count does NOT reset at launch

FD count accumulates across sessions and only resets when blueos-core is recreated (core restart / reboot). Cumulative page-open time, not wall-clock time, drives time-to-failure (~11 h at ~100 FD/min). Leave that page open two hours a day and the wall is hit inside a week without a single long session.

Why this matters for field operations

The exposure is not leakage during a dive (topside is disconnected; nothing polls). The exposure is launching with a nearly-exhausted FD table after pre-dive setup — precisely when the board-config page is most likely to have been open for hours. Mid-dive, with no operator, ardupilot_manager then fails on the next handle it needs.

Mitigation

Dev (effective immediately, zero code): Do not leave the BlueOS Autopilot Firmware / board-config page open. Open it, do the job, close it. Other BlueOS pages and Cockpit are unaffected and do not leak.

Field — NOT YET IMPLEMENTED (both are open items):

• (a) FD-count check on ardupilot_manager as part of the AUV pre-dive gate, blocking launch above a threshold. Operator discipline is not a field-safe control.
• (b) failsafe_monitor must detect FC comms loss so W1 goes RED when the board is unreachable — see §15d.

Do NOT patch BlueOS or ArduPilot. Any upstream update reverts local changes. This is an upstream BlueOS 1.4.3 defect; the correct route is an upstream report. All countermeasures must live on our side of the boundary.

The nofile: 65536 setting in /etc/docker/daemon.json remains in place and is harmless — but it is a ceiling raise, not a fix, and must never again be described as one.

---

15d. Vehicle health in the failsafe assessment — RESOLVED 11 Jul 2026

Previously titled "Vehicle health is NOT in the failsafe assessment — OPEN, SAFETY-CRITICAL." Discovered 9 Jul 2026 (below), fixed 11 Jul 2026 via four commits. Corrected root cause of a second, related fault (cockpit_bridge publishing the wrong variable) found and fixed in the same session — see the last commit below.

Original fault (9 Jul 2026)

Discovered while the flight controller was unreachable ("No board running"):

W1 showed GREEN / "Systems nominal".

Every widget indicator reflected the RPi5 autonomy stack, not the vehicle:

• W1 read external/rov-failsafe from failsafe_monitor — which did not gate on FC connectivity.
• W8 reads external/rov-mode, sourced from a YAML file. It would show AUV with the Pixhawk in a drawer.
• Foxglove "Connected: true" means connected to foxglove_bridge on the RPi5 — not to the vehicle.

Why this was critical: in AUV mode, health_role: gate makes W1 the pre-dive GO/NO-GO gate. A gate that cannot see the flight controller is not a gate — it would pass a dive with a dead FC.

Fix (11 Jul 2026) — four commits, verified live end-to-end

1. 86b2728 — comms loss now forces unconditional RED (previously scored as one marginal parameter among several, only reaching AMBER). Added an fcu_ever_connected latch so the startup window before any heartbeat has ever been received is not reported GREEN by default. Cause text ("Heartbeat Lost", "Vehicle Disconnected") added to FailsafeStatus.message, per DIR-7's "state must carry its cause" MUST.
2. 1b0050d — FSM recovery fix: RETURN_TO_SAFE was a terminal trap (only HOLD_AND_RECOVER could transition back to NORMAL); both states now recover on condition-clear. (Mode-blind — see DIR-7's 11 Jul in-mission-recovery addendum for the still-open mode-dependent design intent this mechanism needs to grow into.)
3. dbe9515 — cockpit_bridge publishes a new data lake variable external/rov-failsafe-cause (substring of message before the first |; empty when nominal). Stopgap per DIR-7's cause MUST — the DIR's Parked Design Items records the proper fix (a dedicated string cause field on FailsafeStatus.msg).
4. 3409ae5 — W1 widget shows the state word on #w1-message (state-driven, no flicker) and cause text on #w1-footer (red, 11px, blank when nominal).

Verified live: kill mavros → RED + "Heartbeat Lost"; restart mavros → recovers to GREEN.

Related fault found and fixed in the same session

cockpit_bridge was publishing failsafe_state (the FSM state, which can be 3 or higher) to external/rov-failsafe, not assessment_state (0/1/2) as W1 expects — fixed in commit d854641 (sibling of dbe9515). This was the actual cause of that day's "widgets show no data" symptom (a value of 3 falling outside W1's expected 0–2 range), compounded by a restart-storm that wedged port 9001 while diagnosing it (see CLAUDE.md's one-restart-then-verify-9001 rule, added as a direct result).

Still open (not resolved by this fix — see DIR-7 addenda and Parked Design Items)

• Mode-dependent in-mission recovery (AUV commit-to-recovery vs ROV operator handoff) — ESSENTIALLY COMPLETE 13 Jul 2026, see §15f. Remaining: the arrival-event publisher (temporary operator button, then real navigation).
• Auto-recovery (failsafe_monitor re-establishing the FC link itself)
• In-mission AMBER runtime-response, branched by mode
• Dedicated string cause field on FailsafeStatus.msg (current fix is a substring stopgap)
• FD-headroom check on ardupilot_manager as part of the AUV pre-dive gate (§15a mitigation (a), still not built)

---

15f. Mode system launch-wiring gap + gate-mode recovery arrival event — 13 Jul 2026

Fault 1 — mode_profile_loader never wired into the stack launch file

mode_profile_loader (rov_mission) was built and committed 7 Jul 2026 (4e48dc2) — node, ModeProfile.msg, mode_profiles.yaml, cockpit_bridge flag surfacing, all verified working in isolation at the time. It was never added to rov_full.launch.py. Confirmed on RPi5, 12 Jul 2026: ros2 node list showed no mode node; ros2 topic info /rov/mode/profile reported Publisher count: 0, 2 subscribers waiting. The entire operating-mode profile system had been dark at runtime — zero publishers — for the six days since it was built. failsafe_monitor's health_role had been silently running on its gate default the whole time, not the actually configured mode.

Fix (464e17e): added the Node action for mode_profile_loader to rov_full.launch.py, placed with the other core nodes (before Perception) rather than down in the mission-package include further below, so consumers of its latched output (failsafe_monitor, cockpit_bridge) wait as little as possible on a publisher. respawn=True/respawn_delay=5.0, matching the other core nodes (c099c00, a5e1c99) — if this node dies the whole mode system goes dark again. Verified live: publisher count 0 → 1.

Fault 2 — gate-mode RETURN_TO_SAFE was a terminal trap

With the mode system now actually publishing a real gate profile, a second gap surfaced. failsafe_monitor's mode-aware recovery (2a3e577, same session) correctly commits gate-mode (AUV) HOLD_AND_RECOVER → RETURN_TO_SAFE on condition-clear per the DIR-7 addendum — but then never left RETURN_TO_SAFE. That branch deliberately ignores further condition-clear (recovery, once committed, must run to completion — see the addendum) but nothing had ever been wired to signal that the recovery HAD completed.

Fix (bf815cc): new subscription, /rov/nav/safe_zone_reached (std_msgs/Bool), same plain QoS and latch-on-True callback pattern as the existing flag_manual_abort subscription. When health_role == gate and fsm_state == RETURN_TO_SAFE and a True is received, the FSM transitions to NORMAL and the cause text reads "Safe zone reached — recovery complete" (DIR-7's cause-text MUST, via the existing message-prefix mechanism — no FailsafeStatus.msg change, no cockpit_bridge/W1 change needed). Any other combination (not in RETURN_TO_SAFE, or advisory mode) is ignored — advisory mode keeps its separate, untouched operator-resume mechanism.

This topic and this subscription are the PERMANENT nav-arrival interface — the navigation subsystem will publish the real event here (GPS at surface, EKF dead-reckoning underwater; see the new Parked Design Item in docs/design_intent_register.md for the full sensor-derivation detail). Only today's PUBLISHER is temporary: verified live end-to-end on the bench by publishing the event by hand and watching the FSM release RETURN_TO_SAFE → NORMAL and the motion_controller recovery broadcast stop. The permanent replacement publisher — a temporary W1-contextual operator button (bench scaffolding, rov_api-backed) ahead of navigation raising the event for real — is next (see §0 NEXT).

Pre-existing defects found this session (not caused by this session's work)

flag_manual_abort is a one-way latch, never reset. Set True in _abort_callback on any /rov/mission/abort message; nothing anywhere in failsafe_monitor ever sets it back to False. Once any manual abort fires, Priority 6 (if self.flag_manual_abort: ...; return) returns early on every subsequent evaluation cycle for the rest of the process lifetime — permanently shadowing every lower-priority handler, including the new mode-aware recovery and arrival-event logic added this session (neither is reachable once this latch is set). Discovered during this session's work on 2a3e577; the bug itself predates it. Needs a reset condition — candidates: on reaching NORMAL, on an explicit operator reset, or only on process restart — undecided, needs its own session. Field-deployment concern: a single manual abort currently disables all future automatic recovery for the rest of that dive.

failsafe_monitor produces no log output at all. Silent in journalctl this session — no info/warn lines observed for state transitions or startup. Its state had to be read directly off /rov/failsafe (ros2 topic echo) every time rather than watched in the service log. Minor, but cost real diagnostic time this session. Should log state transitions and cause on each one.

---

MAVLinkRouter vs MAVP2P

BlueOS itself warns "MAVP2P has been presenting issues and is not currently recommended." MAVLinkRouter is the confirmed stable router. Do not recommend or use MAVP2P.

Symptoms (for reference — should not recur):

• Frontend connected/disconnected cycling
• "No board running" / "No endpoints available" in BlueOS UI
• AUTOPILOT_*_FETCH_FAIL (timeout 30000ms) in notifications
• OSError: [Errno 24] Too many open files in ardupilot-manager logs

---

15b. Session 6 Lessons Learned — RPi5 Base Stack Install

SSH on Ubuntu 24.04 Server (cloud-init image)

Ubuntu 24.04 preinstalled server image disables password authentication by default. To enable persistently:

echo "PasswordAuthentication yes" | sudo tee /etc/ssh/sshd_config.d/99-password-auth.conf
sudo systemctl restart ssh

Do NOT add KbdInteractiveAuthentication — removed in OpenSSH 9.x, causes sshd to fail on Ubuntu 24.04.

NVMe clone via dd — PARTUUID handling

dd copies the full disk bit-for-bit including MBR disk identifier, resulting in identical PARTUUIDs on both SD and NVMe. Fix with fdisk. However Ubuntu 24.04 preinstalled image uses LABEL= in both cmdline.txt and fstab, so the duplicate PARTUUID issue does not cause boot failure on this image.

rpi-clone does not support NVMe partition naming

rpi-clone fails on NVMe devices (expects sda1 style, not nvme0n1p1). Use dd for full disk clone.

initramfs rebuild required after NVMe clone on Ubuntu

After cloning to NVMe, rebuild initramfs before booting:

sudo chroot /mnt/nvme-root update-initramfs -u -k all

Ethernet cable quality matters

Two cables failed silently (no LEDs, interface DOWN) before a third worked. Always test with a known-good cable before debugging the OS.

---

15c. Session 7 Lessons Learned — Workspace Deploy & First Full Launch

The committed Gitea workspace was never build-tested — three latent bugs

1. Missing config/ dir in rov_bringup (commit f1c6710)
2. Missing setup.cfg in all 5 Python packages (commit 7590faf) — without them, setuptools installs executables to bin/ instead of lib/<pkg>/, so ros2 launch / ros2 run cannot find them.
3. ekf.yaml mixed int/float (commit 45b3301) — ROS2 YAML parser requires uniform sequence types.

Process lesson: future handovers must distinguish "written" from "built and run".

colcon: --symlink-install works correctly with setup.cfg files in place

With setup.cfg present in each Python package, --symlink-install correctly places executables in lib/<pkg>/. This is now the standard build command.

MAVROS full-launch fixes (commit c94412d) — reusable knowledge

• fcu_url format: udp://@<remote_ip>:14550 is correct (bind any local, connect remote).
• MAVROS plugin collisions on Jazzy: fix with plugin_allowlist: ['sys_status','sys_time','command','imu','global_position','setpoint_velocity'].
• QoS on MAVROS topics: use qos_profile_sensor_data (BEST_EFFORT) on MAVROS-sourced subs.
• SITL/real hardware battery: treat battery_percent < 0 as UNKNOWN, not critically low.
• Rebuild discipline: colcon build --packages-select <pkg> required after editing installed launch/config files.

---

15e. Session — RPi5 Storage Re-Architecture

RPi5 boot facts:

• BOOT_ORDER is a hex value in EEPROM (nibbles, read right-to-left: 1=SD, 6=NVMe, 4=USB, f=restart-loop).
• FIX for wrong-root boot: change /boot/firmware/cmdline.txt and /etc/fstab from LABEL= to PARTUUID= (PARTUUIDs differ between drives; labels collided).
• End-state BOOT_ORDER: 0xf41 (SD-first). NVMe is data-only — nothing to compete.

SD hardening:

• log2ram — /var/log is a RAM disk (128M).
• /tmp → tmpfs (256M) via fstab.
• journald capped: SystemMaxUse=20M in /etc/systemd/journald.conf.

systemd service:

• /etc/argonaut/rov.env — EnvironmentFile: ENV=dev, BLUEOS_IP=192.168.1.100.
• /usr/local/bin/argonaut-stack.sh — wrapper: sources ROS2 setup, execs launch.
• /etc/systemd/system/argonaut.service — User=ubuntu, EnvironmentFile, Restart=on-failure, RestartSec=5.
• DEV POLICY: installed but NOT enabled-on-boot. Start manually: sudo systemctl start argonaut. Field: sudo systemctl enable argonaut.

---

16. Known Gaps and Risks

Gap	Notes	Priority
DVL absent	X/Y position dead-reckoned from IMU — drifts	High — Phase 7
Ping2 sonar not purchased	W6 full test blocked; downward altitude not flowing	Phase 3 (~$280 USD)
FastAPI backend not built	W3 abort, W4 setup, W5 budget, W7 backup all blocked	High — Phase 3 Stage 2
W3 button label not updated	Widget still shows "ABORT MISSION"; should be "RETURN TO SAFE"	Next session
Widget JSON not in Gitea	RESOLVED — W6 committed as w6_altitude.json, probe widget removed (785ec56), see §12	Closed
RPi5 weak SSH password	Password auth enabled, Tailscale service disabled but installed	Pre-field
RPi5 thermal in sealed enclosure	Conduction cooling path unresolved	Phase 3
ML inference compute	RPi5 has no GPU. ONNX/TFLite on-device or Jetson Orin Nano.	Phase 5
Sim/ML dev compute	Track 2 (perception) needs RTX GPU dev PC or cloud. Track 1 (Gazebo) runs on laptop.	Decision pending
Jacket training data	No public dataset. Path = synthetic-from-CAD + SubPipe/COU + field captures.	Phase 5+
PCIe Gen 3 (RPi5 NVMe)	dtparam=pciex1_gen=3 → ~2× throughput but UNCERTIFIED. Benefit nil until camera→MCAP.	Phase 3 revisit
Ops Server separation	Currently same machine as Dev Server	Near-term
Model conversion pipeline	Manual for now	Phase 6
Widget consolidation	Grant to review — may consolidate W0–W7 into single unified widget	Future

---

17. Shopping List

Item	Supplier	Approx Cost	Priority
CA-USB-MTI (Xsens cable)	Newark Electronics	—	High — long lead time
Ping2 Sonar (downward)	Blue Robotics	~$280 USD	Phase 3 — unblocks W6 full test
Ops Server (cloud VPS or local machine)	TBD	—	Medium — before field ops

---

18. Reference Commands

Terminal label convention and shell prompt mapping: [RPi4] → pi@blueos:~$ (192.168.1.100) [RPi5] → ubuntu@rov-brain:~$ (192.168.1.101) [SERVER] → grant@symbytech:~$ (192.168.1.175) [LAPTOP — Git Bash] → grant@BB8 MINGW64 ... Label appears as plain text ABOVE the code block, never inside it.

RPi4 — SSH Access

[LAPTOP — Git Bash]

ssh pi@192.168.1.100

RPi4 — BlueOS UI

[BROWSER]

http://192.168.1.100

RPi4 — BlueOS Container

[RPi4]

sudo docker ps --format "table {{.Names}}\t{{.Status}}"

[RPi4]

sudo docker restart blueos-core

[RPi4]

sudo docker logs blueos-core --tail 30 2>&1

RPi5 — SSH Access

[LAPTOP — Git Bash]

ssh ubuntu@192.168.1.101

RPi5 — argonaut.service

[RPi5]

sudo systemctl start argonaut.service

[RPi5]

sudo systemctl status argonaut.service

[RPi5]

journalctl -u argonaut.service -f

RPi5 — argonaut-api.service (FastAPI backend :8081)

[RPi5]

sudo systemctl start argonaut-api.service

[RPi5]

systemctl is-active argonaut-api.service && curl -s http://localhost:8081/health

[RPi5]

curl -s -X POST http://localhost:8081/abort

[RPi5]

curl -s -X POST http://localhost:8081/mission/start -H "Content-Type: application/json" -d '{"mission_id":"test01","parameters":[]}'

RPi5 — ROS2 Stack Health

[RPi5]

source /data/ros2_ws/install/setup.bash && ros2 node list | wc -l

[RPi5]

ros2 topic echo /mavros/state --once

[RPi5]

ros2 topic echo /rov/recording/active --once

[RPi5]

ros2 topic echo /rov/failsafe --once

RPi5 — Bags

[RPi5]

ls -lh /data/bags/

[RPi5]

du -sh /data/bags/*

RPi5 — Build

[RPi5]

cd /data/ros2_ws && colcon build --packages-select rov_mission --symlink-install 2>&1 | tail -10

Gitea

[LAPTOP — Git Bash]

cd "C:\Users\grant\Desktop\Autonomous ROV System\rov-autonomy"

[LAPTOP — Git Bash]

git pull origin master

[RPi5]

cd /data/ros2_ws/src/rov-autonomy && git pull origin master && git push origin master

Server — General

[SERVER]

df -h && docker ps

[SERVER]

cd ~/harbor && sudo docker compose ps

Server — Harbor

[SERVER]

docker login registry.symbytech.com

Browser — Dev URLs

[BROWSER]

http://192.168.1.100           BlueOS (RPi4)
http://192.168.1.101:8765      Foxglove bridge WebSocket (connect via Foxglove Studio)
https://registry.symbytech.com Harbor
https://git.symbytech.com      Gitea
http://192.168.1.175:81        NPM admin
http://192.168.1.175:9000      Portainer

Cockpit — Widget Import

[COCKPIT]

1. Connect to vehicle (192.168.1.100)
2. Settings -> General -> Enable Pirate Mode (required for DIY widgets, resets each session)
3. Enter edit mode (pencil icon, top right)
4. Add Widget at bottom — scroll right to find </> DIY widget
5. Drag onto main view area and resize
6. Click gear icon on widget -> Import -> select .json from rov-autonomy/widgets/

[COCKPIT]

Generic WebSocket (one-time setup, persists):
Settings -> Generic WebSocket Connections
Add: ws://192.168.1.101:9001

---

19. Document Index

All project docs live in the repo under docs/ (git-tracked, no version number in the filename — the Version: field inside each file is authoritative; project knowledge holds a mirror copy for chat context, re-uploaded after any commit that changes these). Do not cite a version number from memory — check the file itself.

Document	Purpose
docs/handover.md	This document — master reference, single source of truth for current state
CLAUDE.md (repo root)	Rules for Claude — behaviour, conventions, safety, research requirements
docs/design_intent_register.md	Design Intent Register — authoritative design decisions, DIR-1…DIR-12
docs/failsafe_design.md	Failsafe state machine and sensor roadmap
docs/ui_design.md	Cockpit widget specs, data paths, and build status
docs/simulation_ml_plan.md	Simulation + ML tooling: dev-PC tiers, cloud options, training assets
docs/mission_recording_lifecycle.md	Lifecycle research backing DIR-9/9a/10

Server/infra docs (SymbyTech server, Harbor registry) and task/shopping lists stay project-knowledge-only — not part of the rov-autonomy repo docs/ folder.

---

Update this document at the end of each working session. Increment version number and add rows to Version History and Changelog as appropriate. Commit to the repo (docs/handover.md); re-upload to Claude project knowledge for chat context.