Connection Watchdog & Auto-Recovery

Cellular connections are prone to silent lockups, tower handoff freezes, and carrier lease expirations. QManager's Connection Watchdog guarantees unattended 24/7 reliability.

QManager RM551E Connection watchdog screen — RM551E: The watchdog watches the internet link and climbs a recovery ladder when it drops.

Monotonic Staleness Counter#

Rather than relying on basic ping probes that can fail during transient packet drops, QManager uses a Monotonic Staleness Counter:

Sends lightweight ICMP and DNS probes to diverse endpoints (Cloudflare, Google, OpenDNS).
A single failure does not trigger a panic.
If failures persist consecutively across the configured threshold (e.g. 90 seconds), the multi-tier recovery pipeline engages.

The 4-Tier Recovery Pipeline#

QManager avoids premature reboots by attempting least-disruptive interventions first:

[Link Outage Detected]
         │
         ▼
[Tier 1: Interface Reset] ──> Soft restart network interface (ifup / ip link)
         │ (Still offline after 30s)
         ▼
[Tier 2: Radio Toggle]    ──> Cycle baseband RF via AT+CFUN=0 then AT+CFUN=1
         │ (Still offline after 60s)
         ▼
[Tier 3: SIM Failover]     ──> Switch to secondary SIM card or alternate APN
         │ (Still offline after 90s)
         ▼
[Tier 4: System Reboot]   ──> Full hardware reset (reboot)

Hardware Ethernet & PCIe Link Watchdog (ethernet_watchdog.sh)#

Beyond cellular WAN packet failure, standalone carrier boards (such as Rework 5G2PHY or Waveshare PCIe boards) can suffer from PCIe bus desynchronization or Realtek RTL8125 PHY lockups under sustained heavy throughput.

The Upstream `ethernet_watchdog.sh` Architecture#

Upstream research by Cameron Thompson (iamromulan) identified distinct kernel driver failure signatures in the Linux ring buffer:

eth0: cmd = 0xff (Indicates the PCIe bus has fallen off the root complex or register reads return all 1s).
eth0: pci link is down (PHY link partner dropped or PCIe controller entered an unrecoverable sleep state).

The upstream ethernet_watchdog.sh daemon continuously monitors dmesg for these signatures:

When cmd = 0xff is detected, the watchdog avoids an immediate modem reboot.

It attempts to unbind and reload the kernel driver:

Shell

echo "0000:01:00.0" > /sys/bus/pci/drivers/r8125/unbind
sleep 1
echo "0000:01:00.0" > /sys/bus/pci/drivers/r8125/bind

If reloading the driver fails to restore link within 15 seconds, it triggers a hardware PCIe reset line before escalating to a full system restart.

RF State Recovery on Boot (cfunfix.service)#

In certain power loss scenarios or carrier board brownouts, Quectel baseband firmware can initialize with its RF circuitry in offline mode (CFUN=0 or CFUN=4), leaving the modem unable to register on cellular towers despite the Linux OS running normally.

Upstream's cfunfix.service resolves this condition by executing early in the boot sequence:

Queries the current radio state: AT+CFUN?.
If the response is not +CFUN: 1, it issues:
AT
```
AT+CFUN=1
```
Verifies that the SIM card elementary files are unlocked and forces network registration (AT+COPS=0).

Token-Bucket Reboot Protection#

QManager incorporates a Token-Bucket Rate Limiter:

Restricts total reboots to a maximum (e.g. 3 reboots in a 2-hour window).
If the token bucket is exhausted, QManager halts reboot loops, keeps services alive for local access, and sends an alert over the cellular control channel.

Screens in QManager#

QManager RM551E Network events screen — RM551E: A timeline of band changes, handoffs, outages and recoveries.

QManager RM520N Network events screen — RM520N: Network Events logs band changes, handoffs, outages, and recoveries, newest first.

QManager RM551E Latency monitor screen — RM551E: Live and historical latency, plus how fast your favourite sites answer.

QManager RM520N Latency monitor screen — RM520N: Latency Monitor charts round-trip time to the probe target and to the sites you track.

QManager RM551E Connection quality screen — RM551E: Choose the ping targets and the latency and loss limits that count as a bad link.

QManager RM520N Connection quality screen — RM520N: Connection Quality sets the probe targets and the latency and loss limits that raise events.

QManager RM520N Watchdog screen — RM520N: The Watchdog probes the link and climbs a recovery ladder when it goes down.