Initial commit

docs/DESIGN.md

@@ -0,0 +1,102 @@
+# Design
+
+## Separate traffic forwarding from route selection
+
+HAProxy on Oracle forwards TLS traffic. The watchdog at home decides which DNS route new clients should take. Oracle does not run the watchdog and cannot switch DNS by itself. Its normal systemd service starts HAProxy again after reboot.
+
+```mermaid
+flowchart TB
+    W[Home Python watchdog] -->|Verified HTTPS to fixed Oracle IP| O[Oracle relay]
+    O -->|Home WAN 443| N[Home NPM health endpoint]
+    W -->|Verified HTTPS to NPM LAN IP| N
+    W -->|Public DoH lookup, then HTTPS| C[Permanent proxied canary]
+    C -->|Cloudflare to home| N
+    W -->|Scoped token: DNS API| D[Selected application A records]
+```
+
+## DNS ownership
+
+The interactive wizard creates a **fixed home IP** configuration. It discovers record IDs by reading the chosen Cloudflare zone. It does not create records or mutate remote DNS.
+
+| Record | Normal state | Backup state | Writer |
+| --- | --- | --- | --- |
+| `example.com` A | Oracle, DNS only, TTL 60 | Home, proxied, Auto | Watchdog |
+| `*.example.com` A | Oracle, DNS only, TTL 60 | Home, proxied, Auto | Watchdog |
+| `edge-health.example.com` A | Home, proxied, Auto | Same | Operator; read-only to fixed-IP watchdog |
+| Explicit unrelated CNAME/A/Worker records | Existing configuration | Existing configuration | Existing owner |
+| MX/TXT and other unrelated records | Existing configuration | Existing configuration | Existing owner |
+
+An explicit DNS name can override wildcard behavior. The wizard defaults to apex plus wildcard, but `--app` selects other existing A records. The controller never rewrites every record in the zone indiscriminately. See [Cloudflare wildcard behavior](https://developers.cloudflare.com/dns/manage-dns-records/reference/wildcard-dns-records/).
+
+For each managed name, it verifies the A record's exact ID/name and rejects a conflicting A, AAAA, CNAME, HTTPS, or SVCB record at that name. Optional exact comments can strengthen ownership checks. Unrelated records are read during inventory but not patched. Inspect explicit subdomains and Worker routes separately: checking the apex/wildcard does not audit every possible application route.
+
+### Advanced origin-record mode
+
+The controller also supports replacing `home_ip` with an `origin` reference to a dedicated DNS-only DDNS A record. That record remains owned by your DDNS updater. In that mode the watchdog can update the canary to the current DDNS address and then reconcile the application records.
+
+The included Oracle HAProxy renderer still produces a fixed-IP backend. Enabling `origin` mode alone does **not** update Oracle's backend, arrange DDNS, or supply a complete dynamic-home-IP deployment. The installation guide intentionally uses fixed `home_ip`; do not configure both fields.
+
+## Three independent HTTPS checks
+
+| Check | Socket destination | TLS SNI and HTTP Host | Purpose |
+| --- | --- | --- | --- |
+| `primary` | Configured Oracle IPv4:443 | Health hostname | Verifies Oracle → home → NPM regardless of current app DNS |
+| `local` | NPM LAN IPv4:443 | Health hostname | Verifies home NPM is available |
+| `canary` | Public A answer for permanent proxied health hostname | Canary hostname | Verifies Cloudflare → home → NPM |
+
+All require a valid certificate, status 200, the exact configured body including newline, and `Cache-Control: no-store`. Each request includes a random cache-busting query. The canary additionally requires `CF-Ray` and `CF-Cache-Status` of `DYNAMIC` or `BYPASS`.
+
+The canary resolver opens TLS to public `1.1.1.1:443` with SNI `cloudflare-dns.com` and requests the hostname's A records through DNS-over-HTTPS. It validates the resolver certificate and accepts only public IPv4 answers for the exact name. A lookup failure fails the probe; it does not fall back to LAN DNS. This prevents a local wildcard override from making a direct LAN response look like a working Cloudflare route. [Cloudflare DoH JSON API](https://developers.cloudflare.com/1.1.1.1/encryption/dns-over-https/make-api-requests/dns-json/).
+
+The chosen public destination changes with fresh DNS answers; it is not a hardcoded Cloudflare website IP. The original hostname is retained for TLS validation and HTTP routing. Outbound proxies from environment variables are not used for these checks or the DNS API.
+
+Probe jobs run in child processes with a total deadline, including DNS/TLS/body reading. The default health budget is five seconds, with API calls separately bounded at ten seconds. If one child stalls, the parent stops it. System clocks must be accurate for TLS validation; health-duration accounting uses monotonic time.
+
+## Switching state machine
+
+```mermaid
+stateDiagram-v2
+    [*] --> Inspect
+    Inspect --> Oracle: DNS and stored state agree on Oracle
+    Inspect --> Cloudflare: DNS and stored state agree on backup
+    Inspect --> Stopped: Mixed DNS, invalid identities, or state conflict
+    Oracle --> Cloudflare: 4 consecutive primary failures AND local + canary healthy
+    Cloudflare --> Oracle: Primary healthy for 300s AND backup dwell 600s AND local healthy
+```
+
+With a 15-second interval, four failed samples usually require roughly a minute, plus API execution and resolver caching. This is not a hard recovery-time guarantee.
+
+Only return to Oracle is gated by the minimum dwell; a newly failed Oracle route can fail over without waiting ten minutes. Continuous healthy time and dwell both restart when the watchdog starts. A gap longer than twice the sample interval resets continuous health. An API failure also resets the health window and holds/backoffs rather than inventing a successful write.
+
+A healthy Cloudflare canary is required to move into fallback. It is not required to return to a healthy Oracle route. `--check` requires all three checks because it verifies readiness of the complete deployment. Manual `--once --mode … --apply` bypasses waiting periods but still requires valid record identities and a healthy local/selected destination.
+
+## API writes and crash handling
+
+1. Read and validate the current zone inventory.
+2. Probe the routes and compute a desired mode.
+3. Save desired mode and timestamp atomically before a potentially ambiguous DNS write.
+4. Submit only required A-record PATCHes in a batch.
+5. Read DNS records back from the API and verify content, proxy status, and TTL.
+
+During a running process, lost responses or partial outcomes are reconciled against the persisted intent. On a fresh start, mixed records or stored-state disagreement require explicit operator recovery. The controller chooses caution over silently deciding that an unexpected DNS edit represents an outage.
+
+One file lock prevents concurrent controllers sharing a state file. The state write uses a private temporary file, `fsync`, and atomic replacement. These mechanisms assume a suitable local filesystem; they do not coordinate multiple independent watchdog hosts or distinct state files.
+
+Cloudflare API batching is not instantaneous worldwide DNS propagation. Clients may retain old answers, reuse existing connections, or take longer than the displayed TTL to move. [Cloudflare batch changes](https://developers.cloudflare.com/dns/manage-dns-records/how-to/batch-record-changes/), [TTL behavior](https://developers.cloudflare.com/dns/manage-dns-records/reference/ttl/).
+
+## Relay limits
+
+HAProxy routes only an ordinary TLS ClientHello with a matching apex/subdomain SNI to one fixed home TCP 443 backend. No HTTP parser, certificate, private key, or PROXY protocol is configured at Oracle.
+
+The template limits concurrent connections and connection rates. Shared NAT clients can hit per-source limits, so adjust them based on measured use. Five-minute client/server timeouts are inactivity timeouts; reloads drain old processes for up to ten minutes before terminating remaining connections. The backend TCP check only tests connectivity; the home watchdog supplies the HTTPS/body test.
+
+## Failures this cannot solve
+
+- Home power, NPM, WAN, or ISP failure: both routes still depend on home.
+- A stopped watchdog: the last DNS mode remains until someone runs a writer.
+- A DNS API/control-plane outage: route switching may be unavailable.
+- An application failure hidden behind a successful static NPM health page.
+- Every client's network path differing from the home server's observations.
+- In-flight streams/WebSockets: DNS changes affect future connections, not existing sessions.
+
+See [Security](../SECURITY.md) for trust boundaries and [Operations](OPERATIONS.md) for drills and repair.

docs/INSTALL.md

@@ -0,0 +1,189 @@
+# Installation
+
+This guide keeps the existing home WAN 443 → NPM forwarding. It creates a relay on Oracle and runs the DNS watchdog at home. No DNS changes occur merely from running either configuration wizard.
+
+Already deployed? Follow [the update procedure](OPERATIONS.md#updating-an-existing-installation) instead of rerunning the firewall installer.
+
+## 1. Collect the values
+
+| Value | Meaning | Where it is used |
+| --- | --- | --- |
+| Domain | Your domain managed in Cloudflare; called a **zone** there | DNS record names and TLS SNI |
+| Oracle IPv4 | Reserved public IPv4 assigned to your VM | Public DNS, home source allowlist, primary health probe |
+| Home IPv4 | Current public WAN IPv4 | Oracle's forwarding destination and Cloudflare backup |
+| NPM LAN IPv4 | The address already targeted by your home port-forward | The watchdog's local HTTPS check only; it does not change the forward |
+| Administrator CIDR | Your SSH source public IPv4 followed by `/32`, meaning one address | Restricts SSH on Oracle; the installer derives it from the SSH connection |
+| Zone ID | Cloudflare's identifier shown on the zone overview page | API configuration |
+
+`example.com` in this guide is a placeholder. Enter your own values when the scripts prompt. Do not put personal values into tracked source files.
+
+Use a reserved Oracle public IP so stop/start operations do not unexpectedly change your allowlist or DNS. The documented configuration assumes a fixed home address; see [IP changes](OPERATIONS.md#home-or-oracle-ip-changes).
+
+## 2. Keep the home forward; add Oracle to its allowed sources
+
+In OPNsense, keep your existing TCP WAN 443 → NPM LAN 443 forward.
+
+1. Create a host alias, for example `oracle_edge`, containing your Oracle public IPv4.
+2. Include that alias alongside the existing Cloudflare IP ranges in the **source restriction that actually governs this forward**. You can use a combined alias with your existing Cloudflare aliases plus `oracle_edge`.
+3. Apply the change. Keep both Cloudflare and Oracle sources allowed permanently.
+4. Check both the NAT rule and its associated WAN pass rule. Adding an unused alias alone grants no access. A source restriction on the NAT rule must also include Oracle so translation occurs.
+
+Do not add another port-forward. Do not expose NPM's administration port 81. If your existing rules use a different layout, inspect them rather than replacing them with a generic rule. Keep the [official Cloudflare address lists](https://www.cloudflare.com/ips/) current. See [OPNsense NAT documentation](https://docs.opnsense.org/manual/nat.html).
+
+## 3. Create one permanent Cloudflare health hostname
+
+Leave your currently working application DNS in its current mode while preparing the other route. For a new deployment that starts through Cloudflare, both managed application A records should contain your home IPv4 and be proxied.
+
+Add this record in Cloudflare DNS, substituting your own domain and home IP:
+
+| Type | Name | Content | Proxy status | TTL |
+| --- | --- | --- | --- | --- |
+| A | `edge-health` | Home public IPv4 | Proxied / orange | Auto |
+
+The resulting name is `edge-health.example.com`. It stays pointed at home even when applications use Oracle. Do not include it among the switchable records.
+
+Under **SSL/TLS**, use **Full (strict)**. Ensure both the Cloudflare edge certificate and NPM certificate cover all failover hostnames. Avoid any origin rule that sends this health hostname somewhere else or changes its SNI/Host. Existing Worker routes, redirects, or Access policies can also intercept requests.
+
+Full (strict) authenticates and encrypts the Cloudflare-to-home connection; it does not hide HTTP from Cloudflare. [Mode requirements](https://developers.cloudflare.com/ssl/origin-configuration/ssl-modes/full-strict/).
+
+## 4. Add the NPM health host
+
+Create a separate **Proxy Host** in NPM:
+
+| Setting | Value |
+| --- | --- |
+| Domain name | `edge-health.example.com`, using your domain |
+| Scheme / forward host / port | `http` / `127.0.0.1` / `9` |
+| Access list | Publicly Accessible |
+| SSL certificate | Publicly trusted certificate covering this hostname |
+| Advanced | Contents of `npm-health-advanced.conf` |
+
+The forwarding address is an unused UI placeholder: the supplied configuration returns the health marker directly and rejects every other path. It must be on this dedicated health host, not your application hosts. Do not add custom locations that override it.
+
+Your existing valid wildcard certificate can cover this single-level health subdomain. A Cloudflare Origin CA-only certificate is unsuitable for this setup because the direct Oracle route must be trusted by ordinary clients. [Origin CA limitations](https://developers.cloudflare.com/ssl/origin-configuration/origin-ca/).
+
+In Cloudflare, create a **Cache Rule** matching this expression with Cache eligibility set to **Bypass cache**:
+
+```text
+(http.host eq "edge-health.example.com" and http.request.uri.path eq "/__edge_health")
+```
+
+Substitute your domain. Ensure this path does not receive a challenge, Access login, or redirect. Scope any exception to this harmless health hostname/path, preserving application protections. The response must be status 200, exactly `edge-ok-v1` followed by a newline, `Cache-Control: no-store`, `CF-Ray`, and `CF-Cache-Status: DYNAMIC` or `BYPASS`.
+
+## 5. Prepare the Oracle VM
+
+Copy this source folder to the VM using your usual SSH/SCP method. Do not copy home credentials or `runtime/` from another machine. On Oracle, in the source folder:
+
+```bash
+python3 configure-oracle.py
+```
+
+Enter the domain and home public IPv4. This generates ignored `runtime/haproxy.cfg` with private file permissions.
+
+### OCI network rules
+
+In the VM's applicable NSGs/security lists, allow **stateful** IPv4 ingress:
+
+- TCP 443 from `0.0.0.0/0`, because ordinary clients connect directly to Oracle.
+- TCP 22 from your administrator public IPv4 `/32`.
+
+Retain the egress needed for home TCP 443, DNS, package updates, and OCI operation. Audit the combined NSGs and security lists: another broad rule can still grant access. This profile does not require ingress on port 80, 3128, or UDP 443. See [OCI security rules](https://docs.oracle.com/en-us/iaas/Content/Network/Concepts/securityrules.htm).
+
+### Dedicated Ubuntu 24.04 VM only
+
+Have an OCI console recovery method available. Keep your existing SSH session open. The installer expects IPv4 SSH on port 22 and refuses existing UFW/Docker/custom native firewall setups rather than removing their policy.
+
+```bash
+sudo bash preflight-oracle.sh
+sudo env "SSH_CONNECTION=$SSH_CONNECTION" bash setup-oracle.sh
+```
+
+If preflight stops, read [the firewall troubleshooting section](TROUBLESHOOTING.md#oracle-firewall-or-installer-stops). Do not disable a firewall simply to force the installer through.
+
+The installer validates HAProxy's configuration, enables the relay, and stages the host firewall with a **180-second rollback timer**. The host firewall permits administrator SSH and public TCP 443, essential ICMP/DHCP and established traffic, while blocking other new incoming services. It preserves existing OUTPUT rules, including OCI InstanceServices protections. Existing established sessions are intentionally retained.
+
+### Complete the timed firewall verification
+
+Before the timer expires, open a **new** SSH session to Oracle from your administrator address. From the home server or another external terminal, enter the real values and test:
+
+```bash
+read -r -p 'Health hostname: ' HEALTH_HOST
+read -r -p 'Oracle public IPv4: ' ORACLE_IP
+curl --noproxy '*' --fail --show-error --max-time 15 \
+  --resolve "$HEALTH_HOST:443:$ORACLE_IP" \
+  "https://$HEALTH_HOST/__edge_health"
+```
+
+Run those commands in Bash. Require `edge-ok-v1` and a valid certificate. Also test a real NPM application with `--resolve` using its own hostname. Never add `-k` to bypass certificate verification.
+
+After successful tests, run the **exact timer-stop, rollback-status, chain-check, and persistence commands printed by the installer**. Timer names are generated for each run. Verify the rollback service is inactive and both head jumps still exist before saving with `netfilter-persistent save`.
+
+If you miss the deadline, the firewall restores its pre-change rules. Inspect the state before trying again; do not save blindly. The timer rolls back firewall rules, not installed packages or HAProxy configuration.
+
+## 6. Create the Cloudflare API token
+
+Create a custom token with **Zone → DNS → Edit**, scoped to **only the zone being managed**. Record the Zone ID from that zone's overview page. The script requests existing DNS records by that known ID and does not need account administration privileges.
+
+Use a separate token from certificate-renewal/DDNS automation. Enter it into the hidden prompt; do not paste it into a command, tracked file, screenshot, or issue. Cloudflare's token resource scope is the zone; the script's per-record checks narrow its own actions but cannot reduce what a stolen zone token could do. [API token permissions](https://developers.cloudflare.com/fundamentals/api/reference/permissions/).
+
+## 7. Configure and run at home
+
+Copy the source folder to an always-on **home** server. Run from that folder as one consistent user:
+
+```bash
+python3 configure-home.py --local
+python3 watchdog.py --config "$PWD/runtime/watchdog.json" --check
+```
+
+Enter your domain, Oracle IPv4, home IPv4, Zone ID, hidden token, and NPM's existing LAN IPv4. The wizard looks up exact record IDs and saves private local configuration. It does not change DNS or OPNsense.
+
+To switch only selected existing A records, use this in place of the first command (substitute your domain):
+
+```bash
+python3 configure-home.py --local \
+  --app app.example.com --app status.example.com
+```
+
+Quote wildcard arguments, such as `--app '*.example.com'`, so the shell does not expand them. `--health-host` can select a different dedicated proxied health name within the domain.
+
+`--check` requires all three routes to pass. It resolves the Cloudflare canary using authenticated public DNS-over-HTTPS, independently of local split DNS. The home server must reach public `1.1.1.1:443`. Local NPM and Oracle checks use their explicit IPs with the correct TLS hostname.
+
+Once all checks pass, switch to Oracle:
+
+```bash
+python3 watchdog.py --config "$PWD/runtime/watchdog.json" \
+  --once --mode oracle --apply
+```
+
+Verify the managed records now contain the Oracle IP, **DNS only**, TTL 60, and that real applications work from an external network. Explicit DNS records can override wildcard routing.
+
+Start the continuous watchdog:
+
+```bash
+umask 077
+nohup python3 -u watchdog.py --config "$PWD/runtime/watchdog.json" \
+  --run --apply > runtime/watchdog.log 2>&1 &
+echo $! > runtime/watchdog.pid
+tail -n 50 -F runtime/watchdog.log
+```
+
+`Ctrl+C` exits the log viewer only. `nohup` survives terminal closure; it does **not** restart the watchdog after a server reboot or process failure. Use your home server's service supervisor for that, or the systemd option below. Do not run two copies. Without `--apply`, no DNS changes occur.
+
+## 8. Test failover and automatic return
+
+Follow the [outage drill](OPERATIONS.md#outage-drill). Expect a switch after four failing Oracle probes only when the local and public Cloudflare probes are healthy. DNS caches add delay. Recovery is automatic after five continuous healthy minutes and a ten-minute minimum backup dwell; you do not rerun an Oracle script to switch back.
+
+## Optional: systemd on a home Ubuntu 24.04 host
+
+Choose this instead of the plain `nohup` process. Stop any existing local watchdog first. This installer copies the scripts to system paths and prompts for a separate private configuration:
+
+```bash
+sudo bash setup-home.sh
+sudo python3 -I /usr/local/lib/oracle-edge/watchdog.py \
+  --config /etc/oracle-edge/watchdog.json --once --mode oracle --apply
+sudo chown -R oracle-edge:oracle-edge /var/lib/oracle-edge
+sudo systemctl enable --now oracle-edge-watchdog.service
+sudo journalctl -u oracle-edge-watchdog.service -n 50 -f
+```
+
+The service runs as an unprivileged account and reads the token through systemd credentials. The installer does not start the writer until you enable it. Keep `/etc/oracle-edge` and `/var/lib/oracle-edge` out of your repository. For service-based updates and manual switches, see [Operations](OPERATIONS.md).

docs/OPERATIONS.md

@@ -0,0 +1,187 @@
+# Operations
+
+Commands below assume a shell in the installed source folder on the **home server**, unless marked Oracle. Keep the same user and private configuration paths used during setup.
+
+## View logs
+
+For the plain Python process:
+
+```bash
+tail -n 50 -F runtime/watchdog.log
+```
+
+For the optional systemd service:
+
+```bash
+sudo journalctl -u oracle-edge-watchdog.service -n 50 -f
+```
+
+`Ctrl+C` stops the viewer, not the watchdog. Important JSON events:
+
+| Event | Meaning |
+| --- | --- |
+| `startup` | Observed DNS mode and current intent; `apply` shows whether writes are enabled |
+| `health` | `primary`, `local`, and `canary` status plus consecutive primary failures |
+| `hold` | The desired destination was not proven healthy; DNS retained |
+| `dry_run` | Proposed action only; no DNS change |
+| `dns_verified` | API readback matches the desired DNS state; caches may still use the old route |
+| `api_hold` | DNS API error/backoff; inspect authentication, limits, or connectivity |
+| `fatal` | Configuration/identity/state conflict; the process stops for inspection |
+| `stopped` | Graceful shutdown completed |
+
+Logs may include deployment information. Redact IPs, domains, and paths before sharing them. Never share `cf-token` or unredacted configuration.
+
+## Stop and start the watchdog
+
+For the `nohup` process, first inspect the stored PID:
+
+```bash
+ps -p "$(cat runtime/watchdog.pid)" -o pid=,args=
+```
+
+Only if it is this watchdog with the expected config, send SIGTERM:
+
+```bash
+kill -TERM "$(cat runtime/watchdog.pid)"
+tail -n 20 runtime/watchdog.log
+```
+
+Wait for `stopped` or confirm that PID has exited before running a manual command. A stale PID file can refer to another process; do not kill it without inspecting. The state lock also prevents a second instance from running against the same state file. Avoid `kill -9` during a DNS write because it can leave a state/DNS conflict to repair.
+
+Restart with:
+
+```bash
+umask 077
+nohup python3 -u watchdog.py --config "$PWD/runtime/watchdog.json" \
+  --run --apply > runtime/watchdog.log 2>&1 &
+echo $! > runtime/watchdog.pid
+```
+
+This replaces the local log. If you need old logs, rotate/archive them privately before restarting. Configure rotation for a long-running `nohup` deployment; the script itself does not rotate files. For systemd, use `sudo systemctl stop/start oracle-edge-watchdog.service` and journal retention settings instead.
+
+Stopping the watchdog leaves DNS in its last state. It does not shut down HAProxy, NPM, or applications.
+
+## Manual switch or repair
+
+Stop the continuous writer first. To use Oracle:
+
+```bash
+python3 watchdog.py --config "$PWD/runtime/watchdog.json" \
+  --once --mode oracle --apply
+```
+
+To use Cloudflare directly to home:
+
+```bash
+python3 watchdog.py --config "$PWD/runtime/watchdog.json" \
+  --once --mode cloudflare --apply
+```
+
+These bypass recovery timers but require the local and selected route to work. They still enforce record identity checks. `--check` is a separate all-route preflight and cannot be combined with `--apply`.
+
+Mixed records or stored-state disagreement at startup need inspection. Once you have established which route is correct, the explicit one-shot command can reconcile recognized records and replace corrupt/conflicting local state. Do not erase state or change IDs blindly to suppress an error.
+
+For the systemd installation, stop the service and use its installed paths:
+
+```bash
+sudo systemctl stop oracle-edge-watchdog.service
+sudo python3 -I /usr/local/lib/oracle-edge/watchdog.py \
+  --config /etc/oracle-edge/watchdog.json --once --mode cloudflare --apply
+sudo chown -R oracle-edge:oracle-edge /var/lib/oracle-edge
+sudo systemctl start oracle-edge-watchdog.service
+```
+
+Substitute `oracle` to switch back. Run the final start only after the manual command succeeds and the result is understood.
+
+## Outage drill
+
+1. Confirm all three probes pass and the continuous writer logs `apply: true`.
+2. Confirm the managed records point at Oracle with DNS only. Verify a real application from outside the LAN; split DNS may keep local browsers on the LAN route regardless of public failover.
+3. Watch the home log.
+4. **On Oracle**, stop the relay to simulate loss of that route:
+
+   ```bash
+   sudo systemctl stop haproxy
+   ```
+
+5. Expect `primary: false`, `local: true`, and `canary: true`. After four consecutive primary failures, look for `dns_verified` with `desired: cloudflare`.
+6. Confirm both managed A records contain the home IPv4, are proxied, and have Auto TTL. Test an actual application externally. Old DNS answers and existing connections may delay a client moving.
+7. **On Oracle**, restore the relay:
+
+   ```bash
+   sudo systemctl start haproxy
+   sudo systemctl status haproxy --no-pager
+   ```
+
+8. The home watchdog should see `primary: true` again. It returns to Oracle after **300 continuous healthy seconds** and **600 seconds in backup**, whichever condition completes later. A watchdog restart restarts both timers. Watch for `dns_verified` with `desired: oracle`.
+
+No failover script runs on Oracle. If the whole VM reboots, HAProxy starts automatically only if its service is enabled and its startup succeeds. The home watchdog must remain running to perform either DNS transition.
+
+If the backup probe fails, the controller deliberately does not switch into that route. Fix the route/probe; do not delete the check to make a drill appear successful. The static health endpoint does not prove the application itself is working.
+
+## Updating an existing installation
+
+The corrected watchdog accepts the prior `runtime/watchdog.json` schema. You do not need to rerun either configuration wizard, recreate DNS records, regenerate credentials, or rerun the Oracle firewall installer merely to update it.
+
+### Plain Python at home
+
+1. Download/extract the new source separately. Review changes and run its local tests.
+2. Stop the existing watchdog and wait for its state lock to release.
+3. Privately back up the old `watchdog.py` and `runtime/` outside the publication checkout.
+4. Copy the new `watchdog.py` into the existing installation folder, preserving `runtime/` and its permissions. Keeping that directory in place also preserves absolute paths stored in configuration.
+5. With Oracle running, run:
+
+   ```bash
+   python3 watchdog.py --config "$PWD/runtime/watchdog.json" --check
+   ```
+
+6. Require all three checks to pass, then restart the continuous process. If a real Oracle outage is ongoing, `--check` will correctly fail its primary probe; verify a backup-target manual operation rather than forcing an all-route success.
+7. Repeat the controlled outage drill. This is especially important when installing the public-DNS canary correction.
+
+If you relocate the install folder, update absolute `token_file` and `state_file` paths in the private config. Simply renaming the folder does not rewrite them.
+
+### systemd at home
+
+From the new source folder on the home host:
+
+```bash
+sudo systemctl stop oracle-edge-watchdog.service
+sudo install -o root -g root -m 644 watchdog.py /usr/local/lib/oracle-edge/watchdog.py
+sudo python3 -I /usr/local/lib/oracle-edge/watchdog.py \
+  --config /etc/oracle-edge/watchdog.json --check
+sudo chown -R oracle-edge:oracle-edge /var/lib/oracle-edge
+```
+
+After successful checks:
+
+```bash
+sudo systemctl reset-failed oracle-edge-watchdog.service
+sudo systemctl start oracle-edge-watchdog.service
+```
+
+Back up the old script privately first. The existing `/etc/oracle-edge` credentials/config and `/var/lib/oracle-edge` state stay in place. A service-file change additionally needs review, installation, and `systemctl daemon-reload`; replacing Python alone does not require daemon reload.
+
+## Home or Oracle IP changes
+
+For the fixed-IP profile, a home ISP address change is not automatically handled:
+
+1. Stop the watchdog to prevent competing changes.
+2. Update the private home `home_ip`, the permanent proxied canary A record, and any currently active fallback application A records consistently.
+3. On Oracle, render a new private configuration with `configure-oracle.py`. Back up the installed configuration. Validate with `sudo haproxy -c -f runtime/haproxy.cfg` before installing it as `/etc/haproxy/haproxy.cfg`, then reload HAProxy.
+4. Verify all routes and inspect any state/DNS disagreement. Use an explicit tested manual mode if needed, then restart the watchdog.
+
+For an Oracle public IP change, update the home allowed source, private watchdog `oracle_ip`, relevant OCI access configuration, and active Oracle-mode app records. Add the new allowed source before removing the old one. A reserved public IPv4 avoids most routine address changes.
+
+Do not rerun `setup-oracle.sh` to update only HAProxy's backend; it is an initial host installer with firewall staging.
+
+## Emergency DNS fallback
+
+If the writer cannot run but the Cloudflare route is independently known healthy, stop/disable the writer and edit only its managed application A records in Cloudflare:
+
+- Content: current home public IPv4.
+- Proxy: enabled/orange.
+- TTL: Auto.
+
+Leave the permanent health record and unrelated records alone. The next normal watchdog start may report a stored-mode conflict because you changed DNS manually; resolve it with the explicit tested `--once --mode cloudflare --apply` operation.
+
+Changing DNS cannot repair home connectivity or instantly move already connected clients. Keep private backups and a known working route available during maintenance.

docs/PUBLISHING.md

@@ -0,0 +1,53 @@
+# Publish your copy on GitHub
+
+Publish only this sanitized source folder. Do not upload an older deployment folder, attachments, terminal logs, `runtime/`, system configuration, or backups. No real deployment values are needed on GitHub or in GitHub Actions secrets.
+
+## Use the prepared local repository
+
+The delivered local repository has a fresh `main` branch, a generic initial author, and no remote. It does not contain previous deployment history. From that repository folder:
+
+```bash
+git status --short
+git log -1 --format=fuller
+git ls-files
+```
+
+Review these outputs. The initial identity is a generic project identity, not a personal name/email. To use your own identity for future commits, configure a public display name and your GitHub-provided **noreply** email locally; do not paste a private email if you want it kept out of commits.
+
+Create an **empty** GitHub repository named `tls-edge-failover` (or your preferred name). Do not initialize it with another README, license, or `.gitignore`; these are already included. Choose its visibility in GitHub.
+
+Copy its HTTPS or SSH URL, then in Bash:
+
+```bash
+read -r -p 'GitHub repository URL: ' REPO_URL
+git remote add origin "$REPO_URL"
+git push -u origin main
+```
+
+Authenticate with your normal GitHub method. Do not put a token in the URL or a tracked file. These instructions assume no existing `origin`; inspect `git remote -v` before changing an established repository.
+
+## If you downloaded the source ZIP
+
+The ZIP contains the same tracked source, including `.github/` and `.gitignore`, but deliberately excludes `.git` history and runtime files. Extract it into a new folder. You can upload the extracted source using GitHub's web interface, ensuring hidden files are included, or initialize Git locally:
+
+```bash
+git init -b main
+git config user.name 'TLS Edge Failover contributors'
+git config user.email 'contributors@example.invalid'
+git add .
+git diff --cached --stat
+git diff --cached
+git commit -m 'Initial TLS edge failover source'
+```
+
+Use the repository URL/push steps above. You can substitute your preferred public identity and GitHub noreply email before the initial commit.
+
+## Before every publication
+
+- Review `git diff --cached` and `git ls-files` for domains, IPs, IDs, tokens, keys, logs, and local paths introduced by later changes.
+- Keep the tracked example JSON as placeholders; configure real installations in ignored paths.
+- Confirm runtime files are ignored with `git check-ignore runtime/watchdog.json runtime/cf-token runtime/watchdog.log runtime/haproxy.cfg`.
+- If secrets were previously committed, remove them from the history you intend to publish and revoke any real credential. Ignoring the filename does not repair existing history.
+- Keep GitHub Actions free of production DNS credentials. Its tests use fixtures and do not deploy anything.
+
+The included MIT license covers this source. Review it before publishing your copy. GitHub's **Actions** tab will show CI results after the first push; local test success does not mean a workflow has already run on GitHub.

docs/TROUBLESHOOTING.md

@@ -0,0 +1,106 @@
+# Troubleshooting
+
+## Oracle fails but nothing switches
+
+Look at the entire `health` event. Automatic fallback requires four consecutive failing **primary** checks and successful **local** and **canary** checks in the decision cycle. Also check `startup` contains `apply: true` and inspect `api_hold` or `fatal` events.
+
+An increasing failure counter alone does not authorize a switch to an unproven backup. No companion failover script is needed on Oracle; it only needs a working HAProxy service when that route is meant to be available.
+
+## Canary must traverse Cloudflare with cache bypassed
+
+An HTTP 200 from NPM alone is insufficient. The backup check must actually cross Cloudflare, have `CF-Ray`, and have `CF-Cache-Status: DYNAMIC` or `BYPASS`.
+
+For an investigation on the home Linux server, run these in Bash:
+
+```bash
+read -r -p 'Health hostname: ' HEALTH_HOST
+getent ahosts "$HEALTH_HOST"
+dig @1.1.1.1 "$HEALTH_HOST" A +short
+curl --noproxy '*' -sS -D - --max-time 15 \
+  -w '\nConnected to: %{remote_ip}\n' \
+  "https://$HEALTH_HOST/__edge_health?check=$(date +%s)"
+```
+
+If the local answer and curl destination are the NPM LAN IP while public DNS returns Cloudflare addresses, split DNS is bypassing Cloudflare. The DNS record being orange in the dashboard does not override your local resolver.
+
+The corrected watchdog solves this internally: only its canary uses authenticated public DNS-over-HTTPS. It keeps normal local DNS untouched. Update `watchdog.py` using [Operations](OPERATIONS.md#updating-an-existing-installation), stop the old process, run `--check`, then restart it. Plain curl without `--resolve` will still follow your system resolver; that does not mean the corrected watchdog uses it.
+
+If an actual Cloudflare response is returned, inspect the Cache Rule, cache status, WAF/challenge/Access settings, Worker routes, redirects, and origin rules on this dedicated hostname. Do not pin a Cloudflare website IP permanently in `/etc/hosts`; those addresses can change.
+
+## Public canary DNS lookup failed / deadline exceeded
+
+The canary requires outbound TCP 443 to `1.1.1.1`, verified TLS for `cloudflare-dns.com`, and a successful public A answer. Check firewall filtering, intercepting proxies, system CA certificates, and system time. The process bypasses proxy environment variables.
+
+The five-second default is a total budget for the lookup plus the health connection. A persistently slow but otherwise valid route may need a reviewed increase of `health.timeout_seconds` in private config (supported range: 1–15 seconds). Do not disable verification or accept a local DNS fallback.
+
+## Local check fails but primary works
+
+The watchdog belongs **at home**, where it can reach NPM's LAN address. On Oracle, that private IP normally has no route to your home LAN. Confirm `health.local_ip` is NPM's existing address, home routing permits TCP 443, and the NPM certificate covers `health.primary_host`.
+
+From home, force the NPM destination while preserving the hostname:
+
+```bash
+read -r -p 'Health hostname: ' HEALTH_HOST
+read -r -p 'NPM LAN IPv4: ' NPM_IP
+curl --noproxy '*' --fail --show-error --max-time 15 \
+  --resolve "$HEALTH_HOST:443:$NPM_IP" \
+  "https://$HEALTH_HOST/__edge_health"
+```
+
+This is a local diagnostic, not proof that the Cloudflare path works.
+
+## Primary check fails
+
+On Oracle, inspect without changing policy:
+
+```bash
+sudo systemctl status haproxy --no-pager
+sudo journalctl -u haproxy -n 50 --no-pager
+sudo ss -lntp
+sudo haproxy -c -f /etc/haproxy/haproxy.cfg
+sudo iptables -S
+sudo ip6tables -S
+sudo nft list tables
+```
+
+Check Oracle's applicable OCI network rules, host INPUT rule order, HAProxy's home address, and the home NAT/source allowlist. An ACCEPT below an unconditional REJECT/DROP will not permit new traffic. ICMP ping success does not prove TCP 443 works.
+
+To locate missing traffic, capture the relevant interface and protocol. Oracle HTTPS enters the home **WAN** first, then forwarded traffic reaches LAN/NPM. A LAN-only ICMP capture cannot establish that a WAN TCP 443 request never arrived. Filter for the actual Oracle source and TCP port 443 on WAN, then inspect the translated destination on LAN. This repository does not configure WireGuard.
+
+## Oracle firewall or installer stops
+
+The automatic installer targets a dedicated Ubuntu 24.04 VM. Preflight runs before apt installation, service masking, and replacing configuration. It stops for active firewall managers, installed UFW even if inactive, Docker, recognized leftover UFW/Docker/edge chains, or unexpected native nftables tables.
+
+This avoids package conflicts removing an installed-but-inactive firewall package and discovering leftover chains only after services were changed. Package installation also uses `--no-remove`.
+
+On an existing or partly configured VM:
+
+1. Preserve the working SSH session and establish console recovery access.
+2. Inspect the rules/services above and installed firewall packages.
+3. Back up both live rulesets privately before any changes:
+
+   ```bash
+   sudo sh -c 'umask 077
+   iptables-save > /root/edge-before.v4
+   ip6tables-save > /root/edge-before.v6'
+   ```
+
+4. Determine which existing manager owns policy. Use that manager or a deliberately reviewed migration. Do not flush all tables, delete OCI InstanceServices OUTPUT rules, or remove arbitrary chains/ports.
+5. If earlier installation output failed before the firewall helper armed a timer, no rollback timer exists for that attempt. Check actual state rather than assuming a timer is running.
+
+A narrow TCP 443 INPUT fix may restore connectivity on a legacy iptables-managed host, but it does not prove the host's SSH/IPv6 policy is fully hardened. Review the whole policy separately. This guide intentionally does not provide a universal destructive firewall reset.
+
+The timed helper restores pre-change rules after 180 seconds unless you complete the printed verification steps. If the deadline elapsed, inspect the actual head jumps and rollback status before saving. Do not rerun the full package/configuration installer just to repair one rule.
+
+## DNS identity, mixed state, or API errors
+
+- **Replaced record ID / conflicting address record:** inspect the actual names and IDs, including AAAA, CNAME, HTTPS, and SVCB conflicts. Do not delete unrelated records indiscriminately.
+- **Mixed startup mode / state disagrees:** stop the writer, inspect DNS and your chosen healthy destination, then use the explicit manual-mode command in [Operations](OPERATIONS.md#manual-switch-or-repair).
+- **Cloudflare HTTP 401/403:** verify token validity, zone ID, permission, and zone scope privately.
+- **429:** let the controller respect Retry-After/backoff; restarting repeatedly defeats this protection.
+- **Another watchdog owns this state file:** stop the existing process and wait for exit. Deleting the lock file while it is held can let two writers run.
+- **Canary drift in fixed-IP mode:** restore its intended proxied/home/Auto state or coordinate an actual home-IP change. The fixed-IP writer never silently repairs the permanent canary.
+
+## It has not switched back yet
+
+Keep Oracle healthy and the home watchdog running. A single primary success is not enough: five continuous healthy minutes and ten minutes of backup dwell must both pass. Restarting the watchdog restarts these windows. Check `local` remains healthy, API operations succeed, and no fatal event stopped the process. API readback success still leaves resolver-cache delay.

docs/VALIDATION.md

@@ -0,0 +1,16 @@
+# Validation of the initial source
+
+Local validation completed with Python 3.14.7 on macOS:
+
+- **49 automated tests passed**, using mocked network/API calls and isolated fake firewall commands.
+- All four Bash scripts passed individual `bash -n` checks.
+- The generalized renderer was tested for bounded/escaped SNI matching, rejected injection/invalid addresses, private output permissions, and replacement of existing output.
+- Both the default apex/wildcard wizard and a custom selected-app/health-host configuration were tested without remote DNS writes.
+- The public-DNS canary, certificate/body/cache requirements, failure/recovery timing, record boundaries, state reconciliation, and total deadlines retained their regression coverage.
+- Read-only preflight was tested against clean rules, active UFW, installed inactive UFW, leftover chains, and a native nftables table.
+
+The GitHub Actions workflow is configured to run tests on Python 3.9, 3.12, and 3.14 under Ubuntu 24.04, and to parse a rendered configuration with that distribution's HAProxy. Those GitHub jobs have not run merely because the source was prepared locally.
+
+No live Oracle, Cloudflare, OPNsense, or NPM configuration was changed during repository preparation. HAProxy/systemd deployment and the timed live firewall rollback were not exercised on this macOS machine. Complete the installation checks and an external application outage drill before relying on a deployment.
+
+The publication copy uses generic identifiers and excludes runtime/configuration data, secrets, logs, captures, and previous deployment history. Review future additions and commits before publishing; a static initial scan cannot guarantee later commits stay private.