Initial commit

2026-09-15 18:52:00 -05:00
commit 3b5f3b0a16
@@ -0,0 +1,4 @@
 * text=auto eol=lf
 *.sh text eol=lf
 *.py text eol=lf
 *.md text eol=lf
@@ -0,0 +1,6 @@
 version: 2
 updates:
  - package-ecosystem: github-actions
    directory: /
    schedule:
      interval: monthly
@@ -0,0 +1,49 @@
 name: Checks
 on:
  push:
  pull_request:
 permissions:
  contents: read
 jobs:
  python:
    runs-on: ubuntu-24.04
    timeout-minutes: 10
    strategy:
      fail-fast: false
      matrix:
        python: ['3.9', '3.12', '3.14']
    steps:
      - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
        with:
          persist-credentials: false
      - uses: actions/setup-python@83679a892e2d95755f2dac6acb0bfd1e9ac5d548 # v6.1.0
        with:
          python-version: ${{ matrix.python }}
      - name: Regression tests without network or credentials
        run: python -m unittest discover -s tests -v
      - name: Shell syntax
        run: |
          for script in setup-oracle.sh setup-home.sh install-firewall.sh preflight-oracle.sh; do
            bash -n "$script"
          done
  haproxy:
    runs-on: ubuntu-24.04
    timeout-minutes: 10
    steps:
      - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
        with:
          persist-credentials: false
      - name: Install configuration validator
        run: |
          sudo apt-get update
          sudo apt-get --no-remove install -y haproxy
      - name: Parse fixture configuration without starting the relay
        run: |
          python3 configure-oracle.py --domain example.com --home-ip 8.8.8.8
          sudo haproxy -c -f runtime/haproxy.cfg
      - name: Verify generated files remain ignored
        run: git check-ignore runtime/haproxy.cfg
@@ -0,0 +1,51 @@
 # Deployment data: never publish these directories.
 runtime/
 secrets/
 private/
 backups/
 backup.*/
 local/
 # Filled-in configuration and credentials. Examples/templates stay tracked.
 watchdog.json
 state.json
 state.json.lock
 haproxy.cfg
 cf-token
 cf_token
 edge-origin-secret
 edge_origin_secret
 .env
 .env.*
 *.local.*
 *.pem
 *.key
 *.crt
 *.p12
 *.pfx
 id_rsa*
 id_ed25519*
 # Logs, captures, archives, editor/runtime artifacts.
 *.log
 *.pid
 *.pcap
 *.pcapng
 *.bak
 *.backup
 *.zip
 *.tar
 *.tar.gz
 *.tgz
 *.bundle
 __pycache__/
 *.py[cod]
 .venv/
 venv/
 .pytest_cache/
 .coverage
 .DS_Store
 .idea/
 .vscode/
 *~
 *.swp
@@ -0,0 +1,15 @@
 # Changelog
 ## 0.1.0 — Initial public source
 - Generalized domain, Oracle IP, home IP, and health-host configuration; added a private HAProxy template renderer.
 - Preserved existing private watchdog configuration compatibility for upgrades.
 - Fixed backup health checks bypassing Cloudflare under home split DNS by using authenticated public DNS-over-HTTPS, with no LAN fallback.
 - Kept certificate, exact-body, no-store, Cloudflare-header, and uncached-response checks.
 - Added fixed-home-IP mode where the canary is read-only and only configured application A records are updated.
 - Supported apex/wildcard record management while leaving explicit unrelated names and mail records alone.
 - Preserved total network deadlines, API backoff, readback verification, atomic state, process locking, and conservative recovery timers.
 - Moved Oracle firewall conflict detection ahead of package/config changes; refused installed inactive UFW and disabled package removal during installation.
 - Retained timed firewall rollback and existing OCI OUTPUT rules.
 - Added regression tests, CI, installation/operations guides, diagrams, and an explicit explanation of Cloudflare TLS termination.
 - Excluded installation data from source and provided fresh publication instructions.
@@ -0,0 +1,34 @@
 # Contributing
 Use Python 3.9+ on Linux/macOS and Bash for the shell helpers. No Python package installation is needed. Use a maintained Python release for live deployments; compatibility with older Python is not an OS security-support promise.
 ```bash
 python3 -m unittest discover -s tests -v
 for script in setup-oracle.sh setup-home.sh install-firewall.sh preflight-oracle.sh; do
  bash -n "$script" || break
 done
 ```
 The tests cover route validation, state transitions, API ambiguity/readback, deadlines, fixed-IP DNS ownership, wildcard validation, split-DNS avoidance, configuration rendering, private file permissions, and firewall preflight. Network/API calls and firewall commands are mocked. They neither contact the fixture addresses nor require credentials or root.
 If HAProxy is available, render a temporary test configuration and run its syntax validator. The public IP below is only a parser fixture, not a deployment destination:
 ```bash
 python3 configure-oracle.py --domain example.com --home-ip 8.8.8.8
 sudo haproxy -c -f runtime/haproxy.cfg
 ```
 Do not start the sample configuration as a real service. GitHub Actions performs this parser check on Ubuntu 24.04. Tests do not replace checking real NPM certificates, Cloudflare settings, OCI networking, or timed firewall recovery during deployment.
 Preserve these invariants when contributing:
 - No DNS writes without explicit `--apply`.
 - No token or origin-secret forwarding to public health endpoints.
 - No accepting failed certificates, cached health pages, or local split DNS as a Cloudflare canary success.
 - Only configured record identities are eligible for writes.
 - No blanket firewall flushes, package removals, or loss of OCI OUTPUT protections.
 - No installed deployment values, credentials, logs, or captures in source/history.
 The example JSON is deliberately invalid until placeholders are replaced; use the interactive wizard to generate private configuration. Do not fill in the tracked example for a real deployment.
 Submit focused changes with relevant tests and a description of the changed behavior. Follow [SECURITY.md](SECURITY.md) when reporting vulnerabilities.
@@ -0,0 +1,21 @@
 MIT License
 Copyright (c) 2026 Aryan Singh
 Permission is hereby granted, free of charge, to any person obtaining a copy
 of this software and associated documentation files (the "Software"), to deal
 in the Software without restriction, including without limitation the rights
 to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
 copies of the Software, and to permit persons to whom the Software is
 furnished to do so, subject to the following conditions:
 The above copyright notice and this permission notice shall be included in all
 copies or substantial portions of the Software.
 THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
 IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
 FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
 AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
 LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
 OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
 SOFTWARE.
@@ -0,0 +1,94 @@
 # TLS Edge Failover
 A small HTTPS relay with automatic DNS failover for services hosted at home.
 An Oracle Cloud VM forwards encrypted TCP traffic to your existing Nginx Proxy Manager (NPM). A Python watchdog at home checks both routes and switches selected Cloudflare DNS records when the Oracle route fails. It switches back after sustained recovery.
 **Cloudflare terminates TLS whenever its orange-cloud proxy is used.** This project keeps Cloudflare out of the normal application traffic path, but the backup route lets Cloudflare decrypt HTTP. It does not provide encryption that hides HTTP from the Cloudflare proxy.
 ## Two routes
 ### Normal: Oracle, DNS only / grey cloud
 ```mermaid
 flowchart LR
    V[Visitor] -->|Encrypted HTTPS| O[Oracle: HAProxy TCP relay]
    O -->|Same TLS session| F[Home: OPNsense WAN 443]
    F --> N[NPM: TLS ends here]
    N --> A[Application]
 ```
 Cloudflare answers DNS with the Oracle address. The visitor then connects directly to Oracle. Oracle forwards TLS bytes without a certificate or private key. It can see network metadata and ordinary TLS SNI, but not the HTTP content under normal verified TLS. See [Cloudflare proxy status](https://developers.cloudflare.com/dns/proxy-status/).
 ### Backup: Cloudflare, proxied / orange cloud
 ```mermaid
 flowchart LR
    V[Visitor] -->|TLS session 1| C[Cloudflare: TLS ends here]
    C -->|TLS session 2: Full strict| F[Home: OPNsense WAN 443]
    F --> N[NPM: origin TLS ends here]
    N --> A[Application]
 ```
 The A records now contain the home address and have proxying enabled. Oracle is bypassed. Cloudflare can read HTTP requests and responses, including application credentials and cookies. [Cloudflare describes these as two separate connections](https://developers.cloudflare.com/ssl/get-started/).
 | Property | Oracle mode | Cloudflare backup |
 | --- | --- | --- |
 | Managed A record content | Oracle public IPv4 | Home public IPv4 |
 | Proxy status | DNS only | Proxied |
 | TLS termination | Home NPM | Cloudflare, then NPM on a separate connection |
 | Cloudflare WAF / Access in the path | No | Yes, if configured |
 | Oracle in the path | Yes | No |
 Putting an **orange cloud on the Oracle address** creates a third route: visitor → Cloudflare → Oracle → home. Cloudflare still terminates visitor TLS in that arrangement. The watchdog does not select that mode.
 ## What it includes
 - TLS passthrough with domain/SNI checks and connection limits.
 - Three HTTPS checks: direct Oracle, local NPM, and the public Cloudflare backup.
 - Public DNS-over-HTTPS for the backup check, so home split DNS cannot silently bypass Cloudflare.
 - Failover after four consecutive primary failures, provided local NPM and the backup route work.
 - Automatic return after five continuous healthy minutes and a ten-minute minimum stay in backup. Restarts reset these timers conservatively.
 - Exact DNS record IDs, conflict checks, API backoff, total network deadlines, a single-process lock, and API readback after changes.
 - Hidden token entry, private runtime files, and an optional sandboxed systemd service.
 - An optional Ubuntu 24.04 Oracle installer with read-only firewall preflight and timed firewall rollback.
 The default profile manages **only the apex and wildcard A records**. It leaves the permanent health record, mail records, and explicit unrelated records alone. Use repeated `--app` options during configuration to manage a smaller list.
 ## Start here
 1. Read [Installation](docs/INSTALL.md) for a new deployment.
 2. Already running an earlier version? Use [Updating an existing installation](docs/OPERATIONS.md#updating-an-existing-installation). Keep the existing private configuration.
 3. Read [Design](docs/DESIGN.md) for probes, DNS ownership, and failover timing.
 4. Read [Security](SECURITY.md) for exactly what is encrypted, exposed, and trusted.
 5. Use [Operations](docs/OPERATIONS.md) and [Troubleshooting](docs/TROUBLESHOOTING.md) for logs, recovery, and outage drills.
 ## Requirements and scope
 - Existing working NPM HTTPS hosts behind a home TCP 443 port-forward.
 - Publicly trusted certificates on NPM; certificate renewal must work without opening Oracle port 80. DNS-01 is suitable.
 - A reserved Oracle public IPv4; the provided installer targets a dedicated Ubuntu 24.04 VM, including ARM64.
 - A home Linux/Unix host with Python 3.9+ and network access to NPM, Oracle, Cloudflare's API, and public DNS-over-HTTPS. A maintained Python release is recommended for deployment.
 - A Cloudflare zone and a zone-scoped DNS-edit API token.
 - A stable home public IPv4 for the documented profile. Home-IP changes require coordinated configuration changes.
 This is HTTPS over TCP 443. It does not install a forward proxy, VPN, tunnel, HTTP port 80 redirect, UDP/QUIC relay, or public IPv6 listener. It cannot fix a home Internet outage or migrate existing connections during DNS failover. A static NPM health page does not test every application.
 ## Development
 No pip dependencies are required.
 ```bash
 python3 -m unittest discover -s tests -v
 for script in setup-oracle.sh setup-home.sh install-firewall.sh preflight-oracle.sh; do
  bash -n "$script" || break
 done
 ```
 Tests mock DNS/API/HTTPS and firewall commands. GitHub Actions also validates a rendered configuration with Ubuntu's HAProxy; CI does not deploy infrastructure. See [Contributing](CONTRIBUTING.md).
 ## Publishing your copy
 This repository contains generic examples. Setup writes installation-specific files to ignored `runtime/` or system directories. **Do not publish runtime folders, tokens, certificates, logs, backups, or filled-in configuration.** Git ignores prevent accidental staging, not deliberate `git add -f` or secret disclosure in issues.
 See [Publishing](docs/PUBLISHING.md) for exact GitHub upload steps. Source is licensed under [MIT](LICENSE).
@@ -0,0 +1,78 @@
 # Security model
 ## Where TLS ends
 **Oracle mode:** the browser authenticates NPM's publicly trusted certificate. HAProxy forwards the encrypted TCP stream without loading a TLS private key. TLS continues through the home firewall and ends at NPM. Encryption from NPM to each application depends on your existing upstream configuration; this project does not change it.
 **Cloudflare backup:** the browser authenticates a Cloudflare edge certificate. Cloudflare decrypts HTTP, then opens a separate TLS connection to NPM using Full (strict). Cloudflare can read requests, responses, cookies, and credentials carried in HTTP. Its ability to inspect traffic is inherent to the standard proxy. [Cloudflare's connection model](https://developers.cloudflare.com/ssl/get-started/).
 An orange-cloud record pointing at Oracle still terminates browser TLS at Cloudflare. Adding a TCP relay after Cloudflare cannot undo that termination. If hiding application contents from Cloudflare is a requirement, the Cloudflare HTTP proxy cannot serve as that application's fallback in this design. Application-level end-to-end encryption would be a separate feature of the application, not a feature provided here.
 ## What each party can see
 | Party | Visibility / authority |
 | --- | --- |
 | Oracle relay operator | Client/home IPs, ports, timing, connection sizes, and ordinary visible TLS SNI; can drop or disrupt traffic |
 | Cloudflare in normal mode | Authoritative DNS/control-plane information; no normal application HTTP connection through its proxy |
 | Cloudflare in backup mode | HTTP plaintext at its edge plus network metadata |
 | Home NPM | HTTP plaintext after TLS termination in either mode |
 | Application | Its own requests and whatever NPM forwards |
 | Home watchdog | Local token/config/state and health results; authority to edit the scoped zone's DNS through its token |
 “Oracle sees nothing” is incorrect. The narrower claim is that a relay without an accepted certificate/private key cannot normally decrypt a properly verified TLS session. This is not protection against compromised clients, NPM, certificate authorities, DNS control, or stolen signing keys. Cloudflare remains a trusted DNS control-plane provider even when its proxy is bypassed. [HAProxy TCP/TLS routing controls](https://docs.haproxy.org/2.8/configuration.html).
 ## Network boundaries
 - Home WAN 443 accepts the Oracle source IP and Cloudflare source ranges permanently. The existing forward still ends at NPM.
 - This is a network source restriction. Cloudflare's shared address ranges **do not authenticate your specific zone or account**. Do not treat them as per-tenant origin authentication.
 - Oracle accepts public TCP 443. Its SNI allowlist only checks a public hostname and is not authentication. Anyone can send an allowed SNI.
 - Oracle is a fixed-destination TCP relay; clients cannot choose an arbitrary Internet forwarding destination.
 - The optional Oracle firewall restricts new SSH to the current administrator `/32`, blocks other new TCP services, preserves OCI OUTPUT protections, and stages changes with a rollback timer. It retains existing established connections.
 - The installer refuses supported signs of conflicting firewall managers before package changes. This is a conservative compatibility guard, not a complete audit of every possible host configuration.
 - Public TCP 443 traffic in Oracle mode bypasses Cloudflare WAF, Access, bot checks, and HTTP rate limiting. Protect applications with authentication that works at NPM/the application on both routes.
 Keep home allowlists current using [Cloudflare's official IP lists](https://www.cloudflare.com/ips/). Additional per-zone origin authentication needs a separately designed policy compatible with direct Oracle traffic. Requiring Cloudflare client certificates unconditionally on NPM would break ordinary clients arriving through the passthrough relay. This project does not silently install shared-header or mTLS authentication.
 ## Client IP handling
 The relay opens a new TCP connection to NPM, so NPM sees Oracle as the source in normal mode. The configuration deliberately does not enable PROXY protocol because an ordinary NPM HTTPS listener does not accept it automatically.
 Do not trust client-supplied `X-Forwarded-For` or `CF-Connecting-IP` just because a request arrived from Oracle. Attackers can send those HTTP headers through the relay. Cloudflare real-IP handling must trust only actual Cloudflare peers using correctly maintained ranges. This kit does not change your existing real-IP policy. Per-user application authentication should not depend on NPM seeing the original client IP.
 ## Certificates and transport
 - NPM requires valid, publicly trusted certificates for direct clients and all probe hostnames. Cloudflare Origin CA alone is not browser trust. [Origin CA documentation](https://developers.cloudflare.com/ssl/origin-configuration/origin-ca/).
 - Use Full (strict) for fallback, never Flexible or verification-disabled probing. [Full (strict) requirements](https://developers.cloudflare.com/ssl/origin-configuration/ssl-modes/full-strict/).
 - The watchdog uses the system CA store, validates hostname/SNI, requires TLS 1.2 or newer for its connections, and does not follow HTTP redirects.
 - Set application TLS policy and certificate renewal at NPM. The relay does not impose a TLS version policy on client sessions.
 - The SNI parser expects an ordinary TLS ClientHello. Missing SNI or encrypted-client-hello combinations that hide the allowed name may be rejected. Public HTTP/3/QUIC is not relayed. Check inherited HTTPS/SVCB records and alternative-service advertisements before relying on clients that use those features.
 ## Watchdog safeguards and their limits
 The controller checks exact record identities, rejects conflicting address records, and updates only the configured application A records in the documented fixed-IP mode. Unexpected identities or mixed startup state stop the writer. It requires explicit `--apply`, verifies candidate routes, persists intent atomically, and reads the API back after changes.
 The backup probe uses authenticated public DNS-over-HTTPS instead of local DNS overrides. It requires the correct health body, a noncached response, and Cloudflare response headers. These are operational route checks, not cryptographic attestation of the entire application. A static health marker proves NPM and that route are responding; an individual app can still be broken.
 DNS responses and Cloudflare's API remain trusted inputs. API readback does not prove all resolver caches have converged. Batch changes may become visible at different times across Cloudflare's distributed DNS. [Batch behavior](https://developers.cloudflare.com/dns/manage-dns-records/how-to/batch-record-changes/).
 Timeouts, backoff, hysteresis, a process lock, and conservative restart handling reduce accidental switching. They do not create a distributed consensus system. Run one writer per configuration. A home-side measurement can mistake a path-specific home-to-Oracle problem for a general outage. There is no independent external quorum.
 ## Secrets and publication
 The Oracle VM needs no Cloudflare token and no application TLS private keys. The home watchdog's token should have DNS-edit permission for one zone only. Its record checks do not constrain a stolen token outside the program.
 Local configuration lives in `runtime/` with a mode-0700 directory and mode-0600 files. The optional systemd deployment uses root-owned credentials and a separate service account. Root and the relevant local user still have access; file permissions do not protect a compromised host.
 Runtime/config files, certificates, keys, logs, backups, and environment files are ignored by Git. Logs intentionally omit tokens and API response bodies but can contain hostnames, IP addresses, and paths. Redact them before sharing. The source examples use placeholders; tests use synthetic names/IDs and mock all external traffic.
 Before publishing, review staged files and Git history. A `.gitignore` cannot remove data already committed. If a real token is exposed, revoke it; deleting the visible text does not revoke the credential.
 ## Availability and policy
 This setup retains a public home port-forward. It does not hide your home address from Oracle, Cloudflare, historical DNS, or every possible application leak. It does not provide Cloudflare's DDoS protection on the Oracle path, eliminate single points of failure, or guarantee uninterrupted streams.
 Choose fallback-eligible services according to the applicable Cloudflare product and plan. A wildcard can include large downloads or video unintentionally; select explicit application records when needed. TLS passthrough at Oracle does not bypass Cloudflare's policies during backup. [Cloudflare's video/non-HTML guidance](https://developers.cloudflare.com/fundamentals/reference/policies-compliances/delivering-videos-with-cloudflare/).
 ## Reporting vulnerabilities
 Do not post secrets or deployment identifiers in public issues. If the repository owner enables private vulnerability reporting, use the repository's **Security → Report a vulnerability** feature. Otherwise request a private contact method before sending sensitive details. This repository includes automated regression checks, not a third-party security audit or guarantee.
@@ -0,0 +1,109 @@
 #!/usr/bin/env python3
 """Interactive local configuration. Reads Cloudflare DNS; never changes DNS."""
 import getpass
 import argparse
 import grp
 import ipaddress
 import json
 import os
 from pathlib import Path
 import re
 import sys
 import tempfile
 sys.path.insert(0, str(Path(__file__).resolve().parent))
 from watchdog import APIError, Cloudflare, SafetyError, canonical_name, public_v4, load_config, observed_mode, validate_inventory
 def private_write(path, text, mode, gid=None):
    fd, tmp = tempfile.mkstemp(prefix=".configure-", dir=path.parent)
    try:
        with os.fdopen(fd, "w", encoding="utf-8") as stream:
            os.fchmod(stream.fileno(), mode)
            if gid is not None:
                os.fchown(stream.fileno(), 0, gid)
            stream.write(text)
            stream.flush()
            os.fsync(stream.fileno())
        os.replace(tmp, path)
    finally:
        if os.path.exists(tmp):
            os.unlink(tmp)
 def main():
    parser = argparse.ArgumentParser(description=__doc__)
    parser.add_argument("--local", action="store_true", help="Save private runtime files beside this script; no sudo/systemd needed")
    parser.add_argument("--health-host", help="Permanent proxied health hostname (default: edge-health.<domain>)")
    parser.add_argument("--app", action="append", help="Existing A record to manage; repeat as needed (default: apex and wildcard)")
    args = parser.parse_args()
    if not args.local and os.geteuid() != 0:
        raise SafetyError("Use --local without sudo, or run the optional system installer")
    directory = Path(__file__).resolve().parent / "runtime" if args.local else Path("/etc/oracle-edge")
    directory.mkdir(mode=0o700, parents=True, exist_ok=True)
    if args.local:
        os.chmod(directory, 0o700)
    domain = canonical_name(input("Cloudflare domain (for example, example.com): ").strip())
    oracle_ip = public_v4(input("Oracle reserved public IPv4: ").strip())
    home_ip = public_v4(input("Current home public IPv4: ").strip())
    if oracle_ip == home_ip:
        raise SafetyError("Oracle and home IPv4 addresses must differ")
    health_host = canonical_name(args.health_host or "edge-health." + domain)
    apps = [canonical_name(name, allow_wildcard=True) for name in (args.app or ["*." + domain, domain])]
    if any(name != domain and not name.endswith("." + domain) for name in [health_host, *apps]):
        raise SafetyError("All configured names must belong to the chosen domain")
    print("Only these application A records will switch: " + ", ".join(apps))
    zone = input("Cloudflare Zone ID for " + domain + ": ").strip()
    if not re.fullmatch(r"[a-fA-F0-9]{32}", zone):
        raise SafetyError("Zone ID must be 32 hexadecimal characters")
    token = getpass.getpass("Cloudflare API token (input hidden): ").strip()
    if not token or len(token) > 4096 or any(ord(c) < 33 or ord(c) > 126 for c in token):
        raise SafetyError("Invalid API token format")
    local_ip = input("Existing NPM LAN IPv4 (read it from your current OPNsense forward): ").strip()
    address = ipaddress.ip_address(local_ip)
    private_ranges = [ipaddress.ip_network(net) for net in ("10.0.0.0/8", "172.16.0.0/12", "192.168.0.0/16")]
    if address.version != 4 or not any(address in net for net in private_ranges):
        raise SafetyError("Enter NPM's actual private LAN IPv4, not Oracle or your public home IP")
    cfg = {
        "zone_id": zone, "oracle_ip": oracle_ip, "home_ip": home_ip,
        "health": {"primary_host": health_host, "local_ip": str(address),
                   "path": "/__edge_health", "expected_body": "edge-ok-v1\n", "timeout_seconds": 5},
        "token_file": str(directory / "cf-token"),
        "state_file": str(directory / "state.json") if args.local else "/var/lib/oracle-edge/state.json",
        "interval_seconds": 15, "failures_required": 4, "recovery_seconds": 300,
        "minimum_dwell_seconds": 600, "api_timeout_seconds": 10,
    }
    api = Cloudflare(cfg, token)
    inventory = api.inventory()
    def reference(name):
        selected = [r for r in inventory if r.get("name", "").lower().rstrip(".") == name and r.get("type") == "A"]
        if len(selected) != 1:
            raise SafetyError(f"Expected exactly one existing A record for {name}")
        return {"name": name, "id": selected[0]["id"]}
    cfg["canary"] = reference(health_host)
    cfg["apps"] = [reference(name) for name in apps]
    # Validate the complete schema before saving credentials or the active config.
    with tempfile.NamedTemporaryFile(mode="w", suffix=".json", encoding="utf-8") as candidate:
        json.dump(cfg, candidate)
        candidate.flush()
        cfg = load_config(candidate.name)
    by_id, _ = validate_inventory(cfg, inventory)
    mode = observed_mode(cfg, by_id)
    if mode == "mixed":
        raise SafetyError("Managed records disagree; put them in the same tested mode before setup")
    gid = None if args.local else grp.getgrnam("oracle-edge").gr_gid
    private_write(directory / "cf-token", token + "\n", 0o600)
    private_write(directory / "watchdog.json", json.dumps(cfg, indent=2) + "\n", 0o600 if args.local else 0o640,
                  gid)
    print(f"Local config saved. Observed DNS mode: {mode}. No DNS records were changed.")
    return 0
 if __name__ == "__main__":
    try:
        sys.exit(main())
    except (APIError, SafetyError, OSError, ValueError) as exc:
        print("Setup stopped: " + str(exc), file=sys.stderr)
        sys.exit(1)
@@ -0,0 +1,56 @@
 #!/usr/bin/env python3
 """Render a private HAProxy configuration; never installs it or changes a firewall."""
 import argparse
 import os
 from pathlib import Path
 import sys
 import tempfile
 sys.path.insert(0, str(Path(__file__).resolve().parent))
 from watchdog import SafetyError, canonical_name, public_v4
 def render(domain, home_ip):
    domain = canonical_name(domain)
    home_ip = public_v4(home_ip)
    # canonical_name permits only DNS label characters; dots need regex escaping.
    template = (Path(__file__).resolve().parent / "haproxy.cfg.template").read_text()
    result = template.replace("@@DOMAIN_REGEX@@", domain.replace(".", r"\."))
    result = result.replace("@@HOME_IPV4@@", home_ip)
    if "@@" in result:
        raise SafetyError("Unresolved HAProxy template placeholder")
    return result
 def main(argv=None):
    parser = argparse.ArgumentParser(description=__doc__)
    parser.add_argument("--domain", help="Domain served at home; its subdomains are also accepted")
    parser.add_argument("--home-ip", help="Home public IPv4 address")
    parser.add_argument("--output", type=Path, help="Default: ignored runtime/haproxy.cfg beside this script")
    args = parser.parse_args(argv)
    domain = args.domain or input("Domain served at home (for example, example.com): ").strip()
    home_ip = args.home_ip or input("Current home public IPv4: ").strip()
    content = render(domain, home_ip)
    path = args.output or Path(__file__).resolve().parent / "runtime" / "haproxy.cfg"
    path.parent.mkdir(mode=0o700, parents=True, exist_ok=True)
    fd, temporary = tempfile.mkstemp(prefix=".haproxy-", dir=path.parent)
    try:
        with os.fdopen(fd, "w", encoding="utf-8") as stream:
            os.fchmod(stream.fileno(), 0o600)
            stream.write(content)
            stream.flush()
            os.fsync(stream.fileno())
        os.replace(temporary, path)
    finally:
        if os.path.exists(temporary):
            os.unlink(temporary)
    print(f"Saved {path}. No services, firewall rules, or DNS records were changed.")
    return 0
 if __name__ == "__main__":
    try:
        sys.exit(main())
    except (SafetyError, OSError, ValueError) as exc:
        print("Setup stopped: " + str(exc), file=sys.stderr)
        sys.exit(1)
@@ -0,0 +1,102 @@
 # Design
 ## Separate traffic forwarding from route selection
 HAProxy on Oracle forwards TLS traffic. The watchdog at home decides which DNS route new clients should take. Oracle does not run the watchdog and cannot switch DNS by itself. Its normal systemd service starts HAProxy again after reboot.
 ```mermaid
 flowchart TB
    W[Home Python watchdog] -->|Verified HTTPS to fixed Oracle IP| O[Oracle relay]
    O -->|Home WAN 443| N[Home NPM health endpoint]
    W -->|Verified HTTPS to NPM LAN IP| N
    W -->|Public DoH lookup, then HTTPS| C[Permanent proxied canary]
    C -->|Cloudflare to home| N
    W -->|Scoped token: DNS API| D[Selected application A records]
 ```
 ## DNS ownership
 The interactive wizard creates a **fixed home IP** configuration. It discovers record IDs by reading the chosen Cloudflare zone. It does not create records or mutate remote DNS.
 | Record | Normal state | Backup state | Writer |
 | --- | --- | --- | --- |
 | `example.com` A | Oracle, DNS only, TTL 60 | Home, proxied, Auto | Watchdog |
 | `*.example.com` A | Oracle, DNS only, TTL 60 | Home, proxied, Auto | Watchdog |
 | `edge-health.example.com` A | Home, proxied, Auto | Same | Operator; read-only to fixed-IP watchdog |
 | Explicit unrelated CNAME/A/Worker records | Existing configuration | Existing configuration | Existing owner |
 | MX/TXT and other unrelated records | Existing configuration | Existing configuration | Existing owner |
 An explicit DNS name can override wildcard behavior. The wizard defaults to apex plus wildcard, but `--app` selects other existing A records. The controller never rewrites every record in the zone indiscriminately. See [Cloudflare wildcard behavior](https://developers.cloudflare.com/dns/manage-dns-records/reference/wildcard-dns-records/).
 For each managed name, it verifies the A record's exact ID/name and rejects a conflicting A, AAAA, CNAME, HTTPS, or SVCB record at that name. Optional exact comments can strengthen ownership checks. Unrelated records are read during inventory but not patched. Inspect explicit subdomains and Worker routes separately: checking the apex/wildcard does not audit every possible application route.
 ### Advanced origin-record mode
 The controller also supports replacing `home_ip` with an `origin` reference to a dedicated DNS-only DDNS A record. That record remains owned by your DDNS updater. In that mode the watchdog can update the canary to the current DDNS address and then reconcile the application records.
 The included Oracle HAProxy renderer still produces a fixed-IP backend. Enabling `origin` mode alone does **not** update Oracle's backend, arrange DDNS, or supply a complete dynamic-home-IP deployment. The installation guide intentionally uses fixed `home_ip`; do not configure both fields.
 ## Three independent HTTPS checks
 | Check | Socket destination | TLS SNI and HTTP Host | Purpose |
 | --- | --- | --- | --- |
 | `primary` | Configured Oracle IPv4:443 | Health hostname | Verifies Oracle → home → NPM regardless of current app DNS |
 | `local` | NPM LAN IPv4:443 | Health hostname | Verifies home NPM is available |
 | `canary` | Public A answer for permanent proxied health hostname | Canary hostname | Verifies Cloudflare → home → NPM |
 All require a valid certificate, status 200, the exact configured body including newline, and `Cache-Control: no-store`. Each request includes a random cache-busting query. The canary additionally requires `CF-Ray` and `CF-Cache-Status` of `DYNAMIC` or `BYPASS`.
 The canary resolver opens TLS to public `1.1.1.1:443` with SNI `cloudflare-dns.com` and requests the hostname's A records through DNS-over-HTTPS. It validates the resolver certificate and accepts only public IPv4 answers for the exact name. A lookup failure fails the probe; it does not fall back to LAN DNS. This prevents a local wildcard override from making a direct LAN response look like a working Cloudflare route. [Cloudflare DoH JSON API](https://developers.cloudflare.com/1.1.1.1/encryption/dns-over-https/make-api-requests/dns-json/).
 The chosen public destination changes with fresh DNS answers; it is not a hardcoded Cloudflare website IP. The original hostname is retained for TLS validation and HTTP routing. Outbound proxies from environment variables are not used for these checks or the DNS API.
 Probe jobs run in child processes with a total deadline, including DNS/TLS/body reading. The default health budget is five seconds, with API calls separately bounded at ten seconds. If one child stalls, the parent stops it. System clocks must be accurate for TLS validation; health-duration accounting uses monotonic time.
 ## Switching state machine
 ```mermaid
 stateDiagram-v2
    [*] --> Inspect
    Inspect --> Oracle: DNS and stored state agree on Oracle
    Inspect --> Cloudflare: DNS and stored state agree on backup
    Inspect --> Stopped: Mixed DNS, invalid identities, or state conflict
    Oracle --> Cloudflare: 4 consecutive primary failures AND local + canary healthy
    Cloudflare --> Oracle: Primary healthy for 300s AND backup dwell 600s AND local healthy
 ```
 With a 15-second interval, four failed samples usually require roughly a minute, plus API execution and resolver caching. This is not a hard recovery-time guarantee.
 Only return to Oracle is gated by the minimum dwell; a newly failed Oracle route can fail over without waiting ten minutes. Continuous healthy time and dwell both restart when the watchdog starts. A gap longer than twice the sample interval resets continuous health. An API failure also resets the health window and holds/backoffs rather than inventing a successful write.
 A healthy Cloudflare canary is required to move into fallback. It is not required to return to a healthy Oracle route. `--check` requires all three checks because it verifies readiness of the complete deployment. Manual `--once --mode … --apply` bypasses waiting periods but still requires valid record identities and a healthy local/selected destination.
 ## API writes and crash handling
 1. Read and validate the current zone inventory.
 2. Probe the routes and compute a desired mode.
 3. Save desired mode and timestamp atomically before a potentially ambiguous DNS write.
 4. Submit only required A-record PATCHes in a batch.
 5. Read DNS records back from the API and verify content, proxy status, and TTL.
 During a running process, lost responses or partial outcomes are reconciled against the persisted intent. On a fresh start, mixed records or stored-state disagreement require explicit operator recovery. The controller chooses caution over silently deciding that an unexpected DNS edit represents an outage.
 One file lock prevents concurrent controllers sharing a state file. The state write uses a private temporary file, `fsync`, and atomic replacement. These mechanisms assume a suitable local filesystem; they do not coordinate multiple independent watchdog hosts or distinct state files.
 Cloudflare API batching is not instantaneous worldwide DNS propagation. Clients may retain old answers, reuse existing connections, or take longer than the displayed TTL to move. [Cloudflare batch changes](https://developers.cloudflare.com/dns/manage-dns-records/how-to/batch-record-changes/), [TTL behavior](https://developers.cloudflare.com/dns/manage-dns-records/reference/ttl/).
 ## Relay limits
 HAProxy routes only an ordinary TLS ClientHello with a matching apex/subdomain SNI to one fixed home TCP 443 backend. No HTTP parser, certificate, private key, or PROXY protocol is configured at Oracle.
 The template limits concurrent connections and connection rates. Shared NAT clients can hit per-source limits, so adjust them based on measured use. Five-minute client/server timeouts are inactivity timeouts; reloads drain old processes for up to ten minutes before terminating remaining connections. The backend TCP check only tests connectivity; the home watchdog supplies the HTTPS/body test.
 ## Failures this cannot solve
 - Home power, NPM, WAN, or ISP failure: both routes still depend on home.
 - A stopped watchdog: the last DNS mode remains until someone runs a writer.
 - A DNS API/control-plane outage: route switching may be unavailable.
 - An application failure hidden behind a successful static NPM health page.
 - Every client's network path differing from the home server's observations.
 - In-flight streams/WebSockets: DNS changes affect future connections, not existing sessions.
 See [Security](../SECURITY.md) for trust boundaries and [Operations](OPERATIONS.md) for drills and repair.
@@ -0,0 +1,189 @@
 # Installation
 This guide keeps the existing home WAN 443 → NPM forwarding. It creates a relay on Oracle and runs the DNS watchdog at home. No DNS changes occur merely from running either configuration wizard.
 Already deployed? Follow [the update procedure](OPERATIONS.md#updating-an-existing-installation) instead of rerunning the firewall installer.
 ## 1. Collect the values
 | Value | Meaning | Where it is used |
 | --- | --- | --- |
 | Domain | Your domain managed in Cloudflare; called a **zone** there | DNS record names and TLS SNI |
 | Oracle IPv4 | Reserved public IPv4 assigned to your VM | Public DNS, home source allowlist, primary health probe |
 | Home IPv4 | Current public WAN IPv4 | Oracle's forwarding destination and Cloudflare backup |
 | NPM LAN IPv4 | The address already targeted by your home port-forward | The watchdog's local HTTPS check only; it does not change the forward |
 | Administrator CIDR | Your SSH source public IPv4 followed by `/32`, meaning one address | Restricts SSH on Oracle; the installer derives it from the SSH connection |
 | Zone ID | Cloudflare's identifier shown on the zone overview page | API configuration |
 `example.com` in this guide is a placeholder. Enter your own values when the scripts prompt. Do not put personal values into tracked source files.
 Use a reserved Oracle public IP so stop/start operations do not unexpectedly change your allowlist or DNS. The documented configuration assumes a fixed home address; see [IP changes](OPERATIONS.md#home-or-oracle-ip-changes).
 ## 2. Keep the home forward; add Oracle to its allowed sources
 In OPNsense, keep your existing TCP WAN 443 → NPM LAN 443 forward.
 1. Create a host alias, for example `oracle_edge`, containing your Oracle public IPv4.
 2. Include that alias alongside the existing Cloudflare IP ranges in the **source restriction that actually governs this forward**. You can use a combined alias with your existing Cloudflare aliases plus `oracle_edge`.
 3. Apply the change. Keep both Cloudflare and Oracle sources allowed permanently.
 4. Check both the NAT rule and its associated WAN pass rule. Adding an unused alias alone grants no access. A source restriction on the NAT rule must also include Oracle so translation occurs.
 Do not add another port-forward. Do not expose NPM's administration port 81. If your existing rules use a different layout, inspect them rather than replacing them with a generic rule. Keep the [official Cloudflare address lists](https://www.cloudflare.com/ips/) current. See [OPNsense NAT documentation](https://docs.opnsense.org/manual/nat.html).
 ## 3. Create one permanent Cloudflare health hostname
 Leave your currently working application DNS in its current mode while preparing the other route. For a new deployment that starts through Cloudflare, both managed application A records should contain your home IPv4 and be proxied.
 Add this record in Cloudflare DNS, substituting your own domain and home IP:
 | Type | Name | Content | Proxy status | TTL |
 | --- | --- | --- | --- | --- |
 | A | `edge-health` | Home public IPv4 | Proxied / orange | Auto |
 The resulting name is `edge-health.example.com`. It stays pointed at home even when applications use Oracle. Do not include it among the switchable records.
 Under **SSL/TLS**, use **Full (strict)**. Ensure both the Cloudflare edge certificate and NPM certificate cover all failover hostnames. Avoid any origin rule that sends this health hostname somewhere else or changes its SNI/Host. Existing Worker routes, redirects, or Access policies can also intercept requests.
 Full (strict) authenticates and encrypts the Cloudflare-to-home connection; it does not hide HTTP from Cloudflare. [Mode requirements](https://developers.cloudflare.com/ssl/origin-configuration/ssl-modes/full-strict/).
 ## 4. Add the NPM health host
 Create a separate **Proxy Host** in NPM:
 | Setting | Value |
 | --- | --- |
 | Domain name | `edge-health.example.com`, using your domain |
 | Scheme / forward host / port | `http` / `127.0.0.1` / `9` |
 | Access list | Publicly Accessible |
 | SSL certificate | Publicly trusted certificate covering this hostname |
 | Advanced | Contents of `npm-health-advanced.conf` |
 The forwarding address is an unused UI placeholder: the supplied configuration returns the health marker directly and rejects every other path. It must be on this dedicated health host, not your application hosts. Do not add custom locations that override it.
 Your existing valid wildcard certificate can cover this single-level health subdomain. A Cloudflare Origin CA-only certificate is unsuitable for this setup because the direct Oracle route must be trusted by ordinary clients. [Origin CA limitations](https://developers.cloudflare.com/ssl/origin-configuration/origin-ca/).
 In Cloudflare, create a **Cache Rule** matching this expression with Cache eligibility set to **Bypass cache**:
 ```text
 (http.host eq "edge-health.example.com" and http.request.uri.path eq "/__edge_health")
 ```
 Substitute your domain. Ensure this path does not receive a challenge, Access login, or redirect. Scope any exception to this harmless health hostname/path, preserving application protections. The response must be status 200, exactly `edge-ok-v1` followed by a newline, `Cache-Control: no-store`, `CF-Ray`, and `CF-Cache-Status: DYNAMIC` or `BYPASS`.
 ## 5. Prepare the Oracle VM
 Copy this source folder to the VM using your usual SSH/SCP method. Do not copy home credentials or `runtime/` from another machine. On Oracle, in the source folder:
 ```bash
 python3 configure-oracle.py
 ```
 Enter the domain and home public IPv4. This generates ignored `runtime/haproxy.cfg` with private file permissions.
 ### OCI network rules
 In the VM's applicable NSGs/security lists, allow **stateful** IPv4 ingress:
 - TCP 443 from `0.0.0.0/0`, because ordinary clients connect directly to Oracle.
 - TCP 22 from your administrator public IPv4 `/32`.
 Retain the egress needed for home TCP 443, DNS, package updates, and OCI operation. Audit the combined NSGs and security lists: another broad rule can still grant access. This profile does not require ingress on port 80, 3128, or UDP 443. See [OCI security rules](https://docs.oracle.com/en-us/iaas/Content/Network/Concepts/securityrules.htm).
 ### Dedicated Ubuntu 24.04 VM only
 Have an OCI console recovery method available. Keep your existing SSH session open. The installer expects IPv4 SSH on port 22 and refuses existing UFW/Docker/custom native firewall setups rather than removing their policy.
 ```bash
 sudo bash preflight-oracle.sh
 sudo env "SSH_CONNECTION=$SSH_CONNECTION" bash setup-oracle.sh
 ```
 If preflight stops, read [the firewall troubleshooting section](TROUBLESHOOTING.md#oracle-firewall-or-installer-stops). Do not disable a firewall simply to force the installer through.
 The installer validates HAProxy's configuration, enables the relay, and stages the host firewall with a **180-second rollback timer**. The host firewall permits administrator SSH and public TCP 443, essential ICMP/DHCP and established traffic, while blocking other new incoming services. It preserves existing OUTPUT rules, including OCI InstanceServices protections. Existing established sessions are intentionally retained.
 ### Complete the timed firewall verification
 Before the timer expires, open a **new** SSH session to Oracle from your administrator address. From the home server or another external terminal, enter the real values and test:
 ```bash
 read -r -p 'Health hostname: ' HEALTH_HOST
 read -r -p 'Oracle public IPv4: ' ORACLE_IP
 curl --noproxy '*' --fail --show-error --max-time 15 \
  --resolve "$HEALTH_HOST:443:$ORACLE_IP" \
  "https://$HEALTH_HOST/__edge_health"
 ```
 Run those commands in Bash. Require `edge-ok-v1` and a valid certificate. Also test a real NPM application with `--resolve` using its own hostname. Never add `-k` to bypass certificate verification.
 After successful tests, run the **exact timer-stop, rollback-status, chain-check, and persistence commands printed by the installer**. Timer names are generated for each run. Verify the rollback service is inactive and both head jumps still exist before saving with `netfilter-persistent save`.
 If you miss the deadline, the firewall restores its pre-change rules. Inspect the state before trying again; do not save blindly. The timer rolls back firewall rules, not installed packages or HAProxy configuration.
 ## 6. Create the Cloudflare API token
 Create a custom token with **Zone → DNS → Edit**, scoped to **only the zone being managed**. Record the Zone ID from that zone's overview page. The script requests existing DNS records by that known ID and does not need account administration privileges.
 Use a separate token from certificate-renewal/DDNS automation. Enter it into the hidden prompt; do not paste it into a command, tracked file, screenshot, or issue. Cloudflare's token resource scope is the zone; the script's per-record checks narrow its own actions but cannot reduce what a stolen zone token could do. [API token permissions](https://developers.cloudflare.com/fundamentals/api/reference/permissions/).
 ## 7. Configure and run at home
 Copy the source folder to an always-on **home** server. Run from that folder as one consistent user:
 ```bash
 python3 configure-home.py --local
 python3 watchdog.py --config "$PWD/runtime/watchdog.json" --check
 ```
 Enter your domain, Oracle IPv4, home IPv4, Zone ID, hidden token, and NPM's existing LAN IPv4. The wizard looks up exact record IDs and saves private local configuration. It does not change DNS or OPNsense.
 To switch only selected existing A records, use this in place of the first command (substitute your domain):
 ```bash
 python3 configure-home.py --local \
  --app app.example.com --app status.example.com
 ```
 Quote wildcard arguments, such as `--app '*.example.com'`, so the shell does not expand them. `--health-host` can select a different dedicated proxied health name within the domain.
 `--check` requires all three routes to pass. It resolves the Cloudflare canary using authenticated public DNS-over-HTTPS, independently of local split DNS. The home server must reach public `1.1.1.1:443`. Local NPM and Oracle checks use their explicit IPs with the correct TLS hostname.
 Once all checks pass, switch to Oracle:
 ```bash
 python3 watchdog.py --config "$PWD/runtime/watchdog.json" \
  --once --mode oracle --apply
 ```
 Verify the managed records now contain the Oracle IP, **DNS only**, TTL 60, and that real applications work from an external network. Explicit DNS records can override wildcard routing.
 Start the continuous watchdog:
 ```bash
 umask 077
 nohup python3 -u watchdog.py --config "$PWD/runtime/watchdog.json" \
  --run --apply > runtime/watchdog.log 2>&1 &
 echo $! > runtime/watchdog.pid
 tail -n 50 -F runtime/watchdog.log
 ```
 `Ctrl+C` exits the log viewer only. `nohup` survives terminal closure; it does **not** restart the watchdog after a server reboot or process failure. Use your home server's service supervisor for that, or the systemd option below. Do not run two copies. Without `--apply`, no DNS changes occur.
 ## 8. Test failover and automatic return
 Follow the [outage drill](OPERATIONS.md#outage-drill). Expect a switch after four failing Oracle probes only when the local and public Cloudflare probes are healthy. DNS caches add delay. Recovery is automatic after five continuous healthy minutes and a ten-minute minimum backup dwell; you do not rerun an Oracle script to switch back.
 ## Optional: systemd on a home Ubuntu 24.04 host
 Choose this instead of the plain `nohup` process. Stop any existing local watchdog first. This installer copies the scripts to system paths and prompts for a separate private configuration:
 ```bash
 sudo bash setup-home.sh
 sudo python3 -I /usr/local/lib/oracle-edge/watchdog.py \
  --config /etc/oracle-edge/watchdog.json --once --mode oracle --apply
 sudo chown -R oracle-edge:oracle-edge /var/lib/oracle-edge
 sudo systemctl enable --now oracle-edge-watchdog.service
 sudo journalctl -u oracle-edge-watchdog.service -n 50 -f
 ```
 The service runs as an unprivileged account and reads the token through systemd credentials. The installer does not start the writer until you enable it. Keep `/etc/oracle-edge` and `/var/lib/oracle-edge` out of your repository. For service-based updates and manual switches, see [Operations](OPERATIONS.md).
@@ -0,0 +1,187 @@
 # Operations
 Commands below assume a shell in the installed source folder on the **home server**, unless marked Oracle. Keep the same user and private configuration paths used during setup.
 ## View logs
 For the plain Python process:
 ```bash
 tail -n 50 -F runtime/watchdog.log
 ```
 For the optional systemd service:
 ```bash
 sudo journalctl -u oracle-edge-watchdog.service -n 50 -f
 ```
 `Ctrl+C` stops the viewer, not the watchdog. Important JSON events:
 | Event | Meaning |
 | --- | --- |
 | `startup` | Observed DNS mode and current intent; `apply` shows whether writes are enabled |
 | `health` | `primary`, `local`, and `canary` status plus consecutive primary failures |
 | `hold` | The desired destination was not proven healthy; DNS retained |
 | `dry_run` | Proposed action only; no DNS change |
 | `dns_verified` | API readback matches the desired DNS state; caches may still use the old route |
 | `api_hold` | DNS API error/backoff; inspect authentication, limits, or connectivity |
 | `fatal` | Configuration/identity/state conflict; the process stops for inspection |
 | `stopped` | Graceful shutdown completed |
 Logs may include deployment information. Redact IPs, domains, and paths before sharing them. Never share `cf-token` or unredacted configuration.
 ## Stop and start the watchdog
 For the `nohup` process, first inspect the stored PID:
 ```bash
 ps -p "$(cat runtime/watchdog.pid)" -o pid=,args=
 ```
 Only if it is this watchdog with the expected config, send SIGTERM:
 ```bash
 kill -TERM "$(cat runtime/watchdog.pid)"
 tail -n 20 runtime/watchdog.log
 ```
 Wait for `stopped` or confirm that PID has exited before running a manual command. A stale PID file can refer to another process; do not kill it without inspecting. The state lock also prevents a second instance from running against the same state file. Avoid `kill -9` during a DNS write because it can leave a state/DNS conflict to repair.
 Restart with:
 ```bash
 umask 077
 nohup python3 -u watchdog.py --config "$PWD/runtime/watchdog.json" \
  --run --apply > runtime/watchdog.log 2>&1 &
 echo $! > runtime/watchdog.pid
 ```
 This replaces the local log. If you need old logs, rotate/archive them privately before restarting. Configure rotation for a long-running `nohup` deployment; the script itself does not rotate files. For systemd, use `sudo systemctl stop/start oracle-edge-watchdog.service` and journal retention settings instead.
 Stopping the watchdog leaves DNS in its last state. It does not shut down HAProxy, NPM, or applications.
 ## Manual switch or repair
 Stop the continuous writer first. To use Oracle:
 ```bash
 python3 watchdog.py --config "$PWD/runtime/watchdog.json" \
  --once --mode oracle --apply
 ```
 To use Cloudflare directly to home:
 ```bash
 python3 watchdog.py --config "$PWD/runtime/watchdog.json" \
  --once --mode cloudflare --apply
 ```
 These bypass recovery timers but require the local and selected route to work. They still enforce record identity checks. `--check` is a separate all-route preflight and cannot be combined with `--apply`.
 Mixed records or stored-state disagreement at startup need inspection. Once you have established which route is correct, the explicit one-shot command can reconcile recognized records and replace corrupt/conflicting local state. Do not erase state or change IDs blindly to suppress an error.
 For the systemd installation, stop the service and use its installed paths:
 ```bash
 sudo systemctl stop oracle-edge-watchdog.service
 sudo python3 -I /usr/local/lib/oracle-edge/watchdog.py \
  --config /etc/oracle-edge/watchdog.json --once --mode cloudflare --apply
 sudo chown -R oracle-edge:oracle-edge /var/lib/oracle-edge
 sudo systemctl start oracle-edge-watchdog.service
 ```
 Substitute `oracle` to switch back. Run the final start only after the manual command succeeds and the result is understood.
 ## Outage drill
 1. Confirm all three probes pass and the continuous writer logs `apply: true`.
 2. Confirm the managed records point at Oracle with DNS only. Verify a real application from outside the LAN; split DNS may keep local browsers on the LAN route regardless of public failover.
 3. Watch the home log.
 4. **On Oracle**, stop the relay to simulate loss of that route:
   ```bash
   sudo systemctl stop haproxy
   ```
 5. Expect `primary: false`, `local: true`, and `canary: true`. After four consecutive primary failures, look for `dns_verified` with `desired: cloudflare`.
 6. Confirm both managed A records contain the home IPv4, are proxied, and have Auto TTL. Test an actual application externally. Old DNS answers and existing connections may delay a client moving.
 7. **On Oracle**, restore the relay:
   ```bash
   sudo systemctl start haproxy
   sudo systemctl status haproxy --no-pager
   ```
 8. The home watchdog should see `primary: true` again. It returns to Oracle after **300 continuous healthy seconds** and **600 seconds in backup**, whichever condition completes later. A watchdog restart restarts both timers. Watch for `dns_verified` with `desired: oracle`.
 No failover script runs on Oracle. If the whole VM reboots, HAProxy starts automatically only if its service is enabled and its startup succeeds. The home watchdog must remain running to perform either DNS transition.
 If the backup probe fails, the controller deliberately does not switch into that route. Fix the route/probe; do not delete the check to make a drill appear successful. The static health endpoint does not prove the application itself is working.
 ## Updating an existing installation
 The corrected watchdog accepts the prior `runtime/watchdog.json` schema. You do not need to rerun either configuration wizard, recreate DNS records, regenerate credentials, or rerun the Oracle firewall installer merely to update it.
 ### Plain Python at home
 1. Download/extract the new source separately. Review changes and run its local tests.
 2. Stop the existing watchdog and wait for its state lock to release.
 3. Privately back up the old `watchdog.py` and `runtime/` outside the publication checkout.
 4. Copy the new `watchdog.py` into the existing installation folder, preserving `runtime/` and its permissions. Keeping that directory in place also preserves absolute paths stored in configuration.
 5. With Oracle running, run:
   ```bash
   python3 watchdog.py --config "$PWD/runtime/watchdog.json" --check
   ```
 6. Require all three checks to pass, then restart the continuous process. If a real Oracle outage is ongoing, `--check` will correctly fail its primary probe; verify a backup-target manual operation rather than forcing an all-route success.
 7. Repeat the controlled outage drill. This is especially important when installing the public-DNS canary correction.
 If you relocate the install folder, update absolute `token_file` and `state_file` paths in the private config. Simply renaming the folder does not rewrite them.
 ### systemd at home
 From the new source folder on the home host:
 ```bash
 sudo systemctl stop oracle-edge-watchdog.service
 sudo install -o root -g root -m 644 watchdog.py /usr/local/lib/oracle-edge/watchdog.py
 sudo python3 -I /usr/local/lib/oracle-edge/watchdog.py \
  --config /etc/oracle-edge/watchdog.json --check
 sudo chown -R oracle-edge:oracle-edge /var/lib/oracle-edge
 ```
 After successful checks:
 ```bash
 sudo systemctl reset-failed oracle-edge-watchdog.service
 sudo systemctl start oracle-edge-watchdog.service
 ```
 Back up the old script privately first. The existing `/etc/oracle-edge` credentials/config and `/var/lib/oracle-edge` state stay in place. A service-file change additionally needs review, installation, and `systemctl daemon-reload`; replacing Python alone does not require daemon reload.
 ## Home or Oracle IP changes
 For the fixed-IP profile, a home ISP address change is not automatically handled:
 1. Stop the watchdog to prevent competing changes.
 2. Update the private home `home_ip`, the permanent proxied canary A record, and any currently active fallback application A records consistently.
 3. On Oracle, render a new private configuration with `configure-oracle.py`. Back up the installed configuration. Validate with `sudo haproxy -c -f runtime/haproxy.cfg` before installing it as `/etc/haproxy/haproxy.cfg`, then reload HAProxy.
 4. Verify all routes and inspect any state/DNS disagreement. Use an explicit tested manual mode if needed, then restart the watchdog.
 For an Oracle public IP change, update the home allowed source, private watchdog `oracle_ip`, relevant OCI access configuration, and active Oracle-mode app records. Add the new allowed source before removing the old one. A reserved public IPv4 avoids most routine address changes.
 Do not rerun `setup-oracle.sh` to update only HAProxy's backend; it is an initial host installer with firewall staging.
 ## Emergency DNS fallback
 If the writer cannot run but the Cloudflare route is independently known healthy, stop/disable the writer and edit only its managed application A records in Cloudflare:
 - Content: current home public IPv4.
 - Proxy: enabled/orange.
 - TTL: Auto.
 Leave the permanent health record and unrelated records alone. The next normal watchdog start may report a stored-mode conflict because you changed DNS manually; resolve it with the explicit tested `--once --mode cloudflare --apply` operation.
 Changing DNS cannot repair home connectivity or instantly move already connected clients. Keep private backups and a known working route available during maintenance.
@@ -0,0 +1,53 @@
 # Publish your copy on GitHub
 Publish only this sanitized source folder. Do not upload an older deployment folder, attachments, terminal logs, `runtime/`, system configuration, or backups. No real deployment values are needed on GitHub or in GitHub Actions secrets.
 ## Use the prepared local repository
 The delivered local repository has a fresh `main` branch, a generic initial author, and no remote. It does not contain previous deployment history. From that repository folder:
 ```bash
 git status --short
 git log -1 --format=fuller
 git ls-files
 ```
 Review these outputs. The initial identity is a generic project identity, not a personal name/email. To use your own identity for future commits, configure a public display name and your GitHub-provided **noreply** email locally; do not paste a private email if you want it kept out of commits.
 Create an **empty** GitHub repository named `tls-edge-failover` (or your preferred name). Do not initialize it with another README, license, or `.gitignore`; these are already included. Choose its visibility in GitHub.
 Copy its HTTPS or SSH URL, then in Bash:
 ```bash
 read -r -p 'GitHub repository URL: ' REPO_URL
 git remote add origin "$REPO_URL"
 git push -u origin main
 ```
 Authenticate with your normal GitHub method. Do not put a token in the URL or a tracked file. These instructions assume no existing `origin`; inspect `git remote -v` before changing an established repository.
 ## If you downloaded the source ZIP
 The ZIP contains the same tracked source, including `.github/` and `.gitignore`, but deliberately excludes `.git` history and runtime files. Extract it into a new folder. You can upload the extracted source using GitHub's web interface, ensuring hidden files are included, or initialize Git locally:
 ```bash
 git init -b main
 git config user.name 'TLS Edge Failover contributors'
 git config user.email 'contributors@example.invalid'
 git add .
 git diff --cached --stat
 git diff --cached
 git commit -m 'Initial TLS edge failover source'
 ```
 Use the repository URL/push steps above. You can substitute your preferred public identity and GitHub noreply email before the initial commit.
 ## Before every publication
 - Review `git diff --cached` and `git ls-files` for domains, IPs, IDs, tokens, keys, logs, and local paths introduced by later changes.
 - Keep the tracked example JSON as placeholders; configure real installations in ignored paths.
 - Confirm runtime files are ignored with `git check-ignore runtime/watchdog.json runtime/cf-token runtime/watchdog.log runtime/haproxy.cfg`.
 - If secrets were previously committed, remove them from the history you intend to publish and revoke any real credential. Ignoring the filename does not repair existing history.
 - Keep GitHub Actions free of production DNS credentials. Its tests use fixtures and do not deploy anything.
 The included MIT license covers this source. Review it before publishing your copy. GitHub's **Actions** tab will show CI results after the first push; local test success does not mean a workflow has already run on GitHub.
@@ -0,0 +1,106 @@
 # Troubleshooting
 ## Oracle fails but nothing switches
 Look at the entire `health` event. Automatic fallback requires four consecutive failing **primary** checks and successful **local** and **canary** checks in the decision cycle. Also check `startup` contains `apply: true` and inspect `api_hold` or `fatal` events.
 An increasing failure counter alone does not authorize a switch to an unproven backup. No companion failover script is needed on Oracle; it only needs a working HAProxy service when that route is meant to be available.
 ## Canary must traverse Cloudflare with cache bypassed
 An HTTP 200 from NPM alone is insufficient. The backup check must actually cross Cloudflare, have `CF-Ray`, and have `CF-Cache-Status: DYNAMIC` or `BYPASS`.
 For an investigation on the home Linux server, run these in Bash:
 ```bash
 read -r -p 'Health hostname: ' HEALTH_HOST
 getent ahosts "$HEALTH_HOST"
 dig @1.1.1.1 "$HEALTH_HOST" A +short
 curl --noproxy '*' -sS -D - --max-time 15 \
  -w '\nConnected to: %{remote_ip}\n' \
  "https://$HEALTH_HOST/__edge_health?check=$(date +%s)"
 ```
 If the local answer and curl destination are the NPM LAN IP while public DNS returns Cloudflare addresses, split DNS is bypassing Cloudflare. The DNS record being orange in the dashboard does not override your local resolver.
 The corrected watchdog solves this internally: only its canary uses authenticated public DNS-over-HTTPS. It keeps normal local DNS untouched. Update `watchdog.py` using [Operations](OPERATIONS.md#updating-an-existing-installation), stop the old process, run `--check`, then restart it. Plain curl without `--resolve` will still follow your system resolver; that does not mean the corrected watchdog uses it.
 If an actual Cloudflare response is returned, inspect the Cache Rule, cache status, WAF/challenge/Access settings, Worker routes, redirects, and origin rules on this dedicated hostname. Do not pin a Cloudflare website IP permanently in `/etc/hosts`; those addresses can change.
 ## Public canary DNS lookup failed / deadline exceeded
 The canary requires outbound TCP 443 to `1.1.1.1`, verified TLS for `cloudflare-dns.com`, and a successful public A answer. Check firewall filtering, intercepting proxies, system CA certificates, and system time. The process bypasses proxy environment variables.
 The five-second default is a total budget for the lookup plus the health connection. A persistently slow but otherwise valid route may need a reviewed increase of `health.timeout_seconds` in private config (supported range: 1–15 seconds). Do not disable verification or accept a local DNS fallback.
 ## Local check fails but primary works
 The watchdog belongs **at home**, where it can reach NPM's LAN address. On Oracle, that private IP normally has no route to your home LAN. Confirm `health.local_ip` is NPM's existing address, home routing permits TCP 443, and the NPM certificate covers `health.primary_host`.
 From home, force the NPM destination while preserving the hostname:
 ```bash
 read -r -p 'Health hostname: ' HEALTH_HOST
 read -r -p 'NPM LAN IPv4: ' NPM_IP
 curl --noproxy '*' --fail --show-error --max-time 15 \
  --resolve "$HEALTH_HOST:443:$NPM_IP" \
  "https://$HEALTH_HOST/__edge_health"
 ```
 This is a local diagnostic, not proof that the Cloudflare path works.
 ## Primary check fails
 On Oracle, inspect without changing policy:
 ```bash
 sudo systemctl status haproxy --no-pager
 sudo journalctl -u haproxy -n 50 --no-pager
 sudo ss -lntp
 sudo haproxy -c -f /etc/haproxy/haproxy.cfg
 sudo iptables -S
 sudo ip6tables -S
 sudo nft list tables
 ```
 Check Oracle's applicable OCI network rules, host INPUT rule order, HAProxy's home address, and the home NAT/source allowlist. An ACCEPT below an unconditional REJECT/DROP will not permit new traffic. ICMP ping success does not prove TCP 443 works.
 To locate missing traffic, capture the relevant interface and protocol. Oracle HTTPS enters the home **WAN** first, then forwarded traffic reaches LAN/NPM. A LAN-only ICMP capture cannot establish that a WAN TCP 443 request never arrived. Filter for the actual Oracle source and TCP port 443 on WAN, then inspect the translated destination on LAN. This repository does not configure WireGuard.
 ## Oracle firewall or installer stops
 The automatic installer targets a dedicated Ubuntu 24.04 VM. Preflight runs before apt installation, service masking, and replacing configuration. It stops for active firewall managers, installed UFW even if inactive, Docker, recognized leftover UFW/Docker/edge chains, or unexpected native nftables tables.
 This avoids package conflicts removing an installed-but-inactive firewall package and discovering leftover chains only after services were changed. Package installation also uses `--no-remove`.
 On an existing or partly configured VM:
 1. Preserve the working SSH session and establish console recovery access.
 2. Inspect the rules/services above and installed firewall packages.
 3. Back up both live rulesets privately before any changes:
   ```bash
   sudo sh -c 'umask 077
   iptables-save > /root/edge-before.v4
   ip6tables-save > /root/edge-before.v6'
   ```
 4. Determine which existing manager owns policy. Use that manager or a deliberately reviewed migration. Do not flush all tables, delete OCI InstanceServices OUTPUT rules, or remove arbitrary chains/ports.
 5. If earlier installation output failed before the firewall helper armed a timer, no rollback timer exists for that attempt. Check actual state rather than assuming a timer is running.
 A narrow TCP 443 INPUT fix may restore connectivity on a legacy iptables-managed host, but it does not prove the host's SSH/IPv6 policy is fully hardened. Review the whole policy separately. This guide intentionally does not provide a universal destructive firewall reset.
 The timed helper restores pre-change rules after 180 seconds unless you complete the printed verification steps. If the deadline elapsed, inspect the actual head jumps and rollback status before saving. Do not rerun the full package/configuration installer just to repair one rule.
 ## DNS identity, mixed state, or API errors
 - **Replaced record ID / conflicting address record:** inspect the actual names and IDs, including AAAA, CNAME, HTTPS, and SVCB conflicts. Do not delete unrelated records indiscriminately.
 - **Mixed startup mode / state disagrees:** stop the writer, inspect DNS and your chosen healthy destination, then use the explicit manual-mode command in [Operations](OPERATIONS.md#manual-switch-or-repair).
 - **Cloudflare HTTP 401/403:** verify token validity, zone ID, permission, and zone scope privately.
 - **429:** let the controller respect Retry-After/backoff; restarting repeatedly defeats this protection.
 - **Another watchdog owns this state file:** stop the existing process and wait for exit. Deleting the lock file while it is held can let two writers run.
 - **Canary drift in fixed-IP mode:** restore its intended proxied/home/Auto state or coordinate an actual home-IP change. The fixed-IP writer never silently repairs the permanent canary.
 ## It has not switched back yet
 Keep Oracle healthy and the home watchdog running. A single primary success is not enough: five continuous healthy minutes and ten minutes of backup dwell must both pass. Restarting the watchdog restarts these windows. Check `local` remains healthy, API operations succeed, and no fatal event stopped the process. API readback success still leaves resolver-cache delay.
@@ -0,0 +1,16 @@
 # Validation of the initial source
 Local validation completed with Python 3.14.7 on macOS:
 - **49 automated tests passed**, using mocked network/API calls and isolated fake firewall commands.
 - All four Bash scripts passed individual `bash -n` checks.
 - The generalized renderer was tested for bounded/escaped SNI matching, rejected injection/invalid addresses, private output permissions, and replacement of existing output.
 - Both the default apex/wildcard wizard and a custom selected-app/health-host configuration were tested without remote DNS writes.
 - The public-DNS canary, certificate/body/cache requirements, failure/recovery timing, record boundaries, state reconciliation, and total deadlines retained their regression coverage.
 - Read-only preflight was tested against clean rules, active UFW, installed inactive UFW, leftover chains, and a native nftables table.
 The GitHub Actions workflow is configured to run tests on Python 3.9, 3.12, and 3.14 under Ubuntu 24.04, and to parse a rendered configuration with that distribution's HAProxy. Those GitHub jobs have not run merely because the source was prepared locally.
 No live Oracle, Cloudflare, OPNsense, or NPM configuration was changed during repository preparation. HAProxy/systemd deployment and the timed live firewall rollback were not exercised on this macOS machine. Complete the installation checks and an external application outage drill before relying on a deployment.
 The publication copy uses generic identifiers and excludes runtime/configuration data, secrets, logs, captures, and previous deployment history. Review future additions and commits before publishing; a static initial scan cannot guarantee later commits stay private.
@@ -0,0 +1,27 @@
 {
  "zone_id": "REPLACE_WITH_ZONE_ID",
  "oracle_ip": "REPLACE_WITH_ORACLE_PUBLIC_IPV4",
  "home_ip": "REPLACE_WITH_HOME_PUBLIC_IPV4",
  "canary": {
    "id": "REPLACE_WITH_CANARY_A_RECORD_ID",
    "name": "edge-health.example.com"
  },
  "apps": [
    {"id": "REPLACE_WITH_WILDCARD_A_RECORD_ID", "name": "*.example.com"},
    {"id": "REPLACE_WITH_APEX_A_RECORD_ID", "name": "example.com"}
  ],
  "health": {
    "primary_host": "edge-health.example.com",
    "local_ip": "REPLACE_WITH_NPM_LAN_IPV4",
    "path": "/__edge_health",
    "expected_body": "edge-ok-v1\n",
    "timeout_seconds": 5
  },
  "token_file": "/ABSOLUTE/INSTALL/PATH/runtime/cf-token",
  "state_file": "/ABSOLUTE/INSTALL/PATH/runtime/state.json",
  "interval_seconds": 15,
  "failures_required": 4,
  "recovery_seconds": 300,
  "minimum_dwell_seconds": 600,
  "api_timeout_seconds": 10
 }
@@ -0,0 +1,5 @@
 # /etc/systemd/system/haproxy.service.d/limits.conf
 [Service]
 LimitNOFILE=16384
 Restart=on-failure
 RestartSec=3
@@ -0,0 +1,50 @@
 # Template for a dedicated Ubuntu 24.04 relay. Render with configure-oracle.py.
 # No TLS termination, certificates, PROXY protocol, or personal proxy on Oracle.
 global
    log /dev/log local0
    user haproxy
    group haproxy
    daemon
    maxconn 2048
    maxconnrate 400
    stats socket /run/haproxy/admin.sock mode 600 level admin
    stats timeout 10s
    # Reloads drain old connections for at most 10 minutes, then cut them.
    hard-stop-after 10m
 defaults
    log global
    mode tcp
    option tcplog
    option dontlognull
    timeout connect 5s
    timeout client 5m
    timeout server 5m
    timeout client-fin 15s
    timeout server-fin 15s
    timeout check 5s
    timeout queue 5s
 frontend home_https
    bind 0.0.0.0:443
    maxconn 2048
    backlog 2048
    stick-table type ip size 100k expire 30s store conn_cur,conn_rate(1s)
    tcp-request connection track-sc0 src
    tcp-request connection reject if { sc0_conn_cur gt 100 }
    tcp-request connection reject if { sc0_conn_rate gt 30 }
    # Wait at most 5s for an ordinary TLS ClientHello for an allowed SNI.
    # This is routing hygiene, not user authentication or HTTP filtering.
    tcp-request inspect-delay 5s
    acl hello req.ssl_hello_type 1
    acl allowed_sni req.ssl_sni -m reg -i ^([a-z0-9-]+\.)*@@DOMAIN_REGEX@@$
    tcp-request content reject if hello !allowed_sni
    tcp-request content accept if hello allowed_sni
    tcp-request content reject if WAIT_END
    default_backend home_npm
 backend home_npm
    # TCP checks only. The home watchdog verifies HTTPS and the response body.
    # Fixed current home IPv4. Update this if your ISP changes the address.
    server home @@HOME_IPV4@@:443 check inter 5s fall 3 rise 2
@@ -0,0 +1,162 @@
 #!/usr/bin/env bash
 # Dedicated OCI Ubuntu 24.04 only. Run on Oracle, not on your home firewall.
 # Requires existing iptables, nftables, netfilter-persistent, Python and systemd.
 set -Eeuo pipefail
 umask 077
 die() { printf 'ERROR: %s\n' "$*" >&2; exit 1; }
 [[ $EUID -eq 0 ]] || die 'Run as root, preserving SSH_CONNECTION when using sudo.'
 [[ $# -ge 1 && $# -le 2 ]] || die 'Usage: install-firewall.sh ADMIN_IPV4_CIDR [--console]'
 [[ ${2:-} == '' || ${2:-} == --console ]] || die 'Only --console is supported as the second argument.'
 for executable in python3 iptables ip6tables iptables-save ip6tables-save iptables-restore ip6tables-restore nft systemctl systemd-run netfilter-persistent; do
    command -v "$executable" >/dev/null || die "Required command missing: $executable"
 done
 [[ -r /etc/os-release ]] || die 'Cannot identify the operating system.'
 . /etc/os-release
 [[ ${ID:-} == ubuntu && ${VERSION_ID:-} == 24.04 ]] || die 'This helper requires Ubuntu 24.04.'
 [[ $(iptables --version) == *nf_tables* && $(ip6tables --version) == *nf_tables* ]] || die 'Expected the stock iptables-nft backend for both families.'
 [[ -n ${SSH_CONNECTION:-} || ${2:-} == --console ]] || die 'SSH_CONNECTION is absent. Preserve it through sudo, or use --console only from the OCI console.'
 admin_cidr=$(python3 - "$1" "${SSH_CONNECTION:-}" <<'PY'
 import ipaddress, sys
 network = ipaddress.ip_network(sys.argv[1], strict=False)
 if network.version != 4:
    raise SystemExit('The administrator CIDR must be IPv4.')
 if network.prefixlen == 0:
    raise SystemExit('Refusing unrestricted public SSH; use your administrator /32 or trusted network.')
 if sys.argv[2]:
    fields = sys.argv[2].split()
    if len(fields) != 4:
        raise SystemExit('Malformed SSH_CONNECTION.')
    if fields[3] != '22':
        raise SystemExit('This helper permits SSH port 22 only; current SSH uses another port.')
    source = ipaddress.ip_address(fields[0])
    if source.version != 4 or source not in network:
        raise SystemExit('Current SSH source is outside the administrator IPv4 CIDR.')
 print(network)
 PY
 )
 for manager in ufw firewalld nftables docker; do
    if systemctl is-active --quiet "$manager.service"; then
        die "Competing service is active: $manager. Use a fresh dedicated VM."
    fi
 done
 command -v docker >/dev/null && die 'Docker is installed; this helper is for a dedicated VM without Docker.'
 rules_v4=$(iptables-save 2>&1)
 rules_v6=$(ip6tables-save 2>&1)
 rule_inventory="$rules_v4$rules_v6"
 [[ $rule_inventory != *Warning* && $rule_inventory != *incompatible* ]] || die 'iptables reports incompatible/native or legacy rules; inspect manually.'
 [[ $rule_inventory != *EDGE_INPUT* && $rule_inventory != *DOCKER* && $rule_inventory != *ufw-* ]] || die 'Existing edge, Docker or UFW chains found; refusing to overwrite them.'
 nft -j list tables | python3 -c '
 import json, sys
 allowed = {"filter", "nat", "mangle", "raw", "security"}
 for item in json.load(sys.stdin)["nftables"]:
    table = item.get("table")
    if table and (table["family"] not in {"ip", "ip6"} or table["name"] not in allowed):
        raise SystemExit("Unexpected native nftables table; inspect manually: " + str(table))
 '
 backup_dir=$(mktemp -d /root/oracle-edge-backup.XXXXXXXX)
 iptables-save >"$backup_dir/live.v4"
 ip6tables-save >"$backup_dir/live.v6"
 for family in v4 v6; do
    if [[ -e /etc/iptables/rules.$family ]]; then
        cp /etc/iptables/rules."$family" "$backup_dir/persistent.$family"
    else
        touch "$backup_dir/persistent.$family.absent"
    fi
 done
 chmod 600 "$backup_dir"/*
 unit_name="oracle-edge-rollback-$(date +%s)-$$"
 printf 'BACKUP_DIR=%q\n' "$backup_dir" >"$backup_dir/rollback.env"
 cat >"$backup_dir/rollback.sh" <<'ROLLBACK'
 #!/usr/bin/env bash
 set -uo pipefail
 umask 077
 . "$(dirname "$0")/rollback.env"
 result=0
 # Restore both complete pre-change rulesets, retaining OCI OUTPUT protections.
 iptables-restore --wait 10 <"$BACKUP_DIR/live.v4" || result=1
 ip6tables-restore --wait 10 <"$BACKUP_DIR/live.v6" || result=1
 for family in v4 v6; do
    if [[ -f $BACKUP_DIR/persistent.$family.absent ]]; then
        rm -f /etc/iptables/rules."$family" || result=1
    else
        mkdir -p /etc/iptables || result=1
        install -m 600 "$BACKUP_DIR/persistent.$family" /etc/iptables/rules."$family" || result=1
    fi
 done
 exit "$result"
 ROLLBACK
 chmod 700 "$backup_dir/rollback.sh"
 rollback_on_error() {
    local result=$1
    trap - ERR INT TERM
    printf 'Installation failed; restoring backups from %s\n' "$backup_dir" >&2
    if bash "$backup_dir/rollback.sh"; then
        systemctl stop "$unit_name.timer" 2>/dev/null || true
    else
        printf 'Rollback reported an error; use the OCI console. Timer remains armed.\n' >&2
    fi
    exit "$result"
 }
 trap 'rollback_on_error $?' ERR
 trap 'rollback_on_error 130' INT
 trap 'rollback_on_error 143' TERM
 # Arm recovery before making any rule changes. Nothing is persisted yet.
 systemd-run --unit="$unit_name" --on-active=180s --timer-property=AccuracySec=1s \
    --property=Type=oneshot /bin/bash "$backup_dir/rollback.sh"
 systemctl is-active --quiet "$unit_name.timer"
 # Build unattached chains. Existing INPUT paths remain effective during staging.
 iptables --wait 10 -N EDGE_INPUT
 ip6tables --wait 10 -N EDGE_INPUT6
 for family in v4 v6; do
    if [[ $family == v4 ]]; then
        firewall=iptables
        chain=EDGE_INPUT
    else
        firewall=ip6tables
        chain=EDGE_INPUT6
    fi
    "$firewall" --wait 10 -A "$chain" -i lo -j ACCEPT
    "$firewall" --wait 10 -A "$chain" -m conntrack --ctstate INVALID -j DROP
    "$firewall" --wait 10 -A "$chain" -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
    if [[ $family == v4 ]]; then
        "$firewall" --wait 10 -A "$chain" -p icmp -j ACCEPT
        "$firewall" --wait 10 -A "$chain" -p udp --sport 67 --dport 68 -j ACCEPT
        "$firewall" --wait 10 -A "$chain" -s "$admin_cidr" -p tcp --dport 22 -m conntrack --ctstate NEW -j ACCEPT
        "$firewall" --wait 10 -A "$chain" -p tcp --dport 443 -m conntrack --ctstate NEW -j ACCEPT
    else
        "$firewall" --wait 10 -A "$chain" -p ipv6-icmp -j ACCEPT
        "$firewall" --wait 10 -A "$chain" -p udp --sport 547 --dport 546 -j ACCEPT
    fi
    "$firewall" --wait 10 -A "$chain" -j DROP
 done
 # Within each family the completed chain becomes live with one head insertion.
 iptables --wait 10 -I INPUT 1 -j EDGE_INPUT
 ip6tables --wait 10 -I INPUT 1 -j EDGE_INPUT6
 trap - ERR INT TERM
 cat <<INSTRUCTIONS
 Firewall active temporarily. Automatic rollback is armed for 180 seconds.
 Backup and manual rollback: sudo bash '$backup_dir/rollback.sh'
 Before that deadline, keep this session open and test from a NEW terminal:
  ssh YOUR_ORACLE_USER@ORACLE_PUBLIC_IP
  curl --fail --resolve YOUR_HOST:443:ORACLE_PUBLIC_IP https://YOUR_HOST/
 The relay must already be running for the curl check to pass.
 Also confirm unauthorized sources cannot open SSH; IPv6 has no new TCP ingress.
 Only after the tests pass, run ON ORACLE:
  sudo systemctl stop '$unit_name.timer'
  sudo systemctl is-active '$unit_name.service'
 The service must be inactive, not activating/active. If rollback already ran,
 or the timer expired, DO NOT save: inspect the live rules and rerun the helper.
  sudo iptables -C INPUT -j EDGE_INPUT
  sudo ip6tables -C INPUT -j EDGE_INPUT6
  sudo netfilter-persistent save
 No rules have been persisted and the rollback timer has not been cancelled.
 INSTRUCTIONS
@@ -0,0 +1,10 @@
 # Main Advanced tab of the NEW edge-health.example.com Proxy Host only.
 # It exposes only this small static marker; no application content is served here.
 if ($uri != /__edge_health) {
    return 404;
 }
 location = /__edge_health {
    default_type text/plain;
    add_header Cache-Control "no-store, max-age=0" always;
    return 200 "edge-ok-v1\n";
 }
@@ -0,0 +1,42 @@
 [Unit]
 Description=Conservative Oracle and Cloudflare DNS failover
 Wants=network-online.target
 After=network-online.target
 StartLimitIntervalSec=300
 StartLimitBurst=3
 [Service]
 Type=simple
 User=oracle-edge
 Group=oracle-edge
 StateDirectory=oracle-edge
 StateDirectoryMode=0700
 UMask=0077
 LoadCredential=cf_token:/etc/oracle-edge/cf-token
 ExecStart=/usr/bin/python3 -I /usr/local/lib/oracle-edge/watchdog.py --config /etc/oracle-edge/watchdog.json --run --apply
 Restart=on-failure
 RestartPreventExitStatus=2
 RestartSec=15
 TimeoutStopSec=35
 KillMode=control-group
 NoNewPrivileges=yes
 PrivateTmp=yes
 PrivateDevices=yes
 ProtectSystem=strict
 ProtectHome=yes
 ProtectKernelTunables=yes
 ProtectKernelModules=yes
 ProtectKernelLogs=yes
 ProtectControlGroups=yes
 ProtectClock=yes
 RestrictSUIDSGID=yes
 LockPersonality=yes
 RestrictRealtime=yes
 CapabilityBoundingSet=
 RestrictAddressFamilies=AF_UNIX AF_INET AF_INET6
 TasksMax=32
 MemoryMax=256M
 LimitNOFILE=256
 [Install]
 WantedBy=multi-user.target
@@ -0,0 +1,29 @@
 #!/usr/bin/env bash
 # Read-only checks. Run before package installation, masking units or replacing config.
 set -Eeuo pipefail
 die() { printf 'Preflight stopped: %s\n' "$*" >&2; exit 1; }
 for command_name in iptables-save ip6tables-save nft python3 systemctl dpkg-query; do
    command -v "$command_name" >/dev/null || die "Missing inspection tool: $command_name; inspect this VM before installing firewall packages."
 done
 for manager in ufw firewalld nftables docker; do
    if systemctl is-active --quiet "$manager.service"; then
        die "$manager is active. No changes made; retain and inspect its existing policy."
    fi
 done
 command -v docker >/dev/null && die 'Docker is installed; automatic firewall replacement is not supported.'
 ufw_status=$(dpkg-query -W -f='${Status}' ufw 2>/dev/null || true)
 [[ $ufw_status != 'install ok installed' ]] || die 'UFW is installed, even if inactive. No packages will be removed; inspect the current firewall first.'
 rules_v4=$(iptables-save 2>&1) || die 'Cannot read IPv4 rules.'
 rules_v6=$(ip6tables-save 2>&1) || die 'Cannot read IPv6 rules.'
 inventory="$rules_v4$rules_v6"
 [[ $inventory != *Warning* && $inventory != *incompatible* ]] || die 'Incompatible/native firewall rules need inspection.'
 [[ $inventory != *EDGE_INPUT* && $inventory != *DOCKER* && $inventory != *ufw-* ]] || die 'Existing edge, Docker or UFW chains found. No changes made; inspect them first.'
 nft -j list tables | python3 -c '
 import json, sys
 allowed = {"filter", "nat", "mangle", "raw", "security"}
 for item in json.load(sys.stdin)["nftables"]:
    table = item.get("table")
    if table and (table["family"] not in {"ip", "ip6"} or table["name"] not in allowed):
        raise SystemExit("Preflight stopped: native nftables table needs inspection: " + str(table))
 '
 printf 'Read-only firewall preflight passed.\n'
@@ -0,0 +1,43 @@
 #!/usr/bin/env bash
 # Run on the home Ubuntu Linux host, not Oracle or OPNsense.
 set -Eeuo pipefail
 umask 077
 [[ $EUID -eq 0 ]] || { echo 'Run: sudo bash setup-home.sh' >&2; exit 1; }
 . /etc/os-release
 [[ ${ID:-} == ubuntu && ${VERSION_ID:-} == 24.04 ]] || {
    echo 'This home installer targets Ubuntu 24.04. Do not run it on OPNsense.' >&2
    exit 1
 }
 cd -- "$(dirname -- "${BASH_SOURCE[0]}")"
 apt-get update
 apt-get --no-remove install -y python3 ca-certificates
 if ! id oracle-edge >/dev/null 2>&1; then
    useradd --system --user-group --home-dir /nonexistent --shell /usr/sbin/nologin oracle-edge
 fi
 if systemctl cat oracle-edge-failover.service >/dev/null 2>&1; then
    systemctl disable --now oracle-edge-failover.service
 fi
 if systemctl cat oracle-edge-watchdog.service >/dev/null 2>&1; then
    systemctl stop oracle-edge-watchdog.service
 fi
 install -d -o root -g oracle-edge -m 750 /etc/oracle-edge
 install -d -o root -g root -m 755 /usr/local/lib/oracle-edge
 install -d -o oracle-edge -g oracle-edge -m 700 /var/lib/oracle-edge
 if [[ -f /etc/oracle-edge/watchdog.json || -f /etc/oracle-edge/cf-token ]]; then
    backup_dir=$(mktemp -d /etc/oracle-edge/backup.XXXXXXXX)
    for name in watchdog.json cf-token; do
        if [[ -f /etc/oracle-edge/$name ]]; then
            cp -p "/etc/oracle-edge/$name" "$backup_dir/$name"
        fi
    done
    printf 'Previous local credentials/config backed up to %s\n' "$backup_dir"
 fi
 install -o root -g root -m 644 watchdog.py configure-home.py /usr/local/lib/oracle-edge/
 python3 -I /usr/local/lib/oracle-edge/configure-home.py
 install -o root -g root -m 644 oracle-edge-watchdog.service /etc/systemd/system/oracle-edge-watchdog.service
 systemd-analyze verify /etc/systemd/system/oracle-edge-watchdog.service
 systemctl daemon-reload
 trap 'chown -R oracle-edge:oracle-edge /var/lib/oracle-edge' EXIT
 python3 -I /usr/local/lib/oracle-edge/watchdog.py --config /etc/oracle-edge/watchdog.json --check
 echo 'Preflight passed. The watchdog is installed but NOT running; DNS is unchanged.'
 echo 'Continue with docs/INSTALL.md (systemd option) to switch and enable automation.'
@@ -0,0 +1,45 @@
 #!/usr/bin/env bash
 # Dedicated Ubuntu 24.04 OCI VM. DNS is never changed by this installer.
 set -Eeuo pipefail
 umask 077
 [[ $EUID -eq 0 ]] || { echo 'Run with sudo, preserving SSH_CONNECTION.' >&2; exit 1; }
 . /etc/os-release
 [[ ${ID:-} == ubuntu && ${VERSION_ID:-} == 24.04 ]] || { echo 'Requires Ubuntu 24.04.' >&2; exit 1; }
 [[ -n ${SSH_CONNECTION:-} ]] || { echo 'Use: sudo env "SSH_CONNECTION=$SSH_CONNECTION" bash setup-oracle.sh' >&2; exit 1; }
 admin_ip=$(python3 - "$SSH_CONNECTION" <<'PY'
 import ipaddress, sys
 fields = sys.argv[1].split()
 if len(fields) != 4 or fields[3] != '22':
    raise SystemExit('Requires an existing IPv4 SSH session on port 22.')
 address = ipaddress.ip_address(fields[0])
 if address.version != 4:
    raise SystemExit('Use an IPv4 SSH connection for this IPv4-only profile.')
 print(address)
 PY
 )
 for manager in ufw firewalld nftables docker; do
    if systemctl is-active --quiet "$manager.service"; then
        echo "Existing $manager manager is active. Use its existing policy or a clean dedicated VM." >&2
        exit 1
    fi
 done
 cd -- "$(dirname -- "${BASH_SOURCE[0]}")"
 [[ -f runtime/haproxy.cfg ]] || { echo 'First run: python3 configure-oracle.py' >&2; exit 1; }
 bash ./preflight-oracle.sh
 systemctl mask nftables.service
 apt-get update
 apt-get --no-remove install -y haproxy ca-certificates curl iptables iptables-persistent nftables unattended-upgrades
 backup_dir=$(mktemp -d /root/tls-edge-failover-config-backup.XXXXXXXX)
 cp -a /etc/haproxy "$backup_dir/"
 printf 'Existing HAProxy configuration saved to %s\n' "$backup_dir"
 haproxy -c -f runtime/haproxy.cfg
 install -o root -g root -m 644 runtime/haproxy.cfg /etc/haproxy/haproxy.cfg
 install -d -m 755 /etc/systemd/system/haproxy.service.d
 install -o root -g root -m 644 haproxy-systemd.conf /etc/systemd/system/haproxy.service.d/limits.conf
 haproxy -c -f /etc/haproxy/haproxy.cfg
 systemctl daemon-reload
 systemctl enable --now haproxy
 systemctl reload haproxy
 bash ./install-firewall.sh "$admin_ip/32"
 printf '\nOracle relay uses runtime/haproxy.cfg. Cloudflare DNS is unchanged.\n'
 printf 'Complete the NEW SSH and HTTPS tests before the firewall rollback timer expires.\n'
@@ -0,0 +1,171 @@
 """Setup tests use temporary files and isolated mock commands, never a real firewall."""
 import contextlib
 import importlib.util
 import io
 import json
 import os
 from pathlib import Path
 import re
 import subprocess
 import sys
 import tempfile
 import unittest
 from unittest import mock
 ROOT = Path(__file__).resolve().parents[1]
 sys.path.insert(0, str(ROOT))
 import watchdog as w
 def module_from(filename, name):
    spec = importlib.util.spec_from_file_location(name, ROOT / filename)
    module = importlib.util.module_from_spec(spec)
    spec.loader.exec_module(module)
    return module
 class OracleRendererTests(unittest.TestCase):
    @classmethod
    def setUpClass(cls):
        cls.renderer = module_from("configure-oracle.py", "configure_oracle")
    def test_sni_domain_is_escaped_and_bounded(self):
        # Public resolver IP is only a validation fixture; no socket is opened.
        result = self.renderer.render("Example.COM.", "8.8.8.8")
        expression = next(line.split()[-1] for line in result.splitlines() if "acl allowed_sni" in line)
        for good in ("example.com", "app.example.com", "a.b.example.com"):
            self.assertIsNotNone(re.fullmatch(expression, good))
        for bad in ("exampleXcom", "example.com.attacker.test", "other.test", "notexample.com"):
            self.assertIsNone(re.fullmatch(expression, bad))
        self.assertIn("server home 8.8.8.8:443", result)
        self.assertNotIn("@@", result)
        executable = "\n".join(line for line in result.splitlines() if not line.lstrip().startswith("#"))
        self.assertNotRegex(executable, r"\b(?:ssl|crt|send-proxy)\b")
    def test_injection_and_invalid_destinations_rejected(self):
        for domain, address in (("example.com\nbind :80", "8.8.8.8"),
                                ("*.example.com", "8.8.8.8"),
                                ("example.com", "127.0.0.1"),
                                ("example.com", "203.0.113.10"),
                                ("example.com", "8.8.8.8:443")):
            with self.subTest(domain=domain, address=address), self.assertRaises(w.SafetyError):
                self.renderer.render(domain, address)
    def test_rendered_file_is_private_and_existing_file_replaced_atomically(self):
        with tempfile.TemporaryDirectory() as directory:
            destination = Path(directory) / "private" / "haproxy.cfg"
            with contextlib.redirect_stdout(io.StringIO()):
                self.renderer.main(["--domain", "example.com", "--home-ip", "8.8.8.8", "--output", str(destination)])
                self.renderer.main(["--domain", "example.net", "--home-ip", "1.1.1.1", "--output", str(destination)])
            self.assertEqual(destination.stat().st_mode & 0o777, 0o600)
            self.assertEqual(destination.parent.stat().st_mode & 0o777, 0o700)
            self.assertIn(r"example\.net", destination.read_text())
            self.assertEqual([path.name for path in destination.parent.iterdir()], ["haproxy.cfg"])
 class HomeWizardTests(unittest.TestCase):
    def setUp(self):
        self.wizard = module_from("configure-home.py", "configure_home_general")
    def test_selected_apps_and_custom_canary_generate_only_requested_references(self):
        records = [
            {"id": "1" * 32, "type": "A", "name": "check.example.net", "content": "1.1.1.1", "proxied": True, "ttl": 1},
            {"id": "2" * 32, "type": "A", "name": "app.example.net", "content": "8.8.8.8", "proxied": False, "ttl": 60},
            {"id": "3" * 32, "type": "A", "name": "example.net", "content": "8.8.8.8", "proxied": False, "ttl": 60},
        ]
        api = mock.Mock()
        api.inventory.return_value = records
        with tempfile.TemporaryDirectory() as directory, \
             mock.patch.object(self.wizard, "__file__", str(Path(directory) / "configure-home.py")), \
             mock.patch.object(sys, "argv", ["configure-home.py", "--local", "--health-host", "check.example.net", "--app", "app.example.net"]), \
             mock.patch("builtins.input", side_effect=["example.net", "8.8.8.8", "1.1.1.1", "f" * 32, "10.50.0.10"]), \
             mock.patch.object(self.wizard.getpass, "getpass", return_value="fixture-token"), \
             mock.patch.object(self.wizard, "Cloudflare", return_value=api), \
             contextlib.redirect_stdout(io.StringIO()):
            self.wizard.main()
            cfg = json.loads((Path(directory) / "runtime/watchdog.json").read_text())
        self.assertEqual(cfg["canary"]["name"], "check.example.net")
        self.assertEqual(cfg["health"]["primary_host"], "check.example.net")
        self.assertEqual(cfg["apps"], [{"id": "2" * 32, "name": "app.example.net"}])
        self.assertEqual(api.mock_calls, [mock.call.inventory()])
    def test_outside_domain_is_rejected_before_token_or_api_access(self):
        for option in ("--app", "--health-host"):
            with self.subTest(option=option), tempfile.TemporaryDirectory() as directory, \
                 mock.patch.object(self.wizard, "__file__", str(Path(directory) / "configure-home.py")), \
                 mock.patch.object(sys, "argv", ["configure-home.py", "--local", option, "attacker.test"]), \
                 mock.patch("builtins.input", side_effect=["example.net", "8.8.8.8", "1.1.1.1"]), \
                 mock.patch.object(self.wizard.getpass, "getpass") as token, \
                 mock.patch.object(self.wizard, "Cloudflare") as api:
                with self.assertRaises(w.SafetyError):
                    self.wizard.main()
                token.assert_not_called()
                api.assert_not_called()
                self.assertFalse((Path(directory) / "runtime/cf-token").exists())
 class FirewallPreflightTests(unittest.TestCase):
    def run_profile(self, profile):
        with tempfile.TemporaryDirectory() as directory:
            binary = Path(directory)
            (binary / "python3").symlink_to(sys.executable)
            dispatcher = """#!/usr/bin/env python3
 import json, os, sys
 from pathlib import Path
 name = Path(sys.argv[0]).name
 profile = os.environ['PREFLIGHT_PROFILE']
 with open(os.environ['PREFLIGHT_CALLS'], 'a') as stream:
    stream.write(name + ' ' + ' '.join(sys.argv[1:]) + '\\n')
 if name == 'systemctl':
    assert sys.argv[1:3] == ['is-active', '--quiet']
    sys.exit(0 if profile == 'active' and sys.argv[-1] == 'ufw.service' else 3)
 if name == 'dpkg-query':
    print('install ok installed' if profile == 'installed' else '')
 elif name in ('iptables-save', 'ip6tables-save'):
    print('*filter\\n:INPUT ACCEPT [0:0]\\n' + (':ufw-before-input - [0:0]\\n' if profile == 'leftover' else '') + 'COMMIT')
 elif name == 'nft':
    assert sys.argv[1:] == ['-j', 'list', 'tables']
    print(json.dumps({'nftables': [{'table': {'family': 'inet' if profile == 'native' else 'ip', 'name': 'custom' if profile == 'native' else 'filter'}}]}))
 else:
    raise RuntimeError('Unexpected command')
 """
            for name in ("systemctl", "dpkg-query", "iptables-save", "ip6tables-save", "nft"):
                path = binary / name
                path.write_text(dispatcher)
                path.chmod(0o700)
            env = {**os.environ, "PATH": directory, "PREFLIGHT_PROFILE": profile,
                   "PREFLIGHT_CALLS": str(binary / "calls")}
            result = subprocess.run(["/bin/bash", str(ROOT / "preflight-oracle.sh")], env=env,
                                    text=True, capture_output=True, timeout=15)
            calls = (binary / "calls").read_text()
        self.assertNotIn(" stop ", calls)
        self.assertNotIn(" mask ", calls)
        return result
    def test_clean_compatibility_rules_pass(self):
        result = self.run_profile("clean")
        self.assertEqual(result.returncode, 0, result.stderr)
    def test_active_manager_stops(self):
        result = self.run_profile("active")
        self.assertNotEqual(result.returncode, 0)
        self.assertIn("ufw is active", result.stderr)
    def test_installed_but_inactive_ufw_stops(self):
        result = self.run_profile("installed")
        self.assertNotEqual(result.returncode, 0)
        self.assertIn("UFW is installed", result.stderr)
    def test_leftover_chains_stop(self):
        result = self.run_profile("leftover")
        self.assertNotEqual(result.returncode, 0)
        self.assertIn("chains found", result.stderr)
    def test_native_table_stops(self):
        result = self.run_profile("native")
        self.assertNotEqual(result.returncode, 0)
        self.assertIn("native nftables table", result.stderr)
 if __name__ == "__main__":
    unittest.main()
@@ -0,0 +1,493 @@
 #!/usr/bin/env python3
 """Controller tests; all Cloudflare and route operations mocked.
 Public resolver IPs are validation fixtures only. No fixture address is contacted.
 Names and record IDs are synthetic, unrelated to a real deployment.
 """
 import copy
 import importlib.util
 import io
 import json
 import multiprocessing
 import os
 from pathlib import Path
 import sys
 import tempfile
 import time
 import unittest
 from unittest import mock
 ROOT = Path(__file__).resolve().parents[1]
 sys.path.insert(0, str(ROOT))
 import watchdog as w
 def config():
    return {
        "zone_id": "f" * 32, "oracle_ip": "8.8.8.8",
        "origin": {"id": "1" * 32, "name": "origin.example.com"},
        "canary": {"id": "2" * 32, "name": "standby-health.example.com"},
        "apps": [{"id": "3" * 32, "name": "a.example.com"}, {"id": "4" * 32, "name": "b.example.com"}],
        "health": {"primary_host": "health.example.com", "local_ip": "192.168.1.20",
                   "path": "/edge-health", "expected_body": "edge-ok\n", "timeout_seconds": 5},
        "interval_seconds": 15, "failures_required": 4, "recovery_seconds": 300,
        "minimum_dwell_seconds": 600, "api_timeout_seconds": 10,
    }
 def records(cfg):
    return [
        {**cfg["origin"], "type": "A", "content": "1.1.1.1", "proxied": False, "ttl": 60},
        {**cfg["canary"], "type": "A", "content": "1.1.1.1", "proxied": True, "ttl": 1},
        *[{**ref, "type": "A", "content": cfg["oracle_ip"], "proxied": False, "ttl": 60} for ref in cfg["apps"]],
    ]
 def probes(primary=True, local=True, canary=True):
    return {name: {"ok": good, "error": "simulated failure"} for name, good in
            {"primary": primary, "local": local, "canary": canary}.items()}
 class FakeAPI:
    def __init__(self, data):
        self.data = copy.deepcopy(data)
        self.writes = []
        self.fail_partial_once = False
    def inventory(self):
        return copy.deepcopy(self.data)
    def batch_patch(self, patches):
        self.writes.append(copy.deepcopy(patches))
        for index, patch in enumerate(patches):
            next(r for r in self.data if r["id"] == patch["id"]).update(patch)
            if index == 0 and self.fail_partial_once:
                self.fail_partial_once = False
                raise w.APIError("Simulated ambiguous write", True)
 class MemoryStore:
    def __init__(self, state=None):
        self.state = state
        self.writes = []
    def read(self):
        return self.state
    def write(self, mode, last_switch):
        self.state = {"version": 1, "mode": mode, "last_switch": last_switch}
        self.writes.append(copy.deepcopy(self.state))
 def slow_worker(connection, job):
    time.sleep(30)
 class InventoryTests(unittest.TestCase):
    def setUp(self):
        self.cfg = config()
        self.data = records(self.cfg)
    def test_reject_aaaa_at_managed_name(self):
        self.data.append({"id": "5" * 32, "name": "a.example.com", "type": "AAAA", "content": "2606:4700::1111"})
        with self.assertRaises(w.SafetyError):
            w.validate_inventory(self.cfg, self.data)
    def test_reject_replaced_record_id(self):
        self.data[-1]["id"] = "5" * 32
        with self.assertRaises(w.SafetyError):
            w.validate_inventory(self.cfg, self.data)
    def test_reject_origin_proxy_or_oracle_address(self):
        for field, value in [("proxied", True), ("content", "8.8.8.8"), ("content", "192.168.1.1")]:
            with self.subTest(field=field, value=value):
                data = copy.deepcopy(self.data)
                data[0][field] = value
                with self.assertRaises(w.SafetyError):
                    w.validate_inventory(self.cfg, data)
    def test_old_global_proxied_home_address_is_allowed_for_ddns_reconciliation(self):
        for item in self.data[2:]:
            item.update(content="9.9.9.9", proxied=True, ttl=1)
        by_id, home = w.validate_inventory(self.cfg, self.data)
        self.assertEqual(w.observed_mode(self.cfg, by_id), "cloudflare")
        self.assertEqual(len(w.patches_for(self.cfg["apps"], by_id, home, True)), 2)
    def test_ttl_is_part_of_noop_comparison(self):
        by_id, _ = w.validate_inventory(self.cfg, self.data)
        self.assertEqual(w.patches_for(self.cfg["apps"], by_id, "8.8.8.8", False), [])
        by_id["3" * 32]["ttl"] = 300
        self.assertEqual(len(w.patches_for(self.cfg["apps"], by_id, "8.8.8.8", False)), 1)
    def test_origin_secret_sent_to_local_probe_only(self):
        jobs = w.make_probe_jobs(self.cfg, "a" * 64)
        self.assertEqual(jobs["local"]["origin_secret"], "a" * 64)
        self.assertNotIn("origin_secret", jobs["primary"])
        self.assertNotIn("origin_secret", jobs["canary"])
 class HealthTests(unittest.TestCase):
    def setUp(self):
        self.cfg = config()
    def test_failover_requires_four_consecutive_primary_failures_and_both_candidates(self):
        window = w.HealthWindow(self.cfg)
        for now in [0, 15, 30]:
            window.sample(probes(primary=False), now)
            self.assertEqual(window.target("oracle", probes(primary=False), now, 0), "oracle")
        window.sample(probes(primary=False), 45)
        self.assertEqual(window.target("oracle", probes(primary=False), 45, 0), "cloudflare")
        self.assertEqual(window.target("oracle", probes(primary=False, local=False), 45, 0), "oracle")
        self.assertEqual(window.target("oracle", probes(primary=False, canary=False), 45, 0), "oracle")
    def test_recovery_requires_true_elapsed_300_seconds_and_dwell_600(self):
        window = w.HealthWindow(self.cfg)
        for now in range(0, 301, 15):
            window.sample(probes(), now)
        self.assertEqual(window.target("cloudflare", probes(), 299, 600), "cloudflare")
        self.assertEqual(window.target("cloudflare", probes(), 300, 599), "cloudflare")
        self.assertEqual(window.target("cloudflare", probes(), 300, 600), "oracle")
    def test_sample_gap_resets_continuity(self):
        window = w.HealthWindow(self.cfg)
        window.sample(probes(), 0)
        window.sample(probes(), 31)
        self.assertEqual(window.healthy_since, 31)
    def test_success_resets_failure_streak(self):
        window = w.HealthWindow(self.cfg)
        window.sample(probes(primary=False), 0)
        window.sample(probes(), 15)
        window.sample(probes(primary=False), 30)
        self.assertEqual(window.failures, 1)
 class ControllerTests(unittest.TestCase):
    def setUp(self):
        self.cfg = config()
        self.api = FakeAPI(records(self.cfg))
        self.store = MemoryStore()
        self.logger = mock.patch.object(w, "log")
        self.logger.start()
        self.addCleanup(self.logger.stop)
    def controller(self, **kwargs):
        return w.Controller(self.cfg, self.api, self.store, **kwargs)
    def test_default_dry_run_has_no_remote_or_persistent_state_writes(self):
        self.api.data[1]["content"] = "9.9.9.9"
        ctl = self.controller()
        with mock.patch.object(w, "bounded_jobs", return_value=probes()):
            ctl.cycle()
        self.assertEqual(self.api.writes, [])
        self.assertEqual(self.store.writes, [])
    def test_canary_refreshed_in_oracle_mode_without_touching_origin(self):
        self.api.data[1]["content"] = "9.9.9.9"
        ctl = self.controller(apply=True)
        with mock.patch.object(w, "bounded_jobs", return_value=probes()):
            ctl.cycle()
        patched = [p for batch in self.api.writes for p in batch]
        self.assertEqual([p["id"] for p in patched], ["2" * 32])
        self.assertEqual(patched[0]["content"], "1.1.1.1")
    def test_mixed_startup_refuses_every_write(self):
        self.api.data[2].update(content="1.1.1.1", proxied=True, ttl=1)
        self.api.data[1]["content"] = "9.9.9.9"
        with self.assertRaises(w.SafetyError):
            self.controller(apply=True).cycle()
        self.assertEqual(self.api.writes, [])
        self.assertEqual(self.store.writes, [])
    def test_explicit_manual_mode_repairs_mixed_records_after_healthy_target(self):
        self.api.data[2].update(content="1.1.1.1", proxied=True, ttl=1)
        ctl = self.controller(apply=True, manual_mode="cloudflare")
        with mock.patch.object(w, "bounded_jobs", return_value=probes(primary=False)):
            ctl.cycle()
        self.assertEqual(ctl.mode, "cloudflare")
        self.assertTrue(all(r["proxied"] for r in self.api.data[2:]))
    def test_manual_mode_rejects_unhealthy_target(self):
        ctl = self.controller(apply=True, manual_mode="cloudflare")
        with mock.patch.object(w, "bounded_jobs", return_value=probes(canary=False)):
            with self.assertRaises(w.SafetyError):
                ctl.cycle()
        self.assertEqual(self.api.writes, [])
        self.assertEqual(self.store.writes, [])
    def test_ambiguous_partial_write_keeps_intent_and_repairs_after_primary_recovers(self):
        ctl = self.controller(apply=True)
        with mock.patch.object(w, "bounded_jobs", return_value=probes(primary=False)):
            for _ in range(3):
                ctl.cycle()
            self.api.fail_partial_once = True
            with self.assertRaises(w.APIError):
                ctl.cycle()
        self.assertEqual(ctl.mode, "cloudflare")
        self.assertEqual(self.store.state["mode"], "cloudflare")
        self.assertTrue(self.api.data[2]["proxied"])
        self.assertFalse(self.api.data[3]["proxied"])
        with mock.patch.object(w, "bounded_jobs", return_value=probes()):
            ctl.cycle()
        self.assertTrue(all(r["proxied"] for r in self.api.data[2:]))
    def test_readonly_check_makes_no_writes(self):
        ctl = self.controller(check=True)
        with mock.patch.object(w, "bounded_jobs", return_value=probes()):
            ctl.cycle()
        self.assertEqual(self.api.writes, [])
        self.assertEqual(self.store.writes, [])
    def test_runtime_drift_reconciled_even_without_mode_transition(self):
        ctl = self.controller(apply=True)
        with mock.patch.object(w, "bounded_jobs", return_value=probes()):
            ctl.cycle()
            self.api.data[2].update(content="1.1.1.1", proxied=True, ttl=1)
            ctl.cycle()
        self.assertFalse(self.api.data[2]["proxied"])
        self.assertEqual(self.api.data[2]["content"], "8.8.8.8")
    def test_runtime_drift_not_written_to_unhealthy_target(self):
        ctl = self.controller(apply=True)
        with mock.patch.object(w, "bounded_jobs", return_value=probes()):
            ctl.cycle()
        self.api.data[2].update(content="1.1.1.1", proxied=True, ttl=1)
        with mock.patch.object(w, "bounded_jobs", return_value=probes(primary=False)):
            ctl.cycle()
        self.assertTrue(self.api.data[2]["proxied"])
 class PersistenceAndNetworkTests(unittest.TestCase):
    def test_atomic_state_roundtrip_and_private_mode(self):
        with tempfile.TemporaryDirectory() as directory:
            path = Path(directory) / "state.json"
            store = w.StateStore(path)
            store.write("cloudflare", 123)
            self.assertEqual(store.read(), {"version": 1, "mode": "cloudflare", "last_switch": 123})
            self.assertEqual(path.stat().st_mode & 0o777, 0o600)
            self.assertEqual(len(list(Path(directory).iterdir())), 1)
    def test_corrupt_state_is_not_silently_discarded(self):
        with tempfile.TemporaryDirectory() as directory:
            path = Path(directory) / "state.json"
            path.write_text('{"bad": true}')
            with self.assertRaises(w.SafetyError):
                w.StateStore(path).read()
    def test_rate_limit_retry_after_prevents_another_request(self):
        api = w.Cloudflare(config(), "secret-never-logged")
        response = {"api": {"ok": False, "error": "Cloudflare HTTP 429", "retryable": True, "retry_after": 120}}
        with mock.patch.object(w, "bounded_jobs", return_value=response) as network:
            with mock.patch.object(w.time, "monotonic", return_value=10):
                with self.assertRaises(w.APIError):
                    api.request("GET", "/test")
            with mock.patch.object(w.time, "monotonic", return_value=20):
                with self.assertRaises(w.Backoff):
                    api.request("GET", "/test")
            self.assertEqual(network.call_count, 1)
        self.assertGreaterEqual(api.retry_at, 130)
    def test_real_worker_deadline_terminates_stalled_process(self):
        start = time.monotonic()
        with mock.patch.object(w, "network_worker", slow_worker):
            result = w.bounded_jobs({"hung": {"timeout": 0.2}})
        self.assertFalse(result["hung"]["ok"])
        self.assertIn("deadline", result["hung"]["error"])
        self.assertLess(time.monotonic() - start, 3)
        self.assertEqual(multiprocessing.active_children(), [])
 class HTTPHealthTests(unittest.TestCase):
    def probe(self, *, status=200, body=b"edge-ok\n", headers=None, cloudflare=False):
        if headers is None:
            headers = {"Cache-Control": "no-store", "CF-Ray": "test-ray", "CF-Cache-Status": "DYNAMIC"}
        response = mock.MagicMock()
        response.status = status
        response.read.return_value = body
        response.getheader.side_effect = lambda name, default=None: headers.get(name, default)
        job = w.make_probe_jobs(config(), None)["canary" if cloudflare else "primary"]
        with mock.patch.object(w.socket, "create_connection"), \
             mock.patch.object(w.ssl, "create_default_context"), \
             mock.patch.object(w, "public_canary_addresses", return_value=["1.0.0.1"]), \
             mock.patch.object(w.http.client, "HTTPResponse", return_value=response):
            return w.probe_http(job)
    def test_rejects_redirect_challenge_and_wrong_body(self):
        for status, body in [(301, b"edge-ok\n"), (403, b"challenge"), (200, b"wrong"), (200, b"edge-ok")]:
            with self.subTest(status=status, body=body):
                self.assertFalse(self.probe(status=status, body=body)["ok"])
    def test_requires_no_store_from_origin(self):
        self.assertFalse(self.probe(headers={})["ok"])
    def test_canary_rejects_cached_or_non_cloudflare_response(self):
        for headers in [
            {"Cache-Control": "no-store", "CF-Ray": "x", "CF-Cache-Status": "HIT"},
            {"Cache-Control": "no-store", "CF-Cache-Status": "BYPASS"},
            {"Cache-Control": "no-store", "CF-Ray": "x"},
        ]:
            with self.subTest(headers=headers):
                self.assertFalse(self.probe(headers=headers, cloudflare=True)["ok"])
    def test_valid_uncached_canary_accepted(self):
        self.assertTrue(self.probe(cloudflare=True)["ok"])
    def test_certificate_verification_error_is_not_accepted(self):
        context = mock.MagicMock()
        context.wrap_socket.side_effect = w.ssl.SSLCertVerificationError("simulated certificate failure")
        with mock.patch.object(w.socket, "create_connection"), \
             mock.patch.object(w.ssl, "create_default_context", return_value=context):
            with self.assertRaises(w.ssl.SSLCertVerificationError):
                w.probe_http(w.make_probe_jobs(config(), None)["primary"])
    def test_canary_connects_to_public_result_with_original_sni_and_host(self):
        response = mock.MagicMock(status=200)
        response.read.return_value = b"edge-ok\n"
        headers = {"Cache-Control": "no-store", "CF-Ray": "x", "CF-Cache-Status": "DYNAMIC"}
        response.getheader.side_effect = lambda name, default=None: headers.get(name, default)
        context = mock.MagicMock()
        job = w.make_probe_jobs(config(), None)["canary"]
        with mock.patch.object(w, "public_canary_addresses", return_value=["1.0.0.1"]) as resolver, \
             mock.patch.object(w.socket, "create_connection") as connect, \
             mock.patch.object(w.ssl, "create_default_context", return_value=context), \
             mock.patch.object(w.http.client, "HTTPResponse", return_value=response):
            self.assertTrue(w.probe_http(job)["ok"])
        connect.assert_called_once_with(("1.0.0.1", 443), timeout=5)
        resolver.assert_called_once_with(job["host"], 5)
        self.assertEqual(context.wrap_socket.call_args.kwargs["server_hostname"], job["host"])
        request = context.wrap_socket.return_value.__enter__.return_value.sendall.call_args.args[0]
        self.assertIn(("Host: " + job["host"] + "\r\n").encode(), request)
    def test_public_dns_failure_never_falls_back_to_lan_resolution(self):
        with mock.patch.object(w, "public_canary_addresses", side_effect=OSError), \
             mock.patch.object(w.socket, "create_connection") as connect:
            result = w.probe_http(w.make_probe_jobs(config(), None)["canary"])
        self.assertFalse(result["ok"])
        connect.assert_not_called()
 class PublicDNSTests(unittest.TestCase):
    def resolve(self, payload):
        response = mock.MagicMock(status=200)
        response.read.return_value = json.dumps(payload).encode()
        with mock.patch.object(w.socket, "create_connection") as connect, \
             mock.patch.object(w.ssl, "create_default_context") as context, \
             mock.patch.object(w.http.client, "HTTPResponse", return_value=response):
            result = w.public_canary_addresses("edge-health.example.com", 5)
        connect.assert_called_once_with(("1.1.1.1", 443), timeout=5)
        self.assertEqual(context.return_value.wrap_socket.call_args.kwargs["server_hostname"], "cloudflare-dns.com")
        return result
    def test_bootstrapped_public_resolver_returns_addresses_for_exact_name(self):
        self.assertEqual(self.resolve({"Status": 0, "Answer": [
            {"name": "edge-health.example.com.", "type": 1, "data": "1.0.0.1"},
            {"name": "unrelated.example.com", "type": 1, "data": "8.8.8.8"}
        ]}), ["1.0.0.1"])
    def test_private_or_failed_dns_answers_rejected(self):
        for payload in [
            {"Status": 3},
            {"Status": 0, "TC": True},
            {"Status": 0, "Answer": []},
            {"Status": 0, "Answer": [{"name": "edge-health.example.com", "type": 1, "data": "192.168.50.10"}]},
        ]:
            with self.subTest(payload=payload), self.assertRaises(w.SafetyError):
                self.resolve(payload)
 class WildcardFixedIPTests(unittest.TestCase):
    def setUp(self):
        self.cfg = config()
        del self.cfg["origin"]
        self.cfg["home_ip"] = "1.1.1.1"
        self.cfg["oracle_ip"] = "8.8.8.8"
        self.cfg["canary"]["name"] = "edge-health.example.com"
        self.cfg["health"]["primary_host"] = self.cfg["canary"]["name"]
        self.cfg["apps"][0]["name"] = "*.example.com"
        self.cfg["apps"][1]["name"] = "example.com"
        self.data = [
            {**self.cfg["canary"], "type": "A", "content": self.cfg["home_ip"], "proxied": True, "ttl": 1},
            *[{**ref, "type": "A", "content": self.cfg["oracle_ip"], "proxied": False, "ttl": 60}
              for ref in self.cfg["apps"]],
            {"id": "a" * 32, "name": "example.com", "type": "MX", "content": "route3.mx.cloudflare.net"},
            {"id": "b" * 32, "name": "example.com", "type": "TXT", "content": "v=spf1 include:_spf.mx.cloudflare.net ~all"},
            {"id": "c" * 32, "name": "test.example.com", "type": "CNAME", "content": "external.example.net"},
            {"id": "d" * 32, "name": "worker.example.com", "type": "Worker", "content": "example-worker"},
        ]
    def load(self, cfg):
        with tempfile.TemporaryDirectory() as directory:
            path = Path(directory) / "config.json"
            path.write_text(json.dumps(cfg))
            return w.load_config(path)
    def test_config_accepts_wildcard_apps_and_same_health_host_for_forced_primary(self):
        cfg = self.load(self.cfg)
        self.assertEqual(cfg["apps"][0]["name"], "*.example.com")
        self.assertEqual(cfg["health"]["primary_host"], cfg["canary"]["name"])
    def test_wildcards_forbidden_in_canary_and_health_hostname(self):
        for location in ("canary", "health"):
            cfg = copy.deepcopy(self.cfg)
            cfg[location]["name" if location == "canary" else "primary_host"] = "*.example.com"
            with self.subTest(location=location), self.assertRaises(w.SafetyError):
                self.load(cfg)
    def test_embedded_and_multiple_wildcards_rejected(self):
        for value in ["app*.example.com", "*.*.example.com", "example.*"]:
            with self.subTest(value=value), self.assertRaises(w.SafetyError):
                w.canonical_name(value, allow_wildcard=True)
    def test_fixed_canary_drift_stops_without_writes(self):
        self.data[0]["content"] = "9.9.9.9"
        api = FakeAPI(self.data)
        with self.assertRaises(w.SafetyError):
            w.Controller(self.cfg, api, MemoryStore(), apply=True).cycle()
        self.assertEqual(api.writes, [])
    def test_only_apex_and_wildcard_written_mail_worker_and_canary_untouched(self):
        api = FakeAPI(self.data)
        ctl = w.Controller(self.cfg, api, MemoryStore(), apply=True, manual_mode="cloudflare")
        before = copy.deepcopy(api.data)
        with mock.patch.object(w, "bounded_jobs", return_value=probes()), mock.patch.object(w, "log"):
            ctl.cycle()
        self.assertEqual({patch["id"] for batch in api.writes for patch in batch}, {"3" * 32, "4" * 32})
        self.assertEqual(api.data[0], before[0])
        self.assertEqual(api.data[3:], before[3:])
        self.assertTrue(all(item["content"] == self.cfg["home_ip"] and item["proxied"] for item in api.data[1:3]))
    def test_cannot_configure_both_fixed_ip_and_ddns_origin(self):
        self.cfg["origin"] = {"id": "1" * 32, "name": "origin.example.com"}
        with self.assertRaises(w.SafetyError):
            self.load(self.cfg)
    def test_local_configuration_needs_no_root_and_never_writes_dns(self):
        spec = importlib.util.spec_from_file_location("configure_home", ROOT / "configure-home.py")
        module = importlib.util.module_from_spec(spec)
        spec.loader.exec_module(module)
        api = FakeAPI(self.data)
        with tempfile.TemporaryDirectory() as directory:
            output = io.StringIO()
            with mock.patch.object(module, "__file__", str(Path(directory) / "configure-home.py")), \
                 mock.patch.object(module.sys, "argv", ["configure-home.py", "--local"]), \
                 mock.patch.object(module.os, "geteuid", return_value=1000), \
                 mock.patch("builtins.input", side_effect=["example.com", "8.8.8.8", "1.1.1.1", "f" * 32, "192.168.20.10"]), \
                 mock.patch.object(module.getpass, "getpass", return_value="private-test-token"), \
                 mock.patch.object(module, "Cloudflare", return_value=api), \
                 mock.patch.object(module.sys, "stdout", output):
                self.assertEqual(module.main(), 0)
            runtime = Path(directory) / "runtime"
            cfg = json.loads((runtime / "watchdog.json").read_text())
            self.assertEqual(cfg["home_ip"], "1.1.1.1")
            self.assertEqual({ref["name"] for ref in cfg["apps"]}, {"*.example.com", "example.com"})
            self.assertEqual(cfg["state_file"], str((runtime / "state.json").resolve()))
            self.assertEqual((runtime / "cf-token").stat().st_mode & 0o777, 0o600)
            self.assertEqual((runtime / "watchdog.json").stat().st_mode & 0o777, 0o600)
            self.assertEqual(runtime.stat().st_mode & 0o777, 0o700)
            self.assertNotIn("private-test-token", output.getvalue())
            self.assertEqual(api.writes, [])
 if __name__ == "__main__":
    unittest.main(verbosity=2)
@@ -0,0 +1,708 @@
 #!/usr/bin/env python3
 """Conservative Cloudflare DNS failover controller. Python 3.9+, Linux/macOS.
 No DNS writes occur without --apply. With origin-record mode, OPNsense/DDNS
 owns the origin and this controller owns the canary and application A records.
 With fixed home_ip mode, only application records are written; canary is read-only.
 """
 import argparse
 import email.utils
 import fcntl
 import http.client
 import ipaddress
 import json
 import math
 import multiprocessing
 import multiprocessing.connection
 import os
 from pathlib import Path
 import random
 import re
 import secrets
 import signal
 import socket
 import ssl
 import sys
 import tempfile
 import time
 import urllib.error
 import urllib.parse
 import urllib.request
 API_ROOT = "https://api.cloudflare.com/client/v4"
 MODES = {"oracle", "cloudflare"}
 class SafetyError(Exception):
    """Configuration or observed state requires operator intervention."""
 class APIError(Exception):
    def __init__(self, message, retryable=False, retry_after=0):
        super().__init__(message)
        self.retryable = retryable
        self.retry_after = retry_after
 class Backoff(APIError):
    pass
 def log(event, **fields):
    print(json.dumps({"time": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()),
                      "event": event, **fields}, sort_keys=True), flush=True)
 def canonical_name(value, allow_wildcard=False):
    if not isinstance(value, str):
        raise SafetyError("DNS names must be strings")
    value = value.lower().rstrip(".")
    if allow_wildcard and value.startswith("*."):
        result = "*." + canonical_name(value[2:])
        if len(result) > 253:
            raise SafetyError("DNS name is too long")
        return result
    if len(value) > 253 or "." not in value or any(
        not re.fullmatch(r"[a-z0-9](?:[a-z0-9-]{0,61}[a-z0-9])?", label)
        for label in value.split(".")
    ):
        raise SafetyError("Use fully qualified ASCII hostnames; no wildcards")
    return value
 def record_refs(cfg):
    return ([cfg["origin"]] if "origin" in cfg else []) + [cfg["canary"], *cfg["apps"]]
 def public_v4(value):
    try:
        address = ipaddress.ip_address(value)
    except ValueError:
        raise SafetyError("Invalid public IPv4 address") from None
    if address.version != 4 or not address.is_global or address.is_multicast:
        raise SafetyError("A globally routable unicast IPv4 address is required")
    return str(address)
 def positive_number(value, name, minimum, maximum):
    if isinstance(value, bool) or not isinstance(value, (int, float)):
        raise SafetyError(f"{name} must be numeric")
    if not math.isfinite(value) or not minimum <= value <= maximum:
        raise SafetyError(f"{name} must be between {minimum} and {maximum}")
    return value
 def load_config(path):
    with open(path, encoding="utf-8") as stream:
        cfg = json.load(stream)
    if not isinstance(cfg, dict):
        raise SafetyError("Configuration must be an object")
    allowed = {"zone_id", "oracle_ip", "origin", "home_ip", "canary", "apps", "health",
               "token_file", "origin_secret_file", "state_file", "interval_seconds",
               "failures_required", "recovery_seconds", "minimum_dwell_seconds",
               "api_timeout_seconds"}
    if set(cfg) - allowed:
        raise SafetyError("Unknown configuration fields: " + ", ".join(sorted(set(cfg) - allowed)))
    if not re.fullmatch(r"[a-fA-F0-9]{32}", str(cfg.get("zone_id", ""))):
        raise SafetyError("zone_id must be the exact 32-character Cloudflare ID")
    cfg["oracle_ip"] = public_v4(cfg["oracle_ip"])
    if ("origin" in cfg) == ("home_ip" in cfg):
        raise SafetyError("Configure exactly one of origin (DDNS record) or home_ip (fixed IPv4)")
    if "home_ip" in cfg:
        cfg["home_ip"] = public_v4(cfg["home_ip"])
        if cfg["home_ip"] == cfg["oracle_ip"]:
            raise SafetyError("Home and Oracle IPv4 addresses must differ")
    if not isinstance(cfg.get("apps"), list) or not 1 <= len(cfg["apps"]) <= 100:
        raise SafetyError("Configure between 1 and 100 application records")
    refs = record_refs(cfg)
    for ref in refs:
        if not isinstance(ref, dict) or set(ref) - {"id", "name", "comment"}:
            raise SafetyError("Record references allow id, name, and optional exact comment")
        if not re.fullmatch(r"[a-fA-F0-9]{32}", str(ref.get("id", ""))):
            raise SafetyError("Every DNS record needs its exact 32-character ID")
        ref["name"] = canonical_name(ref["name"], allow_wildcard=any(ref is app for app in cfg["apps"]))
        if "comment" in ref and not isinstance(ref["comment"], str):
            raise SafetyError("Record ownership comment must be a string")
    if len({r["id"] for r in refs}) != len(refs) or len({r["name"] for r in refs}) != len(refs):
        raise SafetyError("Origin, canary, and app IDs and names must all be distinct")
    health = cfg["health"]
    if not isinstance(health, dict) or set(health) - {"primary_host", "local_ip", "path", "expected_body", "timeout_seconds"}:
        raise SafetyError("Unknown or invalid health configuration")
    health["primary_host"] = canonical_name(health["primary_host"])
    if "origin" in cfg and health["primary_host"] == cfg["origin"]["name"]:
        raise SafetyError("The origin DDNS name must not be the primary health hostname")
    local_ip = ipaddress.ip_address(health["local_ip"])
    if local_ip.version != 4 or local_ip.is_multicast or local_ip.is_unspecified:
        raise SafetyError("health.local_ip must be the direct IPv4 address of NPM")
    if str(local_ip) == cfg["oracle_ip"]:
        raise SafetyError("The local NPM probe must not target Oracle")
    health["local_ip"] = str(local_ip)
    path_value = health["path"]
    if not isinstance(path_value, str) or not re.fullmatch(r"/[A-Za-z0-9_./-]*", path_value):
        raise SafetyError("Health path must be an ASCII absolute path without a query")
    marker = health["expected_body"]
    if not isinstance(marker, str) or not 1 <= len(marker.encode()) <= 4096:
        raise SafetyError("expected_body must be 1 to 4096 UTF-8 bytes; newlines are significant")
    defaults = {"interval_seconds": 15, "failures_required": 4, "recovery_seconds": 300,
                "minimum_dwell_seconds": 600, "api_timeout_seconds": 10}
    ranges = {"interval_seconds": (5, 300), "failures_required": (2, 100),
              "recovery_seconds": (60, 86400), "minimum_dwell_seconds": (60, 86400),
              "api_timeout_seconds": (2, 30)}
    for key, default in defaults.items():
        cfg[key] = positive_number(cfg.get(key, default), key, *ranges[key])
    if int(cfg["failures_required"]) != cfg["failures_required"]:
        raise SafetyError("failures_required must be an integer")
    health["timeout_seconds"] = positive_number(health.get("timeout_seconds", 5), "health timeout", 1, 15)
    cfg.setdefault("state_file", "/var/lib/oracle-edge/state.json")
    for key in ("state_file", "token_file", "origin_secret_file"):
        if key in cfg and (not isinstance(cfg[key], str) or not Path(cfg[key]).is_absolute()):
            raise SafetyError(f"{key} must be an absolute path")
    return cfg
 def read_secret(cfg, key, credential, required=True):
    directory = os.environ.get("CREDENTIALS_DIRECTORY")
    candidate = Path(directory) / credential if directory else None
    if candidate is None or not candidate.is_file():
        candidate = Path(cfg[key]) if cfg.get(key) else None
    if candidate is None:
        if required:
            raise SafetyError(f"Configure {key} or the {credential} systemd credential")
        return None
    value = candidate.read_text(encoding="utf-8").strip()
    if not value or len(value) > 4096 or any(ord(c) < 33 or ord(c) > 126 for c in value):
        raise SafetyError(f"Invalid {credential} credential format")
    if credential == "edge_origin_secret" and not re.fullmatch(r"[a-fA-F0-9]{64}", value):
        raise SafetyError("Origin secret must be exactly 64 hexadecimal characters")
    return value
 class NoRedirect(urllib.request.HTTPRedirectHandler):
    def redirect_request(self, req, fp, code, msg, headers, newurl):
        return None
 def retry_seconds(value):
    if not value:
        return 0
    try:
        return max(0, float(value))
    except ValueError:
        try:
            return max(0, email.utils.parsedate_to_datetime(value).timestamp() - time.time())
        except (ValueError, TypeError, OverflowError):
            return 0
 def api_http(job):
    body = None if job["body"] is None else json.dumps(job["body"]).encode()
    request = urllib.request.Request(API_ROOT + job["path"], data=body, method=job["method"],
        headers={"Authorization": "Bearer " + job["token"], "Content-Type": "application/json",
                 "User-Agent": "oracle-edge-watchdog/3"})
    opener = urllib.request.build_opener(urllib.request.ProxyHandler({}), NoRedirect())
    try:
        with opener.open(request, timeout=job["timeout"]) as response:
            data = response.read(4 * 1024 * 1024 + 1)
        if len(data) > 4 * 1024 * 1024:
            return {"ok": False, "error": "API response exceeded size limit", "retryable": False}
        result = json.loads(data)
        if not isinstance(result, dict) or result.get("success") is not True:
            return {"ok": False, "error": "API reported failure", "retryable": False}
        return {"ok": True, "result": result}
    except urllib.error.HTTPError as exc:
        return {"ok": False, "error": f"Cloudflare HTTP {exc.code}",
                "retryable": exc.code == 429 or 500 <= exc.code <= 599,
                "retry_after": retry_seconds(exc.headers.get("Retry-After"))}
 def public_canary_addresses(host, timeout):
    """Resolve the canary through authenticated public DoH, not LAN split DNS.
    Bootstrap directly to 1.1.1.1; verify TLS for cloudflare-dns.com. No system
    hostname resolution, inherited HTTP proxy, API token, or origin secret is used.
    This runs inside the existing hard-deadline probe worker.
    """
    host = canonical_name(host)
    context = ssl.create_default_context()
    context.minimum_version = ssl.TLSVersion.TLSv1_2
    context.set_alpn_protocols(["http/1.1"])
    query = urllib.parse.urlencode({"name": host, "type": "A"})
    request = ("GET /dns-query?" + query + " HTTP/1.1\r\n"
               "Host: cloudflare-dns.com\r\nAccept: application/dns-json\r\n"
               "Connection: close\r\nUser-Agent: oracle-edge-watchdog/3.1\r\n\r\n")
    with socket.create_connection(("1.1.1.1", 443), timeout=timeout) as raw:
        with context.wrap_socket(raw, server_hostname="cloudflare-dns.com") as tls:
            tls.settimeout(timeout)
            tls.sendall(request.encode("ascii"))
            response = http.client.HTTPResponse(tls)
            response.begin()
            body = response.read(65537)
            if response.status != 200 or len(body) > 65536:
                raise SafetyError("Public DNS HTTPS response failed validation")
    payload = json.loads(body)
    if not isinstance(payload, dict) or payload.get("Status") != 0 or payload.get("TC"):
        raise SafetyError("Public DNS did not return a complete successful answer")
    answers = payload.get("Answer", [])
    if not isinstance(answers, list):
        raise SafetyError("Invalid public DNS answer list")
    addresses = []
    for answer in answers:
        if (isinstance(answer, dict) and answer.get("type") == 1 and
                str(answer.get("name", "")).lower().rstrip(".") == host):
            address = public_v4(answer.get("data", ""))
            if address not in addresses:
                addresses.append(address)
    if not addresses:
        raise SafetyError("Public DNS returned no public IPv4 addresses for the canary")
    return addresses
 def probe_http(job):
    target = job["target"]
    if job.get("cloudflare"):
        try:
            target = random.choice(public_canary_addresses(job["host"], job["timeout"]))
        except Exception as exc:
            return {"ok": False, "error": "Public canary DNS lookup failed: " + type(exc).__name__}
    context = ssl.create_default_context()
    context.minimum_version = ssl.TLSVersion.TLSv1_2
    context.set_alpn_protocols(["http/1.1"])
    headers = {"Host": job["host"], "Connection": "close", "Cache-Control": "no-cache, no-store",
               "Pragma": "no-cache", "User-Agent": "oracle-edge-watchdog/3"}
    # The secret is supplied only to the direct LAN probe, never either public route.
    if job.get("origin_secret"):
        headers["X-Edge-Origin-Auth"] = job["origin_secret"]
    path_value = job["path"] + "?_edge_probe=" + secrets.token_hex(16)
    request = "GET " + path_value + " HTTP/1.1\r\n" + "".join(f"{k}: {v}\r\n" for k, v in headers.items()) + "\r\n"
    with socket.create_connection((target, 443), timeout=job["timeout"]) as raw:
        with context.wrap_socket(raw, server_hostname=job["host"]) as tls:
            tls.settimeout(job["timeout"])
            tls.sendall(request.encode("ascii"))
            response = http.client.HTTPResponse(tls)
            response.begin()
            body = response.read(4097)
            if response.status != 200 or body != job["expected"].encode():
                return {"ok": False, "error": "Unexpected HTTP status or health marker"}
            if "no-store" not in response.getheader("Cache-Control", "").lower():
                return {"ok": False, "error": "Health response must include Cache-Control: no-store"}
            if job.get("cloudflare"):
                cache = response.getheader("CF-Cache-Status", "").upper()
                if not response.getheader("CF-Ray") or cache not in {"DYNAMIC", "BYPASS"}:
                    return {"ok": False, "error": "Canary must traverse Cloudflare with cache bypassed"}
    return {"ok": True}
 def network_worker(connection, job):
    # A process boundary imposes a real deadline on DNS, TLS, headers, and body.
    try:
        result = api_http(job) if job["kind"] == "api" else probe_http(job)
    except Exception as exc:
        # Do not include remote response bodies, request headers, or credentials.
        result = {"ok": False, "error": type(exc).__name__, "retryable": True}
    try:
        connection.send(result)
    finally:
        connection.close()
 def bounded_jobs(jobs):
    context = multiprocessing.get_context("spawn")
    pending, results = {}, {}
    try:
        for name, job in jobs.items():
            receiver, sender = context.Pipe(duplex=False)
            process = context.Process(target=network_worker, args=(sender, job), daemon=True)
            process.start()
            sender.close()
            pending[receiver] = (name, process, time.monotonic() + job["timeout"])
        while pending:
            now = time.monotonic()
            wait_for = max(0, min(deadline for _, _, deadline in pending.values()) - now)
            ready = multiprocessing.connection.wait(list(pending), timeout=min(wait_for, 0.25))
            for receiver in list(pending):
                name, process, deadline = pending[receiver]
                if receiver in ready:
                    try:
                        results[name] = receiver.recv()
                    except (EOFError, OSError):
                        results[name] = {"ok": False, "error": "Network worker exited", "retryable": True}
                elif time.monotonic() >= deadline:
                    results[name] = {"ok": False, "error": "Total network deadline exceeded", "retryable": True}
                else:
                    continue
                receiver.close()
                if process.is_alive():
                    process.terminate()
                process.join(0.25)
                if process.is_alive():
                    process.kill()
                    process.join(0.25)
                del pending[receiver]
        return results
    finally:
        for receiver, (_, process, _) in pending.items():
            receiver.close()
            if process.is_alive():
                process.terminate()
            process.join(0.25)
            if process.is_alive():
                process.kill()
                process.join(0.25)
 class Cloudflare:
    def __init__(self, cfg, token):
        self.cfg, self.token = cfg, token
        self.failures = 0
        self.retry_at = 0
    def request(self, method, path, body=None):
        now = time.monotonic()
        if now < self.retry_at:
            raise Backoff("Cloudflare API backoff active", True, self.retry_at - now)
        result = bounded_jobs({"api": {"kind": "api", "token": self.token, "method": method,
            "path": path, "body": body, "timeout": self.cfg["api_timeout_seconds"]}})["api"]
        if result["ok"]:
            self.failures = 0
            self.retry_at = 0
            return result["result"]
        self.failures += 1
        retryable = result.get("retryable", False)
        delay = min(300, 5 * 2 ** min(self.failures - 1, 6)) + random.uniform(0, 2)
        if not retryable:
            delay = 300
        delay = max(delay, result.get("retry_after", 0))
        self.retry_at = time.monotonic() + delay
        raise APIError(result["error"], retryable, delay)
    def inventory(self):
        records = []
        for page in range(1, 101):
            query = urllib.parse.urlencode({"page": page, "per_page": 100})
            result = self.request("GET", f"/zones/{self.cfg['zone_id']}/dns_records?{query}")
            if not isinstance(result.get("result"), list):
                raise SafetyError("Cloudflare returned an invalid DNS inventory")
            records.extend(result["result"])
            info = result.get("result_info", {})
            if not isinstance(info.get("total_pages"), int) or not 0 <= info["total_pages"] <= 100:
                raise SafetyError("DNS inventory pagination is missing or too large")
            if page >= info["total_pages"]:
                return records
        raise SafetyError("DNS inventory exceeded the supported 10,000-record limit")
    def batch_patch(self, patches):
        if patches:
            self.request("POST", f"/zones/{self.cfg['zone_id']}/dns_records/batch", {"patches": patches})
 def validate_inventory(cfg, records):
    if not isinstance(records, list) or any(not isinstance(r, dict) for r in records):
        raise SafetyError("Invalid DNS inventory")
    by_id = {}
    for ref in record_refs(cfg):
        named = [r for r in records if str(r.get("name", "")).lower().rstrip(".") == ref["name"]]
        address_records = [r for r in named if r.get("type") in {"A", "AAAA", "CNAME", "HTTPS", "SVCB"}]
        if len(address_records) != 1:
            raise SafetyError(f"{ref['name']}: require exactly one A and no AAAA/CNAME/HTTPS/SVCB records")
        record = address_records[0]
        if record.get("id") != ref["id"] or record.get("type") != "A":
            raise SafetyError(f"{ref['name']}: DNS record type or ID changed")
        if "comment" in ref and record.get("comment") != ref["comment"]:
            raise SafetyError(f"{ref['name']}: ownership comment changed")
        if not isinstance(record.get("proxied"), bool) or not isinstance(record.get("ttl"), int):
            raise SafetyError(f"{ref['name']}: invalid proxy status or TTL")
        public_v4(record.get("content", ""))
        by_id[ref["id"]] = record
    if "origin" in cfg:
        origin = by_id[cfg["origin"]["id"]]
        home_ip = origin["content"]
        if origin["proxied"] or home_ip == cfg["oracle_ip"]:
            raise SafetyError("Origin must be DNS-only home IPv4, different from Oracle")
    else:
        home_ip = cfg["home_ip"]
    canary = by_id[cfg["canary"]["id"]]
    if not canary["proxied"] or canary["content"] == cfg["oracle_ip"]:
        raise SafetyError("Canary must already be proxied to a home address")
    if "home_ip" in cfg and (canary["content"] != home_ip or canary["ttl"] != 1):
        raise SafetyError("Fixed-IP mode requires the read-only canary to match home_ip, proxied, TTL Auto")
    for ref in cfg["apps"]:
        record = by_id[ref["id"]]
        if record["proxied"]:
            if record["content"] == cfg["oracle_ip"]:
                raise SafetyError(f"{ref['name']}: unexpected proxied Oracle address")
        elif record["content"] != cfg["oracle_ip"]:
            raise SafetyError(f"{ref['name']}: unexpected DNS-only target; refusing takeover")
    return by_id, home_ip
 def observed_mode(cfg, by_id):
    records = [by_id[ref["id"]] for ref in cfg["apps"]]
    if all(not r["proxied"] and r["content"] == cfg["oracle_ip"] for r in records):
        return "oracle"
    if all(r["proxied"] for r in records) and len({r["content"] for r in records}) == 1:
        return "cloudflare"
    return "mixed"
 def patches_for(refs, by_id, content, proxied):
    desired = {"content": content, "proxied": proxied, "ttl": 1 if proxied else 60}
    return [{"id": ref["id"], **desired} for ref in refs
            if any(by_id[ref["id"]].get(key) != value for key, value in desired.items())]
 class StateStore:
    def __init__(self, path):
        self.path = Path(path)
    def read(self):
        if not self.path.exists():
            return None
        try:
            state = json.loads(self.path.read_text(encoding="utf-8"))
            valid = (isinstance(state, dict) and state.get("version") == 1 and
                state.get("mode") in MODES and isinstance(state.get("last_switch"), (int, float)) and
                math.isfinite(state["last_switch"]) and state["last_switch"] >= 0)
            if not valid:
                raise ValueError()
            return state
        except (ValueError, TypeError):
            raise SafetyError("Invalid state file; use explicit --once --mode after investigation") from None
    def write(self, mode, last_switch):
        data = {"version": 1, "mode": mode, "last_switch": last_switch}
        self.path.parent.mkdir(mode=0o700, parents=True, exist_ok=True)
        descriptor, temp_name = tempfile.mkstemp(prefix=".state-", dir=self.path.parent)
        try:
            with os.fdopen(descriptor, "w", encoding="utf-8") as stream:
                json.dump(data, stream, sort_keys=True)
                stream.write("\n")
                stream.flush()
                os.fsync(stream.fileno())
            os.replace(temp_name, self.path)
            directory = os.open(self.path.parent, os.O_RDONLY | os.O_DIRECTORY)
            try:
                os.fsync(directory)
            finally:
                os.close(directory)
        finally:
            if os.path.exists(temp_name):
                os.unlink(temp_name)
 class HealthWindow:
    def __init__(self, cfg):
        self.cfg = cfg
        self.failures = 0
        self.healthy_since = None
        self.last_sample = None
    def reset(self):
        self.failures = 0
        self.healthy_since = None
        self.last_sample = None
    def sample(self, probes, now):
        if self.last_sample is not None and now - self.last_sample > 2 * self.cfg["interval_seconds"]:
            self.reset()
        self.last_sample = now
        if probes["primary"]["ok"]:
            self.failures = 0
            if self.healthy_since is None:
                self.healthy_since = now
        else:
            self.failures += 1
            self.healthy_since = None
    def target(self, mode, probes, now, dwell_elapsed):
        if mode == "oracle" and self.failures >= self.cfg["failures_required"]:
            if probes["local"]["ok"] and probes["canary"]["ok"]:
                return "cloudflare"
        if mode == "cloudflare" and self.healthy_since is not None:
            if (probes["local"]["ok"] and probes["primary"]["ok"] and
                now - self.healthy_since >= self.cfg["recovery_seconds"] and
                dwell_elapsed >= self.cfg["minimum_dwell_seconds"]):
                return "oracle"
        return mode
 def make_probe_jobs(cfg, origin_secret):
    health = cfg["health"]
    shared = {"kind": "probe", "path": health["path"], "expected": health["expected_body"],
              "timeout": health["timeout_seconds"]}
    return {
        "primary": {**shared, "target": cfg["oracle_ip"], "host": health["primary_host"]},
        "local": {**shared, "target": health["local_ip"], "host": health["primary_host"],
                  "origin_secret": origin_secret},
        "canary": {**shared, "target": cfg["canary"]["name"], "host": cfg["canary"]["name"],
                   "cloudflare": True},
    }
 class Controller:
    def __init__(self, cfg, api, store, apply=False, manual_mode=None, check=False, origin_secret=None):
        self.cfg, self.api, self.store = cfg, api, store
        self.apply, self.manual_mode, self.check = apply, manual_mode, check
        self.origin_secret = origin_secret
        self.mode = None
        self.last_switch = None
        self.dwell_started = None
        self.window = HealthWindow(cfg)
        self.startup_complete = False
    def inventory(self):
        return validate_inventory(self.cfg, self.api.inventory())
    def initialize(self, by_id, now):
        observed = observed_mode(self.cfg, by_id)
        if observed == "mixed" and self.manual_mode is None:
            raise SafetyError("Mixed DNS at startup: inspect records, then use --once --mode oracle or cloudflare")
        # Manual recovery can replace corrupt state, but only after successful target probes.
        state = None if self.manual_mode else self.store.read()
        if state and state["mode"] != observed:
            raise SafetyError("Stored mode disagrees with DNS: inspect, then use explicit --once --mode")
        self.mode = state["mode"] if state else (observed if observed in MODES else None)
        self.last_switch = state["last_switch"] if state else time.time()
        # Conservative: every restart restarts minimum dwell and continuous health timers.
        self.dwell_started = now
        self.startup_complete = True
        log("startup", observed=observed, desired=self.mode, apply=self.apply)
    def apply_and_verify(self, patches, refs, content, proxied, expected_home):
        self.api.batch_patch(patches)
        by_id, home_ip = self.inventory()
        if home_ip != expected_home:
            raise APIError("Origin changed during update; reconcile with a fresh inventory", True)
        if patches_for(refs, by_id, content, proxied):
            raise APIError("DNS readback does not match desired state; reconciliation pending", True)
        return by_id, home_ip
    def cycle(self):
        by_id, home_ip = self.inventory()
        if not self.startup_complete:
            self.initialize(by_id, time.monotonic())
        canary_refs = [self.cfg["canary"]]
        # In fixed-IP mode the canary is read-only; only application A records are written.
        canary_patches = patches_for(canary_refs, by_id, home_ip, True) if "origin" in self.cfg else []
        if canary_patches:
            log("canary_update_needed", apply=self.apply and not self.check, home_ip=home_ip)
            if self.apply and not self.check:
                by_id, home_ip = self.apply_and_verify(canary_patches, canary_refs, home_ip, True, home_ip)
        probes = bounded_jobs(make_probe_jobs(self.cfg, self.origin_secret))
        now = time.monotonic()
        self.window.sample(probes, now)
        log("health", mode=self.mode, failures=self.window.failures,
            results={key: ({"ok": value["ok"]} if value["ok"] else {"ok": False, "error": value["error"]})
                     for key, value in probes.items()})
        if self.check:
            if canary_patches or not all(result["ok"] for result in probes.values()):
                raise SafetyError("Preflight incomplete: canary drift or a route health probe failed")
            return
        if self.manual_mode:
            target = self.manual_mode
            healthy = probes["local"]["ok"] and probes["primary" if target == "oracle" else "canary"]["ok"]
            if not healthy or (target == "cloudflare" and canary_patches and not self.apply):
                raise SafetyError("Manual target is not proven healthy; DNS left unchanged")
        else:
            target = self.window.target(self.mode, probes, now, now - self.dwell_started)
        # A stale canary cannot justify switching to a newly discovered home address.
        if target == "cloudflare" and canary_patches and not self.apply:
            log("hold", reason="Dry run cannot repair canary drift before validating failover")
            return
        content, proxied = (self.cfg["oracle_ip"], False) if target == "oracle" else (home_ip, True)
        patches = patches_for(self.cfg["apps"], by_id, content, proxied)
        transition = target != self.mode or bool(self.manual_mode)
        target_ready = probes["local"]["ok"] and probes["primary" if target == "oracle" else "canary"]["ok"]
        if (patches or transition) and not target_ready:
            log("hold", reason="Desired target is not proven healthy; preserving observed application DNS")
            return
        if not patches and not transition:
            if self.apply and self.store.read() is None:
                self.store.write(self.mode, self.last_switch)
            return
        if not self.apply:
            log("dry_run", target=target, patches=patches)
            return
        # Persist intent BEFORE a potentially ambiguous API write. A running process
        # keeps reconciling this intent even if the API response or readback is lost.
        if transition:
            switched_at = time.time()
            self.store.write(target, switched_at)
            self.mode, self.last_switch, self.dwell_started = target, switched_at, now
            self.window.reset()
        else:
            self.store.write(self.mode, self.last_switch)
        self.apply_and_verify(patches, self.cfg["apps"], content, proxied, home_ip)
        log("dns_verified", desired=self.mode, records_changed=len(patches),
            note="API readback verified; resolver caches and existing connections may retain the old route")
 def main(argv=None):
    parser = argparse.ArgumentParser(description=__doc__)
    parser.add_argument("--config", required=True, help="Absolute path to JSON configuration")
    run_mode = parser.add_mutually_exclusive_group()
    run_mode.add_argument("--check", action="store_true", help="Read-only inventory and all-route health preflight")
    run_mode.add_argument("--once", action="store_true", help="One control cycle (default)")
    run_mode.add_argument("--run", action="store_true", help="Continuous controller")
    parser.add_argument("--apply", action="store_true", help="Explicitly permit configured DNS and state writes")
    parser.add_argument("--mode", choices=sorted(MODES), help="Manual one-shot repair/switch after target probes pass")
    args = parser.parse_args(argv)
    if args.check and args.apply:
        parser.error("--check is read-only and cannot be combined with --apply")
    if args.mode and (args.run or args.check):
        parser.error("--mode is for a one-shot operation; stop the running service first")
    cfg = load_config(args.config)
    token = read_secret(cfg, "token_file", "cf_token")
    origin_secret = read_secret(cfg, "origin_secret_file", "edge_origin_secret", required=False)
    state_path = Path(cfg["state_file"])
    # Lock even check/dry-run invocations to avoid diagnostics racing the writer.
    state_path.parent.mkdir(mode=0o700, parents=True, exist_ok=True)
    lock = open(str(state_path) + ".lock", "a", encoding="utf-8")
    os.chmod(lock.name, 0o600)
    try:
        fcntl.flock(lock, fcntl.LOCK_EX | fcntl.LOCK_NB)
    except BlockingIOError:
        raise SafetyError("Another watchdog owns this state file; stop its service before manual commands") from None
    controller = Controller(cfg, Cloudflare(cfg, token), StateStore(state_path), args.apply,
                            args.mode, args.check, origin_secret)
    stopping = False
    def stop(_signum, _frame):
        nonlocal stopping
        stopping = True
    signal.signal(signal.SIGTERM, stop)
    signal.signal(signal.SIGINT, stop)
    next_cycle = time.monotonic()
    while not stopping:
        try:
            controller.cycle()
        except APIError as exc:
            controller.window.reset()
            log("api_hold", reason=str(exc), retry_seconds=round(exc.retry_after, 1))
            if not args.run:
                return 1
        except SafetyError:
            # Invalid identities, conflicting state, or unexpected DNS is not an outage.
            # Exit and let the operator inspect it instead of overwriting unrelated data.
            raise
        if not args.run:
            return 0
        next_cycle += cfg["interval_seconds"]
        now = time.monotonic()
        if next_cycle <= now:
            next_cycle = now + cfg["interval_seconds"]
        while not stopping and time.monotonic() < next_cycle:
            time.sleep(min(0.25, max(0, next_cycle - time.monotonic())))
    log("stopped")
    return 0
 if __name__ == "__main__":
    try:
        sys.exit(main())
    except (SafetyError, KeyError, ValueError, OSError) as exc:
        # Configuration paths are safe to show; credentials and API bodies are never logged.
        log("fatal", reason=str(exc))
        sys.exit(2)