Dual carrier telemetry networks: keeping remote sites online when a carrier fails
Most remote telemetry sites rely on a single mobile network, and when it fails, the site goes quiet. Here is how a dual carrier design works, what it protects against, and where these designs usually break down.
What a dual carrier telemetry network is
A telemetry network carries data between remote sites and a control system: pump states, levels, flows, alarms and control commands. At remote sites the link is usually cellular, because running fibre or copper to every pump station or reservoir is not practical.
A dual carrier design gives each site two independent paths back to the control system, over two different carriers. If one carrier has an outage, a tower fault or poor coverage at that site, traffic moves to the other path without anyone attending site.
The two carrier paths are only the underlay. On top of them sits an encrypted overlay network that carries the actual telemetry, so the design works the same way whichever carrier is in use.
Why one carrier is not enough
Carrier outages are not rare, and they are not always local. On 8 November 2023 a nationwide outage took one carrier's mobile, internet and fixed services offline for around 12 hours, affecting an estimated 10.2 million subscribers and 400,000 small businesses. Customers of smaller providers that resell that carrier's network lost service too.
For a remote site on one carrier, an outage like that means:
- no alarms reach the control room, so a pump failure or overflow risk goes unseen
- no remote control, so operators drive to sites to check or run plant by hand
- gaps in the data used for compliance reporting
- a backlog of site visits once service returns
Smaller events matter too: a single tower losing power, a local backhaul fault, or planned overnight works. A second carrier turns each of these from a site visit into an alarm on a dashboard.
Two common site patterns
Most remote sites fit one of two patterns.
| Pattern | Primary path | Second path | Typical sites |
|---|---|---|---|
| Dual cellular | Carrier A, 4G or 5G | Carrier B, 4G or 5G | Pump stations, reservoirs, bores and monitoring points with no fixed service |
| Fixed plus cellular | Fixed service (fibre, fixed wireless or a private WAN) | Cellular on a different carrier to the fixed service | Treatment plants, depots and larger sites |
In both patterns the second path should use a different carrier and, where possible, different physical infrastructure to the first. Two services that share a tower, a backhaul link or a wholesale network share the same failure.
Design principles
Treat both carriers as untrusted
A private APN keeps telemetry traffic off the public internet, but the carrier still carries it. Design as if either carrier network could be misconfigured or compromised.
- Encrypt and authenticate all traffic at the site router, so the carrier only ever sees encrypted tunnels.
- Put the carrier-facing interfaces in their own routing table (a front-door VRF), so the cellular network cannot route into the site or control network.
- On carrier-facing interfaces, allow only the tunnel protocols (IKE and ESP) and drop everything else.
- Manage routers only through the encrypted overlay, using SSH and SNMPv3, never directly over a carrier interface.
Build an encrypted overlay over both carriers
Use a hub-and-spoke IPsec overlay, such as DMVPN or a comparable SD-WAN overlay. Each site builds a tunnel over each carrier to each hub. Use IKEv2, strong ciphers and per-site credentials, ideally certificates, so one compromised site does not expose the rest.
The overlay carries the site LANs. That keeps telemetry addressing under your control, and changing a carrier becomes an underlay change rather than a redesign.
Keep carrier-side configuration simple
Each carrier needs to do very little: attach the SIM to your private APN and give it a fixed address, assigned and held by the carrier. Fixed addresses let the hubs accept tunnels only from known sites and make faults easier to trace.
Avoid carrier-side arrangements that quietly become part of your network:
- Carrier routing of site LANs. Some designs ask the carrier to route each site's LAN subnet to its SIM (framed routes). The overlay already does this. Carrier routes duplicate it, are invisible to your team, and get left behind when services change.
- Authentication that depends on your own servers. If SIM authentication relies on a RADIUS server you host, sites cannot attach when that server, or the path to it, is down. If you need one, treat it as critical: redundant, monitored and documented with the carrier.
Whatever is configured on the carrier side, write it down and review it at least yearly. Undocumented carrier configuration is the part most likely to break during a migration.
Use two hubs with a clear preference
Terminate the overlay on two hub routers, ideally in separate locations, or at least on separate power and upstream links. Run a routing protocol such as BGP over the tunnels and make the path preference deliberate: which carrier and which hub carry traffic on a normal day, and in what order the others take over.
Make sure return traffic follows the same path. Asymmetric routing through stateful firewalls at the head end is a common cause of failover that only works in one direction.
Detect failure quickly, with more than link state
A cellular interface can show "up" while passing no traffic: registered on the network, but with a stalled data session. Link state alone will not trigger failover.
- Enable IKE dead peer detection (DPD) so tunnels to an unreachable peer are torn down. Without it, one end can hold a tunnel it thinks is up long after the other end has restarted, and traffic is dropped until the security association expires.
- Tune routing timers, or use BFD where supported, so a dead path is withdrawn in seconds rather than minutes.
- Track reachability to the hubs over each carrier, and reset the modem automatically if a path stays down.
Set the targets from the process. If an alarm must reach the control room within a minute, failover has to finish well inside that.
Fail back on its own, without flapping
When the preferred path returns, the site should move back to it without a visit. Add hold-down timers so a marginal signal does not cause repeated switching, and test failback as carefully as failover.
Get the physical layer right
Most telemetry outages that are not carrier outages start at the antenna or the power supply.
- Measure coverage on site. Coverage maps are a guide only. Record signal strength and quality (RSRP, RSRQ and SINR) for both carriers at the actual antenna position before you commit to a design.
- Mind antenna direction. Each carrier's nearest tower may be in a different direction. A directional antenna aimed for one carrier can leave the other with poor signal, so choose antennas per carrier where needed.
- Protect against surges. Fit coaxial surge arrestors where cables enter the cabinet and bond them to the site earth. Assess lightning risk to AS/NZS 1768.
- Keep the router powered. Run the router from the site DC supply or a UPS, and raise a power fail alarm before it drops out.
Monitor both paths, not just the active one
A backup path that is never watched is usually found dead on the day it is needed. Monitor the standby path as closely as the active one.
- Poll signal, tunnel and routing state for each carrier using SNMPv3.
- Send router logs to a central syslog server, with every device on the same time source (NTP), so events line up.
- Alert on the loss of either path, not only on the loss of the whole site.
Plan for replacement
Hold a pre-configured spare router for each site type. Keep configurations templated and in version control, so a replacement is a swap rather than a rebuild. Keep secrets such as keys and passwords out of the templates and in a secure store. Maintain a register that maps each SIM to its site and fixed address.
Where dual carrier designs usually break down
- Shared failure points. Two SIMs from different brands can run on the same wholesale network, and two carriers can share a tower, an antenna, a power supply or a router. Real diversity means separate points of failure.
- Dual SIM, single modem. Many routers hold two SIMs but have one modem, so only one SIM is live at a time. Switching can take minutes, and the standby SIM is never proven until it is needed. Dual-modem routers keep both paths up and monitored.
- Interface up, traffic down. No dead peer detection or path tracking, so the router never notices the path has failed.
- Carrier configuration nobody owns. APN settings, carrier routes or authentication set up years ago and never documented, which break when services are moved or retired.
- Failover tested once. Tested at commissioning and never again, and failback never tested at all.
- Coverage assumed from maps. Designs signed off before anyone measured signal at the site.
- Asymmetric routing at the head end. Traffic leaves through one hub and returns through the other, and a stateful firewall drops it.
- Surge arrestors not bonded. Arrestors fitted but not connected to the site earth, which gives little or no protection.
A pre-cutover test plan
Run these tests at every site before it goes into service, and record the results in the site handover pack.
- Measure and record signal for both carriers at the installed antenna position.
- Confirm each SIM receives its expected fixed address on the private APN.
- Confirm all tunnels are up over both carriers to both hubs.
- Fail the primary carrier (disconnect the antenna or shut the modem) and time how long until SCADA polling resumes on the second carrier.
- Restore the primary carrier, confirm the site fails back without flapping, and record the time.
- Repeat with the second carrier as the failed path.
- Fail each hub in turn and confirm the site stays reachable.
- Power cycle the router and confirm it returns unaided on the preferred path.
- Confirm monitoring raised and cleared an alert for every failure above.
- Confirm logs arrived centrally with correct timestamps.
Repeat the failover tests on a schedule after go-live, for example yearly and after any carrier or hub change.
How we help, and where it stops
We design, build and test dual carrier telemetry networks for utilities and industrial operators. That covers carrier and APN requirements, overlay and routing design, router and hub configuration, firewall rules, site installation standards, failover test plans and handover documentation. We are vendor-neutral and work alongside your SCADA and OT teams.
We do not run the carrier networks. Carrier-side changes still go through the carrier, coverage at each site has to be measured rather than promised, and tower or mast work is done by licensed riggers. This guide is general information, not a design for any specific site.
If your sites also fall under the SOCI Act, see our guide to SOCI CIRMP compliance in 2026.
Sources
- Senate Environment and Communications References Committee, Optus network outage, Chapter 2
- ASD's ACSC and international partners, Principles of operational technology cyber security (October 2024)
- NIST SP 800-82 Rev. 3, Guide to Operational Technology (OT) Security (September 2023)
- IETF RFC 3706, A Traffic-Based Method of Detecting Dead Internet Key Exchange (IKE) Peers
- IETF RFC 7296, Internet Key Exchange Protocol Version 2 (IKEv2)
- Standards Australia, AS/NZS 1768:2021 Lightning protection