Deciding the primary is dead
Automatic failover needs a machine to decide your primary is dead, and that decision is harder than it sounds.
Start from your health check, the deciding input. A shallow one asks whether a connection opens, which catches a dead process and happily passes a server that accepts connections and errors on every query.
A deep one exercises real work, touching your database, and catches more. It also fails when a dependency hiccups rather than the machine itself.
Tune it around a probe every few seconds with a threshold of three consecutive failures. That trades roughly 10 to 30 seconds of detection delay against false alarms from a single dropped packet.
Care about those false alarms, because failover is not free. A primary demoted on a network blip and then promoted back is flapping, and every flap drops connections and risks losing writes in flight. Aggressive thresholds feel responsive right up until they cause more downtime than they prevent.
Dead and unreachable look identical
Face the deep problem. From your checker's chair, a crashed primary and an unreachable primary look identical.
If your primary is actually alive on the far side of a network partition, promoting the standby gives you two machines that both believe they are primary. Both accept writes. That is split brain, and it turns an outage into a data corruption incident. The two sets of data diverge with every write, and somebody has to merge them by hand later.
Use the defences, which are old and proven. Run an odd number of machines and require a majority to elect a primary, so the minority side of a partition demotes itself.
Fence the old primary before promoting anyone, cutting its power or revoking its storage access. Or hand out a timed lease the primary must keep renewing, where no renewal means no authority.
Never build failover with exactly two machines and nothing to break the tie.
Worked example
Tunde inherits a two-node MySQL setup managed by keepalived: whichever node holds the VIP is primary. During a switch firmware upgrade the nodes lose sight of each other for about 90 seconds while both still reach their local app servers. Node B declares A dead and grabs the VIP on its side; A never releases it. For those 90 seconds both accept writes, and when the network heals there are 4,116 rows that exist on one node only, including 212 conflicting order updates. Reconciliation eats the whole weekend, two engineers reading diffs row by row. The rebuild adds a third node in another rack and moves election to a quorum-based tool, so a partitioned minority now demotes itself instead of seceding.