Does anyone do this extra sort of network monitoring? Idea I heard today.

It's straightforward enough:

• Use/keep your existing network, system, and application monitoring (Nagios, Solarwinds, whatever).
• Draft up a simple uniform bash script for each of your systems that does a couple of ping tests 24x7 at various targets. Install and cron it onto each of your core systems.
• Ping common major permanent IPs -- your main router/switch, basically core internal systems across various internal network segments that the system in question that's doing the pinging should always be able to internally reach for critical operations.
• Each box pings the Big Stuff but also it's own specific targets.
• If the pings succeed, do nothing.
• If they fail, write it to a log file and create some sort of marker/alert (whatever works for you, snmp, mail, etc) to pop a notification on your existing monitoring infrastructure to go take a look.

The idea came up because someone had extensive monitoring watching everything from a central Nagios, which worked well enough. However, there were periodic network failures only between two interfaces on two machines talking to each other on a local switch. Turned out a certain transceiver on a certain port was starting to flake out.

The idea was that this extra background monitoring from the internal perspective of the individual systems would help cover that 'last mile' of possibly paranoid level of coverage. Bilateral monitoring?