Did the power outage trigger my disk to fail?

Introduction

In my role as family system administrator, I try to keep the NAS running -- don't want to lose all those billions of cat photos that the kids take.  On my NAS, I run the following servers to collect and export machine measurements:

• collectd  
• Prometheus Collectd Exporter

On another machine, I run Prometheus to collect, store, and analyze my measurements.  For more details about Prometheus, you might want to review the "Prometheus: A practical guide to alerting at scale" presentation and slides.

Bad Disk Investigation

When I awoke on Friday the 27th, I noticed the electric power must have went out during the night because the clock on the microwave was flashing.  The NAS didn't reboot because I use a UPS. That is when I noticed a hard disk failed and now needs to be replaced.  I replaced the drive two days later. But I wondered -- did the power event cause the drive to fail?  The answer turned out to be no -- but I thought it would be useful to capture the data I used to make this conclusion.

Timeline

• Fri, 20 Nov 2015 10:51:48 GMT:monitoring noticed drive failed
• Fri, 27 Nov 2015 05:21:15 GMT:very short power event
• Fri, 27 Nov 2015 12:00:00 GMT:noticed hard disk failed
• Sat, 28 Nov 2015 20:24:52 GMT: replaced failed hard disk

When did the drive fail?

I use smartctl to measure the temperature of my hard disks.  I noticed that the temperature of one of my disks was missing.  This measurement helped me determine when the drive failed.

When did the power event happen?

I poll the status of the UPS every 10 seconds.  The power event had to be shorter than 10 seconds, because the monitoring never even noticed this event -- the UPS status was always "on mains".  I found a spike in the UPS remaining capacity measurement -- that is the only clue I had about when the power event happened.   Comparing these two graphs showed that the disk failed about a week before the power event.

Did any alerts fire?

Prometheus stores pending/firing alerts in the time series database.  I wanted to know if there were any firing alerts that I missed -- did I miss an alert for a week about my bad disk?  Nope -- the only firing alert was when I rebooted the NAS.  I will need to add another alert for failing disks.

What did I learn?

• The FreeBSDism of using 'sysctl -n kern.disks' to find a list of disks in a machine appears to quietly drop failed or missing hard disks -- this is a feature for removable USB drives. I cannot use this measurement to track the disks expected to be alive on a machine.
• Polling the UPS might not give the required data for very short power events.  I might want to enable SNMP traps to capture very short power events.
• I don't want to leave my raid array with one dead disk for a whole week.  I am going to setup some alerts for this!