ZFS Storage Pool Mirror Recovery

Our SmartOS server (hosted at Hetzner) had two identical drives in it. When we installed SmartOS, we configured the drives to be mirrored. One day, while testing an upgrade script, we noticed that one of the hard drives was not listed in the output of the diskinfo command:

[root@smartos ~]# diskinfo
TYPE    DISK                    VID      PID              SIZE          RMV SSD
USB     c1t0d0                  JetFlash Transcend 16GB     14.52 GiB   yes no
UNKNOWN c2t0d0                  TOSHIBA  DT01ACA300       2794.52 GiB   no  no

(This is a test system, so it is not properly monitored for hardware failures.)

We rebooted the system into Hetznerʼs Linux Rescue system and observed that only one of the hard drives was seen by Linux. A Hetzner technician examined the system and replaced the failed drive (after we acknowledged the “loss” of data on the failed drive.)

After booting into SmartOS, both drives were visible again:

[root@smartos ~]# diskinfo
TYPE    DISK                    VID      PID              SIZE          RMV SSD
USB     c1t0d0                  JetFlash Transcend 16GB     14.52 GiB   yes no
UNKNOWN c2t0d0                  TOSHIBA  DT01ACA300       2794.52 GiB   no  no
UNKNOWN c2t1d0                  TOSHIBA  DT01ACA300       2794.52 GiB   no  no

This system uses the SmartOS default configuration of one zpool:

[root@smartos ~]# zpool list
NAME    SIZE  ALLOC   FREE  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
zones  2.72T   335G  2.39T         -     5%    12%  1.00x  ONLINE  -

Only one of the drives is now in the zpool:

[root@smartos ~]# zpool status zones
  pool: zones
 state: ONLINE
  scan: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        zones       ONLINE       0     0     0
          c2t1d0    ONLINE       0     0     0

errors: No known data errors

(At one point, bother drives were in the zpool, but that was many reboots and one hardware change ago.)

Next, we added the new hard drive to the existing zones zpool and let ZFS resilver the drive:

[root@smartos ~]# zpool attach zones c2t1d0 c2t0d0
[root@smartos ~]# zpool status zones
  pool: zones
 state: ONLINE
status: One or more devices is currently being resilvered.  The pool will
        continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
  scan: resilver in progress since Mon Feb 13 07:16:34 2017
    14.5M scanned out of 335G at 1.45M/s, 65h35m to go
    14.5M resilvered, 0.00% done
config:

        NAME        STATE     READ WRITE CKSUM
        zones       ONLINE       0     0     0
          mirror-0  ONLINE       0     0     0
            c2t1d0  ONLINE       0     0     0
            c2t0d0  ONLINE       0     0     0  (resilvering)

errors: No known data errors

The key in the the zpool attach command. Here is the relevant text from man zpool:

zpool attach [-f] pool device new_device
        Attaches new_device to the existing device.  The existing device
        cannot be part of a raidz configuration. If device is not
        currently part of a mirrored configuration, device automatically
        transforms into a two-way mirror of device and new_device.  If
        device is part of a two-way mirror, attaching new_device creates
        a three-way mirror, and so on. In either case, new_device begins
        to resilver immediately.