RAID device not listed by df

craibuc · July 14, 2017, 1:45pm

I followed Paul Totterman’s instructions to create a RAID1 device, which seemed to work.

I manually mounted the device:

mount /dev/md0 /mnt/raid

I was able to write files to the device.

I rebooted the router.

The two HDDs are sda and sdb:

# lsblk
NAME         MAJ:MIN RM  SIZE RO TYPE  MOUNTPOINT
sda            8:0    0  3.7T  0 disk  
`-md0          9:0    0  3.7T  0 raid1 
sdb            8:16   0  3.7T  0 disk  
`-md0          9:0    0  3.7T  0 raid1 
sdc            8:32   1 29.1G  0 disk  
`-sdc1         8:33   1 29.1G  0 part  /tmp/run/mountd/sdc1
mtdblock0     31:0    0    1M  0 disk  
mtdblock1     31:1    0    7M  0 disk  
mmcblk0      179:0    0  7.3G  0 disk  
`-mmcblk0p1  179:1    0  7.3G  0 part  /
mmcblk0boot0 179:8    0    4M  1 disk  
mmcblk0boot1 179:16   0    4M  1 disk  
mmcblk0rpmb  179:24   0    4M  0 disk

Device is not listed:

# df -h
Filesystem                Size      Used Available Use% Mounted on
/dev/mmcblk0p1            7.3G      4.7G      2.7G  64% /
tmpfs                  1011.1M    556.0K   1010.6M   0% /tmp
tmpfs                   512.0K      4.0K    508.0K   1% /dev
/dev/sdc1                28.5G      1.4G     25.6G   5% /mnt/kingston
/dev/sdc1                28.5G      1.4G     25.6G   5% /tmp/run/mountd/sdc1

However, a recursive listing of the device indicates that the files and folders are present:

# ls -lR /mnt/raid
/mnt/raid:
drwxrwxrwx    1 root     root            36 Jul 13 06:58 media

/mnt/raid/media:
drwxrwxrwx    1 root     root            38 Jul 12 19:59 Music
drwxrwxrwx    1 root     root            22 Jul 12 20:00 Pictures
drwxrwxrwx    1 root     root           364 Jul 13 21:53 Video
...

When I attempt to copy a large file (800MB) to the RAID device, it appears that it is copying it to /dev/mmcblk0p1:

# df -h
Filesystem                Size      Used Available Use% Mounted on
/dev/mmcblk0p1            7.3G      4.8G      2.6G  65% /
tmpfs                  1011.1M    560.0K   1010.6M   0% /tmp
tmpfs                   512.0K      4.0K    508.0K   1% /dev
/dev/sdc1                28.5G      1.4G     25.6G   5% /mnt/kingston
/dev/sdc1                28.5G      1.4G     25.6G   5% /tmp/run/mountd/sdc1
# df -h
Filesystem                Size      Used Available Use% Mounted on
/dev/mmcblk0p1            7.3G      5.6G      1.8G  75% /
tmpfs                  1011.1M    560.0K   1010.6M   0% /tmp
tmpfs                   512.0K      4.0K    508.0K   1% /dev
/dev/sdc1                28.5G      1.4G     25.6G   5% /mnt/kingston
/dev/sdc1                28.5G      1.4G     25.6G   5% /tmp/run/mountd/sdc1

But the file seems to have been added to the RAID device:

root@turris:/mnt/raid# ll
drwxr-xr-x    1 root     root           186 Jul 14 08:28 [Folder containing large file]/
drwxrwxrwx    1 root     root            36 Jul 13 06:58 media/

Other symptoms:

# mdadm --assemble --scan
mdadm main: failed to get exclusive lock on mapfile
mdadm: /dev/md/0 has been started with 2 drives.
root@turris:/mnt/raid#

Questions:

why isn’t the device listed by df?
why do copy operations involve /dev/mmcblk0p1? if this is normal, why aren’t the files removed when the copy operation completes?
this unix.stackexchange posting suggests that I should save metadata for boot time (`mdadm --examine --brief /dev/sda /dev/sdb >> /etc/mdadm/mdadm.conf``). Is this necessary?
I’ve also encountered a no space on device message, when there is plenty of room on the RAID; I suspect this relates to files being copied to dev/mmcblk0p1. Is this the problem?

** update **

# mount
/dev/mmcblk0p1 on / type btrfs (rw,noatime,ssd,space_cache,commit=5,subvolid=284,subvol=/@)
proc on /proc type proc (rw,noatime)
sysfs on /sys type sysfs (rw,noatime)
none on /sys/fs/cgroup type cgroup (rw,relatime,cpuset,cpu,cpuacct,blkio,memory,devices,freezer,net_cls)
tmpfs on /tmp type tmpfs (rw,nosuid,nodev,noatime)
tmpfs on /dev type tmpfs (rw,relatime,size=512k,mode=755)
devpts on /dev/pts type devpts (rw,relatime,mode=600,ptmxmode=000)
/dev/sdc1 on /mnt/kingston type ext3 (rw,relatime,data=ordered)
debugfs on /sys/kernel/debug type debugfs (rw,noatime)
mountd(pid2905) on /tmp/run/mountd type autofs (rw,relatime,fd=5,pgrp=2903,timeout=60,minproto=5,maxproto=5,indirect)
/dev/sdc1 on /tmp/run/mountd/sdc1 type ext3 (rw,relatime,data=ordered)

Paul_Totterman · July 14, 2017, 4:08pm

Check the status of what is mounted where by using mount instead of df or lsblk. Also check dmesg.

craibuc · July 14, 2017, 4:14pm

What am I looking for in dmesg? Could you be a bit more specific?

Paul_Totterman · July 14, 2017, 5:00pm

You are looking for the messages in dmesg around the time of your actions

craibuc · July 14, 2017, 7:27pm

I’m not seeing anything that looks relevant:

# dmesg
[    9.472841] sd 1:0:0:0: [sdb] Write Protect is off
[    9.477644] sd 1:0:0:0: [sdb] Mode Sense: 00 3a 00 00
[    9.477679] sd 1:0:0:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[    9.528775] sd 1:0:0:0: [sdb] Attached SCSI disk
[    9.533983] ahci-mvebu f10a8000.sata: AHCI 0001.0000 32 slots 2 ports 6 Gbps 0x3 impl platform mode
[    9.543071] ahci-mvebu f10a8000.sata: flags: 64bit ncq sntf led only pmp fbs pio slum part sxs 
[    9.552496] scsi host2: ahci-mvebu
[    9.556062] scsi host3: ahci-mvebu
[    9.559565] ata3: SATA max UDMA/133 mmio [mem 0xf10a8000-0xf10a9fff] port 0x100 irq 44
[    9.567513] ata4: SATA max UDMA/133 mmio [mem 0xf10a8000-0xf10a9fff] port 0x180 irq 44
[    9.920640] ata4: SATA link down (SStatus 0 SControl 300)
[    9.926071] ata3: SATA link down (SStatus 0 SControl 300)
[    9.932493] uhci_hcd: USB Universal Host Controller Interface driver
[    9.939454] orion_wdt: Initial timeout 171 sec
[    9.947124] xhci-hcd f10f0000.usb3: xHCI Host Controller
[    9.952476] xhci-hcd f10f0000.usb3: new USB bus registered, assigned bus number 2
[    9.960067] xhci-hcd f10f0000.usb3: hcc params 0x0a000990 hci version 0x100 quirks 0x00010010
[    9.968638] xhci-hcd f10f0000.usb3: irq 46, io mem 0xf10f0000
[    9.974664] hub 2-0:1.0: USB hub found
[    9.978434] hub 2-0:1.0: 1 port detected
[    9.982470] xhci-hcd f10f0000.usb3: xHCI Host Controller
[    9.987797] xhci-hcd f10f0000.usb3: new USB bus registered, assigned bus number 3
[    9.995359] usb usb3: We don't know the algorithms for LPM for this host, disabling LPM.
[   10.003693] hub 3-0:1.0: USB hub found
[   10.007461] hub 3-0:1.0: 1 port detected
[   10.011557] xhci-hcd f10f8000.usb3: xHCI Host Controller
[   10.016888] xhci-hcd f10f8000.usb3: new USB bus registered, assigned bus number 4
[   10.024485] xhci-hcd f10f8000.usb3: hcc params 0x0a000990 hci version 0x100 quirks 0x00010010
[   10.033059] xhci-hcd f10f8000.usb3: irq 47, io mem 0xf10f8000
[   10.039106] hub 4-0:1.0: USB hub found
[   10.042885] hub 4-0:1.0: 1 port detected
[   10.046910] xhci-hcd f10f8000.usb3: xHCI Host Controller
[   10.052243] xhci-hcd f10f8000.usb3: new USB bus registered, assigned bus number 5
[   10.059771] usb usb5: We don't know the algorithms for LPM for this host, disabling LPM.
[   10.068102] hub 5-0:1.0: USB hub found
[   10.071876] hub 5-0:1.0: 1 port detected
[   10.077121] usbcore: registered new interface driver usb-storage
[   10.195831] init: - preinit -
[   10.300642] usb 2-1: new high-speed USB device number 2 using xhci-hcd
[   10.504145] usb-storage 2-1:1.0: USB Mass Storage device detected
[   10.510434] scsi host4: usb-storage 2-1:1.0
[   10.514994] random: procd: uninitialized urandom read (4 bytes read, 40 bits of entropy available)
[   11.512594] scsi 4:0:0:0: Direct-Access     Kingston DataTraveler 2.0 PMAP PQ: 0 ANSI: 4
[   11.521506] sd 4:0:0:0: [sdc] 60978816 512-byte logical blocks: (31.2 GB/29.1 GiB)
[   11.529244] sd 4:0:0:0: [sdc] Write Protect is off
[   11.534071] sd 4:0:0:0: [sdc] Mode Sense: 23 00 00 00
[   11.534206] sd 4:0:0:0: [sdc] No Caching mode page found
[   11.539531] sd 4:0:0:0: [sdc] Assuming drive cache: write through
[   11.547926]  sdc: sdc1
[   11.551385] sd 4:0:0:0: [sdc] Attached SCSI removable disk
[   13.246059] mount_root: mounting /dev/root
[   13.251112] BTRFS info (device mmcblk0p1): disk space caching is enabled
[   13.258296] mount_root: loading kmods from internal overlay
[   13.480042] block: attempting to load /etc/config/fstab
[   13.486243] block: extroot: not configured
[   13.492859] procd: - early -
[   13.607782] random: jshn: uninitialized urandom read (4 bytes read, 79 bits of entropy available)
[   13.635269] random: jshn: uninitialized urandom read (4 bytes read, 79 bits of entropy available)
[   13.670982] random: jshn: uninitialized urandom read (4 bytes read, 79 bits of entropy available)
[   13.695413] random: jshn: uninitialized urandom read (4 bytes read, 79 bits of entropy available)
[   13.730849] random: jshn: uninitialized urandom read (4 bytes read, 79 bits of entropy available)
[   13.755370] random: jshn: uninitialized urandom read (4 bytes read, 79 bits of entropy available)
[   13.791035] random: jshn: uninitialized urandom read (4 bytes read, 79 bits of entropy available)
[   13.815712] random: jshn: uninitialized urandom read (4 bytes read, 79 bits of entropy available)
[   13.851458] random: jshn: uninitialized urandom read (4 bytes read, 79 bits of entropy available)
[   14.073737] procd: - watchdog -
[   14.077149] procd: - ubus -
[   14.230902] EXT4-fs (sdc1): mounting ext3 file system using the ext4 subsystem
[   14.252811] EXT4-fs (sdc1): mounted filesystem with ordered data mode. Opts: 
[   15.082780] procd: - init -
[   15.150419] marvell-cesa f1090000.crypto: CESA device successfully registered
[   15.163163] NET: Registered protocol family 38
[   15.179289] md: linear personality registered for level -1
[   15.185414] md: raid0 personality registered for level 0
[   15.191545] md: raid1 personality registered for level 1
[   15.197733] md: raid10 personality registered for level 10
[   15.203647] async_tx: api initialized (async)
[   15.211045] md: raid6 personality registered for level 6
[   15.216373] md: raid5 personality registered for level 5
[   15.221709] md: raid4 personality registered for level 4
[   15.228921] FS-Cache: Loaded
[   15.232306] md: multipath personality registered for level -4
[   15.239269] NET: Registered protocol family 8
[   15.243651] NET: Registered protocol family 20
[   15.251903] device-mapper: ioctl: 4.34.0-ioctl (2015-10-28) initialised: dm-devel@redhat.com
[   15.262873] Key type dns_resolver registered
[   15.270007] NET: Registered protocol family 33
[   15.274479] Key type rxrpc registered
[   15.278148] Key type rxrpc_s registered
[   15.285107] kAFS: Red Hat AFS client v0.1 registering.
[   15.290297] FS-Cache: Netfs 'afs' registered for caching
[   15.324359] RPC: Registered named UNIX socket transport module.
[   15.330297] RPC: Registered udp transport module.
[   15.335027] RPC: Registered tcp transport module.
[   15.339740] RPC: Registered tcp NFSv4.1 backchannel transport module.
[   15.350295] ntfs: driver 2.1.32 [Flags: R/O MODULE].
[   15.360171] RxRPC: Registered security type 2 'rxkad'
[   15.376694] tun: Universal TUN/TAP device driver, 1.6
[   15.381781] tun: (C) 1999-2004 Max Krasnyansky <maxk@qualcomm.com>
[   15.392216] sit: IPv6 over IPv4 tunneling driver
[   15.399051] gre: GRE over IPv4 demultiplexor driver
[   15.404668] ip_gre: GRE over IPv4 tunneling driver
[   15.423742] Installing knfsd (copyright (C) 1996 okir@monad.swb.de).
[   15.432211] PPP generic driver version 2.4.2
[   15.439227] ip6_tables: (C) 2000-2006 Netfilter Core Team
[   15.448749] cryptodev: driver 1.8 loaded.
[   15.457580] Loading modules backported from Linux version wt-2016-10-03-1-g6fcb1a6
[   15.465195] Backport generated by backports.git backports-20160324-9-g0e38f5c
[   15.544315] pci 0000:00:02.0: enabling device (0140 -> 0142)
[   15.550184] ath10k_pci 0000:02:00.0: pci irq msi oper_irq_mode 2 irq_mode 0 reset_mode 0
[   15.724122] ath10k_pci 0000:02:00.0: Direct firmware load for ath10k/pre-cal-pci-0000:02:00.0.bin failed with error -2
[   15.734867] ath10k_pci 0000:02:00.0: Falling back to user helper
[   15.744559] firmware ath10k!pre-cal-pci-0000:02:00.0.bin: firmware_loading_store: map pages failed
[   15.753698] ath10k_pci 0000:02:00.0: Direct firmware load for ath10k/cal-pci-0000:02:00.0.bin failed with error -2
[   15.764100] ath10k_pci 0000:02:00.0: Falling back to user helper
[   15.773639] firmware ath10k!cal-pci-0000:02:00.0.bin: firmware_loading_store: map pages failed
[   15.790715] ath10k_pci 0000:02:00.0: qca988x hw2.0 target 0x4100016c chip_id 0x043202ff sub 0000:0000
[   15.799962] ath10k_pci 0000:02:00.0: kconfig debug 0 debugfs 1 tracing 0 dfs 1 testmode 1
[   15.809545] ath10k_pci 0000:02:00.0: firmware ver 10.2.4.70.54 api 5 features no-p2p,raw-mode,mfp crc32 9d340dd9
[   15.859888] ath10k_pci 0000:02:00.0: Direct firmware load for ath10k/QCA988X/hw2.0/board-2.bin failed with error -2
[   15.870364] ath10k_pci 0000:02:00.0: Falling back to user helper
[   15.880002] firmware ath10k!QCA988X!hw2.0!board-2.bin: firmware_loading_store: map pages failed
[   15.888950] ath10k_pci 0000:02:00.0: board_file api 1 bmi_id N/A crc32 bebc7c08
[   17.031715] ath10k_pci 0000:02:00.0: htt-ver 2.1 wmi-op 5 htt-op 2 cal otp max-sta 128 raw 0 hwcrypto 1
[   17.121847] ath: EEPROM regdomain: 0x0
[   17.121853] ath: EEPROM indicates default country code should be used
[   17.121856] ath: doing EEPROM country->regdmn map search
[   17.121860] ath: country maps to regdmn code: 0x3a
[   17.121863] ath: Country alpha2 being used: US
[   17.121865] ath: Regpair used: 0x3a
[   17.127344] u32 classifier
[   17.130061]     input device check on
[   17.133747]     Actions configured
[   17.140121] Mirror/redirect action on
[   17.145593] GACT probability on
[   17.153597] netem: version 1.3
[   17.161921] fuse init (API version 7.23)
[   17.177818] usbcore: registered new interface driver cdc_wdm
[   17.186616] Ebtables v2.0 registered
[   17.191559] ip_tables: (C) 2000-2006 Netfilter Core Team
[   17.203526] nf_conntrack version 0.5.0 (16384 buckets, 65536 max)
[   17.354585] Netfilter messages via NETLINK v0.30.
[   17.360278] PPP MPPE Compression module registered
[   17.365705] NET: Registered protocol family 24
[   17.373973] usbcore: registered new interface driver ums-alauda
[   17.380885] usbcore: registered new interface driver ums-cypress
[   17.387898] usbcore: registered new interface driver ums-datafab
[   17.394729] usbcore: registered new interface driver ums-freecom
[   17.401651] usbcore: registered new interface driver ums-isd200
[   17.408504] usbcore: registered new interface driver ums-jumpshot
[   17.415436] usbcore: registered new interface driver ums-karma
[   17.422474] usbcore: registered new interface driver ums-sddr09
[   17.429277] usbcore: registered new interface driver ums-sddr55
[   17.436266] usbcore: registered new interface driver ums-usbat
[   17.448083] usbcore: registered new interface driver usbserial
[   17.453992] usbcore: registered new interface driver usbserial_generic
[   17.460561] usbserial: USB Serial support registered for generic
[   17.475588] xt_time: kernel timezone is -0000
[   17.493837] usbcore: registered new interface driver cdc_ether
[   17.504257] ctnetlink v0.93: registering with nfnetlink.
[   17.513352] usbcore: registered new interface driver qmi_wwan
[   17.520255] usbcore: registered new interface driver rndis_host
[   17.533628] pci 0000:00:01.0: enabling device (0140 -> 0142)
[   17.626000] ath: EEPROM regdomain: 0x0
[   17.626004] ath: EEPROM indicates default country code should be used
[   17.626007] ath: doing EEPROM country->regdmn map search
[   17.626010] ath: country maps to regdmn code: 0x3a
[   17.626013] ath: Country alpha2 being used: US
[   17.626015] ath: Regpair used: 0x3a
[   17.637838] ieee80211 phy1: Selected rate control algorithm 'minstrel_ht'
[   17.639263] ieee80211 phy1: Atheros AR9287 Rev:2 mem=0xf1d40000, irq=108
[   17.647717] usbcore: registered new interface driver option
[   17.653387] usbserial: USB Serial support registered for GSM modem (1-port)
[   17.661155] usbcore: registered new interface driver qcserial
[   17.666948] usbserial: USB Serial support registered for Qualcomm USB modem
[   19.760643] random: nonblocking pool is initialized
[   25.952551] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
[   26.032340] IPv6: ADDRCONF(NETDEV_UP): eth1: link is not ready
[   26.035829] IPv6: ADDRCONF(NETDEV_UP): eth2: link is not ready
[   27.776024] device eth0 entered promiscuous mode
[   27.777487] IPv6: ADDRCONF(NETDEV_UP): br-lan: link is not ready
[   27.792674] device eth2 entered promiscuous mode
[   27.872492] IPv6: ADDRCONF(NETDEV_UP): eth1: link is not ready
[   29.613336] IPv6: ADDRCONF(NETDEV_UP): wlan0: link is not ready
[   29.630883] IPv6: ADDRCONF(NETDEV_UP): wlan1: link is not ready
[   29.631911] device wlan0 entered promiscuous mode
[   29.696451] IPv6: ADDRCONF(NETDEV_CHANGE): wlan1: link becomes ready
[   29.770663] mvneta f1030000.ethernet eth0: Link is Up - 1Gbps/Full - flow control off
[   29.770691] br-lan: port 1(eth0) entered forwarding state
[   29.770715] br-lan: port 1(eth0) entered forwarding state
[   29.770757] IPv6: ADDRCONF(NETDEV_CHANGE): br-lan: link becomes ready
[   29.790707] mvneta f1070000.ethernet eth2: Link is Up - 1Gbps/Full - flow control off
[   29.790726] br-lan: port 2(eth2) entered forwarding state
[   29.790751] br-lan: port 2(eth2) entered forwarding state
[   29.987476] IPv6: ADDRCONF(NETDEV_CHANGE): wlan0: link becomes ready
[   29.987531] br-lan: port 3(wlan0) entered forwarding state
[   29.987553] br-lan: port 3(wlan0) entered forwarding state
[   31.770640] br-lan: port 1(eth0) entered forwarding state
[   31.790663] br-lan: port 2(eth2) entered forwarding state
[   31.871059] mvneta f1034000.ethernet eth1: Link is Up - 1Gbps/Full - flow control off
[   31.871077] IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
[   31.980630] br-lan: port 3(wlan0) entered forwarding state

Marek_Ruzicka · July 14, 2017, 10:33pm

It looks like you don’t have the raid device mounted anywhere.

It explaines why you cannot see it in df output (1), as well as why the writes are going to the /dev/mmcblk0p1 device (since I assume that is where your intended mount point is residing) (2), and of course why you don’t see it in mount output.

You need to make that mount /dev/md0 /mnt/raid persistent over reboots.

Either go to Luci and under System -> Mount Points do a few clicks, or
add something like this into your /etc/config/fstab

config mount
        option  target    '/mnt/raid'
        option  enabled   '1'
        option  fstype    'btrfs'
        option  uuid      '<UUID>'
        option  device    '/dev/md0'

where UUID could be obtained from btrfs fi show, but generally could be ommited altogether (comment out that line).

craibuc · July 15, 2017, 1:10pm

Thank you for responding.

It looks like you don’t have the raid device mounted anywhere.

That makes sense. I removed files and directories from /mnt/raid and the space used on /dev/mmcblk0p1 decreased correspondingly.

Steps to fix the problem:

configured fstab:

# cat /etc/config/fstab

config global
	option anon_swap '0'
	option anon_mount '0'
	option auto_swap '1'
	option auto_mount '1'
	option delay_root '5'
	option check_fs '0'

config mount
	option enabled '1'
	option uuid '5e7f5135-c923-4ec5-b6b4-17e76908b4c0'
	option target '/mnt/kingston'

config mount
        option  target    '/mnt/raid'
        option  enabled   '1'
        option  fstype    'btrfs'
        option  device    '/dev/md0'

rebooted:

# reboot

check disks and blocks

# df -h
Filesystem                Size      Used Available Use% Mounted on
/dev/mmcblk0p1            7.3G      4.4G      3.0G  60% /
tmpfs                  1011.1M    400.0K   1010.7M   0% /tmp
tmpfs                   512.0K      4.0K    508.0K   1% /dev

# lsblk
NAME         MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sda            8:0    0  3.7T  0 disk 
sdb            8:16   0  3.7T  0 disk 
mtdblock0     31:0    0    1M  0 disk 
mtdblock1     31:1    0    7M  0 disk 
mmcblk0      179:0    0  7.3G  0 disk 
`-mmcblk0p1  179:1    0  7.3G  0 part /
mmcblk0boot0 179:8    0    4M  1 disk 
mmcblk0boot1 179:16   0    4M  1 disk 
mmcblk0rpmb  179:24   0    4M  0 disk

check the state of the device:

# mdadm --examine --brief /dev/sda /dev/sdb
ARRAY /dev/md/0 metadata=1.2 UUID=91da414e:c83be60a:c33b9493:07123c98 name=turris:0

only internal RAM visible:

# btrfs fi show
Label: none  uuid: 1ecd23ea-3018-4a83-9072-d337b0d00c5c
	Total devices 1 FS bytes used 4.31GiB
	devid    1 size 7.28GiB used 7.28GiB path /dev/mmcblk0p1

bring up the RAID device:

# mdadm --assemble --scan
mdadm main: failed to get exclusive lock on mapfile
mdadm: /dev/md/0 has been started with 2 drives.

mount device:

# /dev/md0 /mnt/raid
-ash: /dev/md0: Permission denied

edit

# cat /proc/mdstat
Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4] [multipath] 
md0 : active raid1 sda[0] sdb[1]
      3906887360 blocks super 1.2 [2/2] [UU]
      [>....................]  resync =  1.1% (46450944/3906887360) finish=324.8min speed=198041K/sec
      
unused devices: <none>

/ edit

edit 2

Does this mean that one of the drives has failed? It was new.

[  608.244476] md/raid1:md0: Disk failure on sdb, disabling device.

full text:

# dmesg
....
[  217.535134] md: md0 stopped.
[  217.536088] md: bind<sdb>
[  217.536287] md: bind<sda>
[  217.536621] md/raid1:md0: not clean -- starting background reconstruction
[  217.536628] md/raid1:md0: active with 2 out of 2 mirrors
[  217.536685] md0: detected capacity change from 0 to 4000652656640
[  217.537211] md: resync of RAID array md0
[  217.537219] md: minimum _guaranteed_  speed: 1000 KB/sec/disk.
[  217.537224] md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for resync.
[  217.537237] md: using 128k window, over a total of 3906887360k.
[  217.786133] BTRFS: device fsid 6282e17f-2797-49dc-b4f9-e9452c35b922 devid 1 transid 14 /dev/md0
[  217.845325] BTRFS info (device md0): disk space caching is enabled
[  217.845332] BTRFS: has skinny extents
[  547.978710] ata1.00: exception Emask 0x0 SAct 0xfc000 SErr 0x0 action 0x6 frozen
[  547.986154] ata1.00: cmd 60/80:70:80:a2:d3/00:00:06:00:00/40 tag 14 ncq 65536 in
[  547.986154]          res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
[  548.001004] ata1.00: cmd 60/80:78:00:a3:d3/00:00:06:00:00/40 tag 15 ncq 65536 in
[  548.001004]          res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
[  548.015906] ata1.00: cmd 60/80:80:80:a3:d3/00:00:06:00:00/40 tag 16 ncq 65536 in
[  548.015906]          res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
[  548.030774] ata1.00: cmd 60/80:88:00:a4:d3/00:00:06:00:00/40 tag 17 ncq 65536 in
[  548.030774]          res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
[  548.045619] ata1.00: cmd 60/80:90:80:a4:d3/00:00:06:00:00/40 tag 18 ncq 65536 in
[  548.045619]          res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
[  548.060494] ata1.00: cmd 60/00:98:00:a5:d3/01:00:06:00:00/40 tag 19 ncq 131072 in
[  548.060494]          res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
[  548.075499] ata1: hard resetting link
[  548.075531] ata2.00: exception Emask 0x0 SAct 0x7e0 SErr 0x0 action 0x6 frozen
[  548.082801] ata2.00: cmd 61/80:28:80:96:d3/00:00:06:00:00/40 tag 5 ncq 65536 out
[  548.082801]          res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
[  548.097710] ata2.00: cmd 61/80:30:00:97:d3/00:00:06:00:00/40 tag 6 ncq 65536 out
[  548.097710]          res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
[  548.112552] ata2.00: cmd 61/00:38:80:97:d3/01:00:06:00:00/40 tag 7 ncq 131072 out
[  548.112552]          res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
[  548.127475] ata2.00: cmd 61/00:40:80:98:d3/01:00:06:00:00/40 tag 8 ncq 131072 out
[  548.127475]          res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
[  548.142429] ata2.00: cmd 61/00:48:80:99:d3/05:00:06:00:00/40 tag 9 ncq 655360 out
[  548.142429]          res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
[  548.157409] ata2.00: cmd 61/00:50:80:9e:d3/04:00:06:00:00/40 tag 10 ncq 524288 out
[  548.157409]          res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
[  548.172442] ata2: hard resetting link
[  558.078352] ata1: softreset failed (1st FIS failed)
[  558.083282] ata1: hard resetting link
[  558.178343] ata2: softreset failed (1st FIS failed)
[  558.183281] ata2: hard resetting link
[  568.088037] ata1: softreset failed (1st FIS failed)
[  568.092932] ata1: hard resetting link
[  568.188065] ata2: softreset failed (1st FIS failed)
[  568.192973] ata2: hard resetting link
[  603.086962] ata1: softreset failed (1st FIS failed)
[  603.091859] ata1: limiting SATA link speed to 3.0 Gbps
[  603.091865] ata1: hard resetting link
[  603.186957] ata2: softreset failed (1st FIS failed)
[  603.191849] ata2: limiting SATA link speed to 3.0 Gbps
[  603.191853] ata2: hard resetting link
[  608.196818] ata2: softreset failed (1st FIS failed)
[  608.201731] ata2: reset failed, giving up
[  608.205752] ata2.00: disabled
[  608.205759] ata2.00: device reported invalid CHS sector 0
[  608.205764] ata2.00: device reported invalid CHS sector 0
[  608.205767] ata2.00: device reported invalid CHS sector 0
[  608.205771] ata2.00: device reported invalid CHS sector 0
[  608.205774] ata2.00: device reported invalid CHS sector 0
[  608.205778] ata2.00: device reported invalid CHS sector 0
[  608.205797] ata2: EH complete
[  608.205834] sd 1:0:0:0: [sdb] tag#11 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=0x00
[  608.205837] sd 1:0:0:0: [sdb] tag#13 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=0x00
[  608.205842] sd 1:0:0:0: [sdb] tag#13 CDB: opcode=0x8a 8a 00 00 00 00 00 06 d3 98 80 00 00 01 00 00 00
[  608.205846] blk_update_request: I/O error, dev sdb, sector 114530432
[  608.205873] sd 1:0:0:0: [sdb] tag#14 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=0x00
[  608.205878] sd 1:0:0:0: [sdb] tag#14 CDB: opcode=0x8a 8a 00 00 00 00 00 06 d3 97 80 00 00 01 00 00 00
[  608.205880] blk_update_request: I/O error, dev sdb, sector 114530176
[  608.205890] sd 1:0:0:0: [sdb] tag#15 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=0x00
[  608.205894] sd 1:0:0:0: [sdb] tag#15 CDB: opcode=0x8a 8a 00 00 00 00 00 06 d3 97 00 00 00 00 80 00 00
[  608.205896] blk_update_request: I/O error, dev sdb, sector 114530048
[  608.205904] sd 1:0:0:0: [sdb] tag#16 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=0x00
[  608.205908] sd 1:0:0:0: [sdb] tag#16 CDB: opcode=0x8a 8a 00 00 00 00 00 06 d3 96 80 00 00 00 80 00 00
[  608.205910] blk_update_request: I/O error, dev sdb, sector 114529920
[  608.231335] sd 1:0:0:0: [sdb] tag#11 CDB: opcode=0x8a 8a 00 00 00 00 00 06 d3 9e 80 00 00 04 00 00 00
[  608.231340] blk_update_request: I/O error, dev sdb, sector 114531968
[  608.237739] sd 1:0:0:0: [sdb] tag#12 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=0x00
[  608.237745] sd 1:0:0:0: [sdb] tag#12 CDB: opcode=0x8a 8a 00 00 00 00 00 06 d3 99 80 00 00 05 00 00 00
[  608.237749] blk_update_request: I/O error, dev sdb, sector 114530688
[  608.244476] md/raid1:md0: Disk failure on sdb, disabling device.
[  608.244476] md/raid1:md0: Operation continuing on 1 devices.
[  608.256273] md: md0: resync interrupted.
[  608.296809] ata1: SATA link up 3.0 Gbps (SStatus 123 SControl 320)
[  613.296664] ata1.00: qc timeout (cmd 0xec)
[  613.296679] ata1.00: failed to IDENTIFY (I/O error, err_mask=0x4)
[  613.296687] ata1.00: revalidation failed (errno=-5)
[  613.301581] ata1: hard resetting link
[  623.306358] ata1: softreset failed (1st FIS failed)
[  623.311257] ata1: hard resetting link
[  633.316065] ata1: softreset failed (1st FIS failed)
[  633.320961] ata1: hard resetting link
[  668.313156] ata1: softreset failed (1st FIS failed)
[  668.318050] ata1: limiting SATA link speed to 1.5 Gbps
[  668.318055] ata1: hard resetting link
[  673.522771] ata1: SATA link up 1.5 Gbps (SStatus 113 SControl 310)
[  683.521994] ata1.00: qc timeout (cmd 0xec)
[  683.522009] ata1.00: failed to IDENTIFY (I/O error, err_mask=0x4)
[  683.522014] ata1.00: revalidation failed (errno=-5)
[  683.526908] ata1: hard resetting link
[  693.531252] ata1: softreset failed (1st FIS failed)
[  693.536151] ata1: hard resetting link
[  703.540541] ata1: softreset failed (1st FIS failed)
[  703.545436] ata1: hard resetting link
[  738.538317] ata1: softreset failed (1st FIS failed)
[  738.543214] ata1: hard resetting link
[  743.747910] ata1: SATA link up 1.5 Gbps (SStatus 113 SControl 310)
[  773.745950] ata1.00: qc timeout (cmd 0xec)
[  773.745967] ata1.00: failed to IDENTIFY (I/O error, err_mask=0x4)
[  773.745972] ata1.00: revalidation failed (errno=-5)
[  773.750863] ata1.00: disabled
[  773.750875] ata1.00: device reported invalid CHS sector 0
[  773.750880] ata1.00: device reported invalid CHS sector 0
[  773.750884] ata1.00: device reported invalid CHS sector 0
[  773.750887] ata1.00: device reported invalid CHS sector 0
[  773.750890] ata1.00: device reported invalid CHS sector 0
[  773.750894] ata1.00: device reported invalid CHS sector 0
[  773.750906] ata1: hard resetting link
[  783.755188] ata1: softreset failed (1st FIS failed)
[  783.760117] ata1: hard resetting link
[  793.764459] ata1: softreset failed (1st FIS failed)
[  793.769357] ata1: hard resetting link
[  828.762068] ata1: softreset failed (1st FIS failed)
[  828.766980] ata1: hard resetting link
[  833.971779] ata1: SATA link up 1.5 Gbps (SStatus 113 SControl 310)
[  833.971810] ata1: EH complete
[  833.971845] sd 0:0:0:0: [sda] tag#28 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=0x00
[  833.971848] sd 0:0:0:0: [sda] tag#22 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=0x00
[  833.971853] sd 0:0:0:0: [sda] tag#22 CDB: opcode=0x88 88 00 00 00 00 00 06 d3 a5 00 00 00 01 00 00 00
[  833.971858] sd 0:0:0:0: [sda] tag#28 CDB: opcode=0x35 35 00 00 00 00 00 00 00 00 00
[  833.971863] blk_update_request: I/O error, dev sda, sector 114533632
[  833.971865] blk_update_request: I/O error, dev sda, sector 8
[  833.971869] md: super_written gets error=-5
[  833.983931] sd 0:0:0:0: [sda] tag#23 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=0x00
[  833.983938] sd 0:0:0:0: [sda] tag#23 CDB: opcode=0x88 88 00 00 00 00 00 06 d3 a4 80 00 00 00 80 00 00
[  833.983942] blk_update_request: I/O error, dev sda, sector 114533504
[  833.990642] sd 0:0:0:0: [sda] tag#24 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=0x00
[  833.990651] sd 0:0:0:0: [sda] tag#24 CDB: opcode=0x88 88 00 00 00 00 00 06 d3 a4 00 00 00 00 80 00 00
[  833.990655] blk_update_request: I/O error, dev sda, sector 114533376
[  833.997048] sd 0:0:0:0: [sda] tag#25 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=0x00
[  833.997055] sd 0:0:0:0: [sda] tag#25 CDB: opcode=0x88 88 00 00 00 00 00 06 d3 a3 80 00 00 00 80 00 00
[  833.997059] blk_update_request: I/O error, dev sda, sector 114533248
[  834.003444] sd 0:0:0:0: [sda] tag#26 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=0x00
[  834.003450] sd 0:0:0:0: [sda] tag#26 CDB: opcode=0x88 88 00 00 00 00 00 06 d3 a3 00 00 00 00 80 00 00
[  834.003459] blk_update_request: I/O error, dev sda, sector 114533120
[  834.009836] sd 0:0:0:0: [sda] tag#27 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=0x00
[  834.009842] sd 0:0:0:0: [sda] tag#27 CDB: opcode=0x88 88 00 00 00 00 00 06 d3 a2 80 00 00 00 80 00 00
[  834.009846] blk_update_request: I/O error, dev sda, sector 114532992
[  834.016230] sd 0:0:0:0: [sda] tag#29 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=0x00
[  834.016236] sd 0:0:0:0: [sda] tag#29 CDB: opcode=0x88 88 00 00 00 00 01 d1 c0 bd 00 00 00 00 08 00 00
[  834.016240] blk_update_request: I/O error, dev sda, sector 7814036736
[  834.022712] sd 0:0:0:0: [sda] tag#30 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=0x00
[  834.022718] sd 0:0:0:0: [sda] tag#30 CDB: opcode=0x88 88 00 00 00 00 00 06 d3 a5 00 00 00 00 08 00 00
[  834.022722] blk_update_request: I/O error, dev sda, sector 114533632
[  834.029236] sd 0:0:0:0: [sda] tag#0 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=0x00
[  834.029243] sd 0:0:0:0: [sda] tag#0 CDB: opcode=0x88 88 00 00 00 00 01 d1 c0 bd 70 00 00 00 08 00 00
[  834.029248] blk_update_request: I/O error, dev sda, sector 7814036848
[  834.035785] md/raid1:md0: sda: unrecoverable I/O read error for block 114271488
[  834.043216] md/raid1:md0: sda: unrecoverable I/O read error for block 114271616
[  834.050624] md/raid1:md0: sda: unrecoverable I/O read error for block 114271360
[  834.058065] md/raid1:md0: sda: unrecoverable I/O read error for block 114271232
[  834.065488] md/raid1:md0: sda: unrecoverable I/O read error for block 114271104
[  834.072917] md/raid1:md0: sda: unrecoverable I/O read error for block 114270976
[  834.080320] md/raid1:md0: sda: unrecoverable I/O read error for block 114270848
[  834.087977] md: checkpointing resync of md0.
[  834.088039] md: super_written gets error=-5
[  834.088104] md: super_written gets error=-5
[  834.088130] RAID1 conf printout:
[  834.088133]  --- wd:1 rd:2
[  834.088137]  disk 0, wo:0, o:1, dev:sda
[  834.088140]  disk 1, wo:1, o:0, dev:sdb
[  834.121736] RAID1 conf printout:
[  834.121742]  --- wd:1 rd:2
[  834.121746]  disk 0, wo:0, o:1, dev:sda
[  834.121796] md: super_written gets error=-5
[  834.121888] md: super_written gets error=-5
[  834.121941] md: super_written gets error=-5
[  834.121998] md: resync of RAID array md0
[  834.122002] md: minimum _guaranteed_  speed: 1000 KB/sec/disk.
[  834.122005] md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for resync.
[  834.122010] md: using 128k window, over a total of 3906887360k.
[  834.122206] md: md0: resync done.
[  834.122439] md: super_written gets error=-5
[  834.122492] md: super_written gets error=-5

/ edit 2

edit 3

both mass-storage devices visible:

# btrfs fi show
Label: none  uuid: 1ecd23ea-3018-4a83-9072-d337b0d00c5c
	Total devices 1 FS bytes used 1.64GiB
	devid    1 size 7.28GiB used 7.28GiB path /dev/mmcblk0p1

Label: none  uuid: 6282e17f-2797-49dc-b4f9-e9452c35b922
	Total devices 1 FS bytes used 1.29GiB
	devid    1 size 3.64TiB used 4.02GiB path /dev/md0

in a separate window, watching dmesg while btrfs fi show operations running:

# dmesg
...
[ 1421.346748] md: md0 stopped.
[ 1421.348325] md: bind<sdb>
[ 1421.348538] md: bind<sda>
[ 1421.348825] md/raid1:md0: not clean -- starting background reconstruction
[ 1421.348832] md/raid1:md0: active with 2 out of 2 mirrors
[ 1421.348882] md0: detected capacity change from 0 to 4000652656640
[ 1421.349202] md: resync of RAID array md0
[ 1421.349209] md: minimum _guaranteed_  speed: 1000 KB/sec/disk.
[ 1421.349212] md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for resync.
[ 1421.349217] md: using 128k window, over a total of 3906887360k.
[ 1421.593735] BTRFS: device fsid 6282e17f-2797-49dc-b4f9-e9452c35b922 devid 1 transid 14 /dev/md0
[ 1421.652649] BTRFS info (device md0): disk space caching is enabled
[ 1421.652657] BTRFS: has skinny extents
[ 1458.949700] ata2.00: exception Emask 0x0 SAct 0x40000003 SErr 0x0 action 0x6 frozen
[ 1458.957386] ata2.00: cmd 61/80:00:00:91:25/07:00:00:00:00/40 tag 0 ncq 983040 out
[ 1458.957386]          res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
[ 1458.972317] ata2.00: cmd 61/00:08:80:98:25/05:00:00:00:00/40 tag 1 ncq 655360 out
[ 1458.972317]          res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
[ 1458.987252] ata2.00: cmd 61/00:f0:00:90:25/01:00:00:00:00/40 tag 30 ncq 131072 out
[ 1458.987252]          res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
[ 1459.002255] ata2: hard resetting link
[ 1461.879489] ata1.00: exception Emask 0x0 SAct 0xc000000 SErr 0x0 action 0x6 frozen
[ 1461.887093] ata1.00: cmd 60/00:d0:80:9d:25/02:00:00:00:00/40 tag 26 ncq 262144 in
[ 1461.887093]          res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
[ 1461.902038] ata1.00: cmd 60/08:d8:00:be:c0/00:00:d1:01:00/40 tag 27 ncq 4096 in
[ 1461.902038]          res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
[ 1461.916877] ata1: hard resetting link
[ 1469.008873] ata2: softreset failed (1st FIS failed)
[ 1469.013768] ata2: hard resetting link
[ 1471.918629] ata1: softreset failed (1st FIS failed)
[ 1471.923526] ata1: hard resetting link
[ 1479.018042] ata2: softreset failed (1st FIS failed)
[ 1479.022940] ata2: hard resetting link
[ 1481.927805] ata1: softreset failed (1st FIS failed)
[ 1481.932717] ata1: hard resetting link
[ 1514.025307] ata2: softreset failed (1st FIS failed)
[ 1514.030202] ata2: limiting SATA link speed to 1.5 Gbps
[ 1514.030208] ata2: hard resetting link
[ 1516.925087] ata1: softreset failed (1st FIS failed)
[ 1516.929982] ata1: limiting SATA link speed to 1.5 Gbps
[ 1516.929987] ata1: hard resetting link
[ 1519.034930] ata2: softreset failed (1st FIS failed)
[ 1519.039832] ata2: reset failed, giving up
[ 1519.043851] ata2.00: disabled
[ 1519.043857] ata2.00: device reported invalid CHS sector 0
[ 1519.043862] ata2.00: device reported invalid CHS sector 0
[ 1519.043867] ata2.00: device reported invalid CHS sector 0
[ 1519.043881] ata2: EH complete
[ 1519.043913] sd 1:0:0:0: [sdb] tag#2 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=0x00
[ 1519.043923] sd 1:0:0:0: [sdb] tag#2 CDB: opcode=0x8a 8a 00 00 00 00 00 00 25 90 00 00 00 01 00 00 00
[ 1519.043927] sd 1:0:0:0: [sdb] tag#3 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=0x00
[ 1519.043932] blk_update_request: I/O error, dev sdb, sector 2461696
[ 1519.043933] sd 1:0:0:0: [sdb] tag#3 CDB: opcode=0x8a 8a 00 00 00 00 00 00 25 98 80 00 00 05 00 00 00
[ 1519.043937] blk_update_request: I/O error, dev sdb, sector 2463872
[ 1519.043982] sd 1:0:0:0: [sdb] tag#4 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=0x00
[ 1519.043987] sd 1:0:0:0: [sdb] tag#4 CDB: opcode=0x8a 8a 00 00 00 00 00 00 25 91 00 00 00 07 80 00 00
[ 1519.043989] blk_update_request: I/O error, dev sdb, sector 2461952
[ 1519.044067] md/raid1:md0: Disk failure on sdb, disabling device.
[ 1519.044067] md/raid1:md0: Operation continuing on 1 devices.
[ 1519.044130] md: md0: resync interrupted.
[ 1522.134703] ata1: SATA link up 1.5 Gbps (SStatus 113 SControl 310)
[ 1527.134192] ata1.00: qc timeout (cmd 0xec)
[ 1527.134207] ata1.00: failed to IDENTIFY (I/O error, err_mask=0x4)
[ 1527.134212] ata1.00: revalidation failed (errno=-5)
[ 1527.139104] ata1: hard resetting link
[ 1537.142861] ata1: softreset failed (1st FIS failed)
[ 1537.147756] ata1: hard resetting link
[ 1547.141609] ata1: softreset failed (1st FIS failed)
[ 1547.146504] ata1: hard resetting link
[ 1582.147707] ata1: softreset failed (1st FIS failed)
[ 1582.152605] ata1: hard resetting link
[ 1587.357190] ata1: SATA link up 1.5 Gbps (SStatus 113 SControl 310)
[ 1597.356112] ata1.00: qc timeout (cmd 0xec)
[ 1597.356127] ata1.00: failed to IDENTIFY (I/O error, err_mask=0x4)
[ 1597.356132] ata1.00: revalidation failed (errno=-5)
[ 1597.361027] ata1: hard resetting link
[ 1607.364978] ata1: softreset failed (1st FIS failed)
[ 1607.369873] ata1: hard resetting link
[ 1617.373904] ata1: softreset failed (1st FIS failed)
[ 1617.378799] ata1: hard resetting link
[ 1652.370519] ata1: softreset failed (1st FIS failed)
[ 1652.375415] ata1: hard resetting link
[ 1657.580053] ata1: SATA link up 1.5 Gbps (SStatus 113 SControl 310)
[ 1687.577076] ata1.00: qc timeout (cmd 0xec)
[ 1687.577091] ata1.00: failed to IDENTIFY (I/O error, err_mask=0x4)
[ 1687.577096] ata1.00: revalidation failed (errno=-5)
[ 1687.582019] ata1.00: disabled
[ 1687.582032] ata1.00: device reported invalid CHS sector 0
[ 1687.582037] ata1.00: device reported invalid CHS sector 0
[ 1687.582049] ata1: hard resetting link
[ 1697.586139] ata1: softreset failed (1st FIS failed)
[ 1697.591034] ata1: hard resetting link
[ 1707.595257] ata1: softreset failed (1st FIS failed)
[ 1707.600178] ata1: hard resetting link
[ 1742.592142] ata1: softreset failed (1st FIS failed)
[ 1742.597039] ata1: hard resetting link
[ 1747.801673] ata1: SATA link up 1.5 Gbps (SStatus 113 SControl 310)
[ 1747.801703] ata1: EH complete
[ 1747.801739] sd 0:0:0:0: [sda] tag#29 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=0x00
[ 1747.801742] sd 0:0:0:0: [sda] tag#30 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=0x00
[ 1747.801748] sd 0:0:0:0: [sda] tag#30 CDB: opcode=0x88 88 00 00 00 00 00 00 25 9d 80 00 00 02 00 00 00
[ 1747.801752] sd 0:0:0:0: [sda] tag#29 CDB: opcode=0x88 88 00 00 00 00 01 d1 c0 be 00 00 00 00 08 00 00
[ 1747.801756] blk_update_request: I/O error, dev sda, sector 7814036992
[ 1747.801778] sd 0:0:0:0: [sda] tag#0 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=0x00
[ 1747.801782] sd 0:0:0:0: [sda] tag#0 CDB: opcode=0x35 35 00 00 00 00 00 00 00 00 00
[ 1747.801787] blk_update_request: I/O error, dev sda, sector 8
[ 1747.801789] md: super_written gets error=-5
[ 1747.801848] sd 0:0:0:0: [sda] tag#1 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=0x00
[ 1747.801853] sd 0:0:0:0: [sda] tag#1 CDB: opcode=0x88 88 00 00 00 00 01 d1 c0 be 00 00 00 00 08 00 00
[ 1747.801855] blk_update_request: I/O error, dev sda, sector 7814036992
[ 1747.801859] Buffer I/O error on dev sda, logical block 976754624, async page read
[ 1747.802151] sd 0:0:0:0: [sda] tag#2 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=0x00
[ 1747.802156] sd 0:0:0:0: [sda] tag#2 CDB: opcode=0x88 88 00 00 00 00 01 d1 c0 bd 00 00 00 00 08 00 00
[ 1747.802159] blk_update_request: I/O error, dev sda, sector 7814036736
[ 1747.802199] sd 0:0:0:0: [sda] tag#4 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=0x00
[ 1747.802203] sd 0:0:0:0: [sda] tag#4 CDB: opcode=0x88 88 00 00 00 00 01 d1 c0 bd 00 00 00 00 08 00 00
[ 1747.802205] blk_update_request: I/O error, dev sda, sector 7814036736
[ 1747.802208] Buffer I/O error on dev md0, logical block 976721824, async page read
[ 1747.802416] sd 1:0:0:0: [sdb] tag#5 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=0x00
[ 1747.802421] sd 1:0:0:0: [sdb] tag#5 CDB: opcode=0x88 88 00 00 00 00 01 d1 c0 be 00 00 00 00 08 00 00
[ 1747.802424] blk_update_request: I/O error, dev sdb, sector 7814036992
[ 1747.802450] sd 1:0:0:0: [sdb] tag#6 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=0x00
[ 1747.802455] sd 1:0:0:0: [sdb] tag#6 CDB: opcode=0x88 88 00 00 00 00 01 d1 c0 be 00 00 00 00 08 00 00
[ 1747.802457] blk_update_request: I/O error, dev sdb, sector 7814036992
[ 1747.802460] Buffer I/O error on dev sdb, logical block 976754624, async page read
[ 1747.868685] blk_update_request: I/O error, dev sda, sector 2465152
[ 1747.874982] sd 0:0:0:0: [sda] tag#5 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=0x00
[ 1747.874995] sd 0:0:0:0: [sda] tag#5 CDB: opcode=0x88 88 00 00 00 00 00 00 25 9d 80 00 00 00 08 00 00
[ 1747.875003] blk_update_request: I/O error, dev sda, sector 2465152
[ 1747.881359] sd 1:0:0:0: [sdb] tag#8 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=0x00
[ 1747.881368] sd 1:0:0:0: [sdb] tag#8 CDB: opcode=0x88 88 00 00 00 00 00 00 25 9d 80 00 00 00 08 00 00
[ 1747.881372] blk_update_request: I/O error, dev sdb, sector 2465152
[ 1747.887606] md/raid1:md0: sda: unrecoverable I/O read error for block 2203008
[ 1747.894891] md/raid1:md0: sda: unrecoverable I/O read error for block 2203136
[ 1747.902134] md/raid1:md0: sda: unrecoverable I/O read error for block 2203264
[ 1747.909350] md/raid1:md0: sda: unrecoverable I/O read error for block 2203392
[ 1747.916810] md: checkpointing resync of md0.
[ 1747.916856] md: super_written gets error=-5
[ 1747.916922] md: super_written gets error=-5
[ 1747.916948] RAID1 conf printout:
[ 1747.916952]  --- wd:1 rd:2
[ 1747.916956]  disk 0, wo:0, o:1, dev:sda
[ 1747.916959]  disk 1, wo:1, o:0, dev:sdb
[ 1747.951666] RAID1 conf printout:
[ 1747.951672]  --- wd:1 rd:2
[ 1747.951676]  disk 0, wo:0, o:1, dev:sda
[ 1747.951746] md: super_written gets error=-5
[ 1747.951804] md: super_written gets error=-5
[ 1747.951848] md: super_written gets error=-5
[ 1747.951902] md: resync of RAID array md0
[ 1747.951907] md: minimum _guaranteed_  speed: 1000 KB/sec/disk.
[ 1747.951911] md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for resync.
[ 1747.951916] md: using 128k window, over a total of 3906887360k.
[ 1747.952111] md: md0: resync done.
[ 1747.952334] md: super_written gets error=-5
[ 1747.952381] md: super_written gets error=-5

oddly enough, the array has been mounted:

# df -h
Filesystem                Size      Used Available Use% Mounted on
/dev/mmcblk0p1            7.3G      1.7G      5.6G  23% /
tmpfs                  1011.1M    468.0K   1010.7M   0% /tmp
tmpfs                   512.0K      4.0K    508.0K   1% /dev
/dev/md0                  3.6T      1.3G      3.6T   0% /mnt/raid

but is read-only:

/mnt/raid# touch foo
touch: foo: Read-only file system

If sdb has failed, maybe this is reason why it is read-only.

/ edit 3

edit 4

Looks like sdb is faulty:

# cat /proc/mdstat
Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4] [multipath] 
md0 : active raid1 sda[0] sdb[1](F)
      3906887360 blocks super 1.2 [2/1] [U_]
      
unused devices: <none>
root@turris:~# mdadm --detail /dev/md0
/dev/md0:
        Version : 1.2
  Creation Time : Fri Jul 14 07:55:09 2017
     Raid Level : raid1
     Array Size : 3906887360 (3725.90 GiB 4000.65 GB)
  Used Dev Size : 3906887360 (3725.90 GiB 4000.65 GB)
   Raid Devices : 2
  Total Devices : 2
    Persistence : Superblock is persistent

    Update Time : Sat Jul 15 09:06:45 2017
          State : clean, degraded 
 Active Devices : 1
Working Devices : 1
 Failed Devices : 1
  Spare Devices : 0

    Number   Major   Minor   RaidDevice State
       0       8        0        0      active sync   /dev/sda
       1       0        0        1      removed

       1       8       16        -      faulty spare   /dev/sdb

Is the drive repairable or do I need to remove it and return it to Amazon?

/ edit 4

Thoughts?

Thanks.