The S.M.A.R.T. test verified my suspicion:
We run the background disk test which does not disturb the main operation:
smartctl -t long /dev/ad5
We wait for finishing execution, viewing the status from time to time:
smartctl -l selftest /dev/ad5
Finally, we look at the log contents, at the very end of the output:
smartctl -a /dev/ad5
We have:
# 1 Extended offline Completed: read failure 90% 2916 10373954
We develop the complete list of bad sectors by reading all data from the disk:
dd if=/dev/ad5 of=/dev/null bs=512 conv=noerror,sync
At some time we will see something like these:
dd: /dev/ad5: Input/output error
10373954+0 records in
10373954+0 records out
5311464448 bytes transferred in 2427.397393 secs (2188131 bytes/sec)
We will see the following in the system log:
kernel: ad5: TIMEOUT - READ_DMA retrying (1 retry left) LBA=10373954
kernel: ad5: TIMEOUT - READ_DMA retrying (0 retries left) LBA=10373954
....
kernel: ad5: FAILURE - READ_DMA timed out LBA=10374109
kernel: ad5: TIMEOUT - READ_DMA retrying (1 retry left) LBA=10374113
We check each sector one more time:
dd if=/dev/ad5 of=/dev/null bs=512 count=1 skip=10373954 conv=noerror,sync
We see which file in the file system comes within this block.
We see and roughly figure out the number of partition on which the bad sector falls (LBA 10373954):
fdisk -s /dev/ad5
/dev/ad5: 775221 cyl 16 hd 63 sec
Part Start Size Type Flags
1: 63 398444067 0xa5 0x80
2: 398444130 382973535 0xa5 0x80
10373954 - 63 = 10373891
disklabel /dev/ad5s1
# /dev/ad5s1:
8 partitions:
# size offset fstype [fsize bsize bps/cpg]
a: 2097152 0 4.2BSD 2048 16384 28552
b: 4194304 2097152 swap
c: 398444067 0 unused 0 0 # "raw" part
d: 2097152 6291456 4.2BSD 2048 16384 28552
e: 10485760 8388608 4.2BSD 2048 16384 28552
f: 104857600 18874368 4.2BSD 2048 16384 28552
g: 104857600 123731968 4.2BSD 2048 16384 28552
h: 162127234 228589568 4.2BSD 2048 16384 28552
We can see that 10373891 falls on the /dev/ad5s1e partition
We calculate the flushing relative to the beginning of the partition
10373891 - 8388608 = 1985283
We find inode, to which the specified block belongs:
fsdb -r /dev/ad5s1e
findblk 1985283
we repeat it for each bad sector:
(you should use use debugfs for Linux - http://smartmontools.sourceforge.net/BadBlockHowTo.txt)
We attempt to write data on the bad sector in order to activate the
remapping process on the disk.
sysctl kern.geom.debugflags=16 # otherwise there will be dd: /dev/ad5: Operation not permitted)
dd if=/dev/zero of=/dev/ad5 bs=512 count=1 skip=10373954 conv=noerror,sync
sysctl kern.geom.debugflags=0
If the sector became readable after the write, everything is OK;
if not, the backup area on the drive is full; we try to bar
adjacent sectors from referencing to the file system:
man badsect
/dev/ad5s1e is mounted as /usr, we create the /usr/BAD directory and execute
badsect /usr/BAD 1985283
fsck -y /dev/ad5s1e
Useful ports:
/usr/ports/sysutils/recoverdm
/usr/ports/sysutils/diskcheckd