kvm虚拟机磁盘出现readonly

Question

场景：

使用kvm+mfs共享存储，搭建的一套虚拟化集群。mfs挂载到宿主机上，vm镜像放在mfs 存储上。使用virsh create /mnt/mfs/dir/vm-xx.xml 运行虚拟机。

近期发现虚拟机有部分机器磁盘出现readonly的情况，查看后端mfs存储，并未见高负载，宿主机也负载也正常。

分析：
linux 磁盘readonly产生原因？
linux 读写io过程出现超时、error等，linux认为磁盘损坏，将磁盘重新挂载为保护模式，只允许读。
猜测分析是由于后端存储响应超时、或者物理硬盘有坏道，导致了问题发生。

解决方法：
目前并未找到非常好的解决方法，仅仅是通过重启虚拟机，使用fsck修复。

尝试的其他方法：
1、修改磁盘io电梯算法，目前使用的是deadline，改为使用cfq模式。
2、加大vm读写超时时间，避免因为超时产生此类问题：
/sys/block//queue/iosched/read_expire
读请求的过期时间
/sys/block/sdb/queue/iosched/write_expire
写请求的过期时间

目前没有其他思路，如有好的解决方法，有偿答谢。

相关报错日志如下：

/var/log/message:

Feb 3 07:31:30 yz3127 kernel: sd 0:0:0:0: [sda] Unhandled error code
Feb 3 07:31:30 yz3127 kernel: sd 0:0:0:0: [sda] Result: hostbyte=DID_OK driverbyte=
Feb 3 07:31:30 yz3127 kernel: sd 0:0:0:0: [sda] CDB: Write(10): 2a 00 00 0b b4 df 00 00 18 00
Feb 3 07:31:30 yz3127 kernel: end_request: I/O error, dev sda, sector 767199
Feb 3 07:31:30 yz3127 kernel: Buffer I/O error on device sda1, logical block 95892
Feb 3 07:31:30 yz3127 kernel: lost page write due to I/O error on sda1
Feb 3 07:31:30 yz3127 kernel: Buffer I/O error on device sda1, logical block 95893
Feb 3 07:31:30 yz3127 kernel: lost page write due to I/O error on sda1
Feb 3 07:31:30 yz3127 kernel: Buffer I/O error on device sda1, logical block 95894
Feb 3 07:31:30 yz3127 kernel: lost page write due to I/O error on sda1

dmesg：

end_request: I/O error, dev sda, sector 767199
Buffer I/O error on device sda1, logical block 95892
lost page write due to I/O error on sda1
Buffer I/O error on device sda1, logical block 95893
lost page write due to I/O error on sda1
Buffer I/O error on device sda1, logical block 95894
lost page write due to I/O error on sda1
Aborting journal on device sda1-8.
EXT4-fs error (device sda1): ext4_journal_start_sb:
EXT4-fs error (device sda1): ext4_journal_start_sb: Detected aborted journal
EXT4-fs (sda1): Remounting filesystem read-only
Detected aborted journal
EXT4-fs error (device sda1): ext4_remount: Abort forced by user

Jan 31 02:36:29 yz10130 kernel: sd 0:0:0:0: [sda] Unhandled error code
Jan 31 02:36:29 yz10130 kernel: sd 0:0:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
Jan 31 02:36:29 yz10130 kernel: sd 0:0:0:0: [sda] CDB: Write(10): 2a 00 00 c0 14 bf 00 00 08 00
Jan 31 02:36:29 yz10130 kernel: end_request: I/O error, dev sda, sector 12588223
Jan 31 02:36:29 yz10130 kernel: Buffer I/O error on device sda1, logical block 1573520
Jan 31 02:36:29 yz10130 kernel: lost page write due to I/O error on sda1

10.39.6.235

Jan 30 06:49:36 yz6235 kernel: Aborting journal on device sda2-8.
Jan 30 06:49:36 yz6235 kernel: EXT4-fs error (device sda2): ext4_journal_start_sb: Detected aborted journal
Jan 30 06:49:36 yz6235 kernel: EXT4-fs (sda2): Remounting filesystem read-only

10.39.3.139

Jan 21 03:17:06 yz3139 kernel: JBD2: Detected IO errors while flushing file data on sda1-8
Jan 21 03:17:12 yz3139 kernel: JBD2: Detected IO errors while flushing file data on sda1-8
Jan 21 03:17:26 yz3139 kernel: JBD2: Detected IO errors while flushing file data on sda1-8
Jan 21 03:17:46 yz3139 kernel: JBD2: Detected IO errors while flushing file data on sda1-8
Jan 21 03:17:52 yz3139 kernel: JBD2: Detected IO errors while flushing file data on sda1-8
Jan 21 03:18:06 yz3139 kernel: JBD2: Detected IO errors while flushing file data on sda1-8
Jan 21 03:18:12 yz3139 kernel: JBD2: Detected IO errors while flushing file data on sda1-8
Jan 21 03:18:18 yz3139 kernel: JBD2: Detected IO errors while flushing file data on sda1-8
Jan 21 03:18:24 yz3139 kernel: JBD2: Detected IO errors while flushing file data on sda1-8
Jan 21 03:18:36 yz3139 kernel: JBD2: Detected IO errors while flushing file data on sda1-8
Jan 21 03:18:42 yz3139 kernel: JBD2: Detected IO errors while flushing file data on sda1-8
Jan 21 03:18:55 yz3139 kernel: JBD2: Detected IO errors while flushing file data on sda1-8
Jan 21 03:19:06 yz3139 kernel: JBD2: Detected IO errors while flushing file data on sda1-8
Jan 21 03:19:26 yz3139 kernel: JBD2: Detected IO errors while flushing file data on sda1-8
Jan 21 03:19:32 yz3139 kernel: JBD2: Detected IO errors while flushing file data on sda1-8
Jan 21 03:19:46 yz3139 kernel: JBD2: Detected IO errors while flushing file data on sda1-8

Jan 24 16:36:37 yz10126 kernel: end_request: I/O error, dev sda, sector 76094911
Jan 24 16:36:37 yz10126 kernel: Buffer I/O error on device sda1, logical block 9511856
Jan 24 16:36:37 yz10126 kernel: lost page write due to I/O error on sda1
Jan 24 16:36:37 yz10126 kernel: JBD2: Detected IO errors while flushing file data on sda1-8