centos 6.8 kernel升级至2.6.32-642.4.2.el6.x86_64后无法ssh问题

Question

IBM x3750M4服务器

通过yum update进行OS更新之后重启，出现可以ping通，但是ssh无法连接，通过zabbix也无法监控到机器的运行状态。经过IDC机房人员协助，发现机器界面处于CentOS的启动界面。如下图：

后来通过重启进入单用户模式，并且开启网络服务，能够进行远程ssh登陆。但是，排查发现一些无用的kernel（见下列表），通过yum remove kernel-2.6.32-573.*之后，重启os，仍然无法恢复原先的可以ssh登陆同时能够被zabbix监控到。

# rpm -qa|grep kernel

kernel-headers-2.6.32-642.4.2.el6.x86_64
kernel-2.6.32-573.7.1.el6.x86_64
kernel-2.6.32-642.1.1.el6.x86_64
kernel-2.6.32-573.22.1.el6.x86_64
kernel-2.6.32-573.26.1.el6.x86_64
kernel-firmware-2.6.32-642.4.2.el6.noarch
kernel-2.6.32-642.4.2.el6.x86_64
dracut-kernel-004-409.el6_8.2.noarch

本次更新了如下rpm：

cat /var/log/yum.log

Aug 31 17:42:16 Updated: initscripts-9.03.53-1.el6.centos.1.x86_64

Aug 31 17:42:17 Updated: udev-147-2.73.el6_8.1.x86_64
Aug 31 17:42:20 Updated: python-libs-2.6.6-66.el6_8.x86_64
Aug 31 17:42:21 Updated: python-2.6.6-66.el6_8.x86_64
Aug 31 17:42:21 Updated: libsss_idmap-1.13.3-22.el6_8.4.x86_64
Aug 31 17:42:21 Updated: libudev-147-2.73.el6_8.1.x86_64
Aug 31 17:42:23 Updated: samba4-libs-4.2.10-7.el6_8.x86_64
Aug 31 17:42:23 Updated: libxml2-2.7.6-21.el6_8.1.x86_64
Aug 31 17:42:24 Updated: device-mapper-1.02.117-7.el6_8.1.x86_64
Aug 31 17:42:24 Updated: device-mapper-libs-1.02.117-7.el6_8.1.x86_64
Aug 31 17:42:24 Updated: device-mapper-event-libs-1.02.117-7.el6_8.1.x86_64
Aug 31 17:42:27 Updated: samba-common-3.6.23-36.el6_8.x86_64
Aug 31 17:42:28 Updated: samba-winbind-clients-3.6.23-36.el6_8.x86_64
Aug 31 17:42:29 Updated: samba-winbind-3.6.23-36.el6_8.x86_64
Aug 31 17:42:29 Updated: libipa_hbac-1.13.3-22.el6_8.4.x86_64
Aug 31 17:42:29 Updated: device-mapper-event-1.02.117-7.el6_8.1.x86_64
Aug 31 17:42:30 Updated: lvm2-libs-2.02.143-7.el6_8.1.x86_64
Aug 31 17:42:30 Updated: python-sssdconfig-1.13.3-22.el6_8.4.noarch
Aug 31 17:42:30 Updated: cronie-anacron-1.4.4-16.el6_8.2.x86_64
Aug 31 17:42:31 Updated: cronie-1.4.4-16.el6_8.2.x86_64
Aug 31 17:42:37 Updated: kernel-firmware-2.6.32-642.4.2.el6.noarch
Aug 31 17:42:45 Installed: kernel-2.6.32-642.4.2.el6.x86_64
Aug 31 17:42:48 Updated: clamav-db-0.99.2-1.el6.x86_64
Aug 31 17:42:48 Updated: sssd-client-1.13.3-22.el6_8.4.x86_64
Aug 31 17:42:49 Updated: sssd-common-1.13.3-22.el6_8.4.x86_64
Aug 31 17:42:49 Updated: sssd-krb5-common-1.13.3-22.el6_8.4.x86_64
Aug 31 17:42:50 Updated: sssd-common-pac-1.13.3-22.el6_8.4.x86_64
Aug 31 17:42:50 Updated: sssd-ad-1.13.3-22.el6_8.4.x86_64
Aug 31 17:42:50 Updated: sssd-ipa-1.13.3-22.el6_8.4.x86_64
Aug 31 17:42:51 Updated: sssd-krb5-1.13.3-22.el6_8.4.x86_64
Aug 31 17:42:51 Updated: sssd-ldap-1.13.3-22.el6_8.4.x86_64
Aug 31 17:42:51 Updated: sssd-proxy-1.13.3-22.el6_8.4.x86_64
Aug 31 17:42:51 Updated: nss-softokn-freebl-3.14.3-23.3.el6_8.x86_64
Aug 31 17:42:52 Updated: tzdata-java-2016f-1.el6.noarch
Aug 31 17:43:00 Updated: 1:java-1.7.0-openjdk-1.7.0.111-2.6.7.2.el6_8.x86_64
Aug 31 17:43:01 Updated: nss-softokn-3.14.3-23.3.el6_8.x86_64
Aug 31 17:43:01 Updated: sssd-1.13.3-22.el6_8.4.x86_64
Aug 31 17:43:02 Updated: clamav-0.99.2-1.el6.x86_64
Aug 31 17:43:03 Updated: 1:autofs-5.0.5-123.el6_8.x86_64
Aug 31 17:43:03 Updated: lvm2-2.02.143-7.el6_8.1.x86_64
Aug 31 17:43:03 Updated: python-libipa_hbac-1.13.3-22.el6_8.4.x86_64
Aug 31 17:43:06 Updated: samba-client-3.6.23-36.el6_8.x86_64
Aug 31 17:43:07 Updated: libxml2-python-2.7.6-21.el6_8.1.x86_64
Aug 31 17:43:07 Updated: libgudev1-147-2.73.el6_8.1.x86_64
Aug 31 17:43:08 Updated: yum-3.2.29-75.el6.centos.noarch
Aug 31 17:43:08 Updated: perf-2.6.32-642.4.2.el6.x86_64
Aug 31 17:43:09 Updated: 1:nfs-utils-1.2.3-70.el6_8.1.x86_64
Aug 31 17:43:10 Updated: mdadm-3.3.4-1.el6_8.5.x86_64
Aug 31 17:43:10 Updated: libcgroup-0.40.rc1-18.el6_8.x86_64
Aug 31 17:43:12 Updated: libtiff-3.9.4-18.el6_8.x86_64
Aug 31 17:43:13 Updated: avahi-libs-0.6.25-15.el6_8.1.x86_64
Aug 31 17:43:13 Updated: tcsh-6.17-35.el6_8.1.x86_64
Aug 31 17:43:15 Updated: tzdata-2016f-1.el6.noarch
Aug 31 17:43:16 Updated: kernel-headers-2.6.32-642.4.2.el6.x86_64
Aug 31 17:43:16 Updated: 2:tar-1.23-15.el6_8.x86_64

Aug 31 17:43:17 Updated: nss-softokn-freebl-3.14.3-23.3.el6_8.i686

有没有谁遇到过类似的问题，求指导！

Answer 1

已经收集了SOS日志信息，接下来基于这个日志进行问题的分析。

Answer 2

排查yum.log没有发现error或者warn级别的记录。

Answer 3

主要的问题可能出在：yum update的kernel版本差异太大方面出现的问题。

Answer 4

引用来自“554330833a”的评论

是不是服务器都用Linux？

Answer 5

是不是服务器都用Linux？

Answer 6

检查grub.conf文件内容如下：

# cat grub.conf 
# grub.conf generated by anaconda
#
# Note that you do not have to rerun grub after making changes to this file
# NOTICE:  You have a /boot partition.  This means that
#          all kernel and initrd paths are relative to /boot/, eg.
#          root (hd0,1)
#          kernel /vmlinuz-version ro root=/dev/sda4
#          initrd /initrd-[generic-]version.img
#boot=/dev/sda1
device (hd0) HD(1,800,32000,c9fe98b1-bece-4b99-b98c-66b675223f2f)
default=0
timeout=5
splashimage=(hd0,1)/grub/splash.xpm.gz
hiddenmenu
title CentOS (2.6.32-642.4.2.el6.x86_64)
        root (hd0,1)
        kernel /vmlinuz-2.6.32-642.4.2.el6.x86_64 ro root=UUID=18f359fe-c779-45af-ab78-77d3055ffff0 rd_NO_LUKS rd_NO_LVM LANG=en_US.UTF-8 rd_NO_MD SYSFONT=latarcyrheb-sun16 crashkernel=auto  KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM rhgb quiet
        initrd /initramfs-2.6.32-642.4.2.el6.x86_64.img
title CentOS (2.6.32-642.1.1.el6.x86_64)
        root (hd0,1)
        kernel /vmlinuz-2.6.32-642.1.1.el6.x86_64 ro root=UUID=18f359fe-c779-45af-ab78-77d3055ffff0 rd_NO_LUKS rd_NO_LVM LANG=en_US.UTF-8 rd_NO_MD SYSFONT=latarcyrheb-sun16 crashkernel=auto  KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM rhgb quiet
        initrd /initramfs-2.6.32-642.1.1.el6.x86_64.img

Answer 7

我试想是否是启动文件的问题，但是发现dracut写入没有问题：

#cat dracut.log
Wed Aug 31 17:43:40 CST 2016 Info: Executing /sbin/dracut -f /boot/initramfs-2.6.32-642.4.2.el6.x86_64.img 2.6.32-642.4.2.el6.x86_64
====此处略去很多字，发不了那么多。===
Wed Aug 31 17:44:35 CST 2016 Info: Wrote /boot/initramfs-2.6.32-642.4.2.el6.x86_64.img
Wed Aug 31 17:44:35 CST 2016 Info: -rw------- 1 root root 22309308 Aug 31 17:44 /boot/initramfs-2.6.32-642.4.2.el6.x86_64.img

Answer 8

没有看到错误日志信息。

Answer 9

回复
嗯ESC键可以把启动日志界面显示出来看卡在哪一步

Answer 10

是的。IBM X系列服务器直接按ESC或者F5,就可以看启动状态详情。前面注意力在解决问题，没及时回复。见谅！感谢支持。

Answer 11

那个启动界面可以看错误信息的啊。你先看点错误信息

Answer 12

这个没办法解释。

Answer 13

ssh是否没运行啊。试下重启看看。

好奇地问下，楼主为什么要升级OS啊。。

Answer 14

升级之前的kernel：kernel-2.6.32-573.7.1.el6.x86_64
升级重启之后的kernel: 2.6.32-642.4.2.el6.x86_64

当前在用的kernel相关rpm如下：

# rpm -qa|grep kernel
kernel-headers-2.6.32-642.4.2.el6.x86_64
kernel-2.6.32-642.1.1.el6.x86_64
kernel-firmware-2.6.32-642.4.2.el6.noarch
kernel-2.6.32-642.4.2.el6.x86_64
dracut-kernel-004-409.el6_8.2.noarch