为何LVS/TUN方式不能正常工作呢?

Question

情况交代:
1>主机:win XP,本地连接的网卡LAN接局域网,IP地址:192.168.1.79/24,本地连接的Internet连接共享属性中,选择VMware Network Adapter VMnet1虚拟网卡,VMware Network Adapter VMnet1的IP地址:192.168.0.1/24

2>vm:ubuntu
Loadbalance:192.168.0.10/24
Realserver1:192.168.0.12/24
Realserver2:192.168.0.13/24

3>在win XP的DOS窗口下可以ping通192.168.0.10-192.168.0.13,在ubuntu下它们之间也可以相互ping通
4>相同的环境下,LVS/DR方式可以顺利工作
5>LVS/TUN方式下的一些配置文件内容摘要:
Loadbalance:

1. echo "1" >/proc/sys/net/ipv4/ip_forward
   
2. ifconfig eth0:0 192.168.0.20 netmask 255.255.255.255 broadcast 192.168.0.20
   
3. $IPVSADM -C
   
4. $IPVSADM -A -t 192.168.0.20:80 -s rr
   
5. $IPVSADM -a -t 192.168.0.20:80 -r 192.168.0.12 -i
   
6. $IPVSADM -a -t 192.168.0.20:80 -r 192.168.0.13 -i
   

复制代码

Realserver1 & Realserver2:

1. ifconfig tunl0 192.168.0.20 netmask 255.255.255.255 broadcast 192.168.0.20
   
2. route add -host 192.168.0.20 dev tunl0
   
3. echo "1" > /proc/sys/net/ipv4/ip_forward
   
4. echo "1" > /proc/sys/net/ipv4/conf/all/hidden
   
5. echo "1" > /proc/sys/net/ipv4/conf/tunl0/hidden

复制代码

6>Realserver1 & Realserver2的kernel都已经支持了IP Tunnel,见图

kernel_jpeg.JPG (89.54 KB, 下载次数: 29)

下载附件

2006-06-30 12:15 上传

7>Loadbalance的route表:

1. root@Loadbalance:/etc/init.d# route -n
   
2. Kernel IP routing table
   
3. Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
   
4. 192.168.0.0     0.0.0.0         255.255.255.0   U     0      0        0 eth0
   
5. 10.0.0.0        0.0.0.0         255.0.0.0       U     0      0        0 eth1
   
6. 0.0.0.0         192.168.0.1     0.0.0.0         UG    0      0        0 eth0

复制代码
8>Realserver的route表:

1. root@Realserver1:/etc/init.d# route -n
   
2. Kernel IP routing table
   
3. Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
   
4. 192.168.0.20    0.0.0.0         255.255.255.255 UH    0      0        0 tunl0
   
5. 192.168.0.0     0.0.0.0         255.255.255.0   U     0      0        0 eth0
   
6. 0.0.0.0         192.168.0.1     0.0.0.0         UG    0      0        0 eth0

复制代码

9>但是,当我在win XP下从IE访问虚拟IP:http://192.168.0.20时,却不能够正常访问到测试的web页面
大家帮我看看我那里出错了,不胜感激!

[ 本帖最后由 Linuxcn.com 于 2006-6-30 12:21 编辑 ]

Answer 1

今天看了LVS的官网http://www.linuxvirtualserver.org中的LVS HOWTO FOR LVS/TUN中的一段:

1. 7.1. You need a tunl0 device
2. In LVS-Tun, the tunl0 device holds the VIP, just as the lo device holds the device for LVS-DR.
4. You need to build the tunl0 device into the Linux kernel (in networking options - IP:tunneling) - it is turned off by default. tunl0 is a networking device like eth0, lo, dummy0. The tunnelling can be built as a module, in which case you'll have to insmod ipip before you can use it, or you can build it directly into the kernel.
6. With a kernel built for tunneling, you should be able to see the unconfigured tunl0 device with ifconfig or with ip addr show (Feb 2004 - my ifconfig used to see the unconfigured tunl0, but it doesn't anymore.)

复制代码

按照它说的,我的IP:tunneling是配置成了模块方式的了,这样可以在ifconfig tunl0 vip network bcast前,要先通过命令insmod ipip来加载IP:tunneling这么模块,
可以,当我这样做的时候:

1. root@Realserver1:/usr/src/linux# insmod ipip
2. insmod: can't read 'ipip': No such file or directory

复制代码

这是为什么呢?

大家都不说话了吗?

Answer 2

大侠们伸个手

Answer 3

那位大哥来指点一下啊

Answer 4

我LVS/DR方式时抓的包:
Loadbalance:

1. root@Loadbalance:/etc/init.d# tcpdump dst 192.168.0.20
2. tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
3. listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes
4. 11:12:14.495745 IP 192.168.0.1.3579 > 192.168.0.20.www: S 1508679512:1508679512(0) win 65535 <mss 1460,nop,nop,sackOK>
5. 11:12:14.515957 IP 192.168.0.1.3579 > 192.168.0.20.www: S 1508679512:1508679512(0) win 65535 <mss 1460,nop,nop,sackOK>
6. 11:12:14.496939 IP 192.168.0.1.3579 > 192.168.0.20.www: . ack 2153076980 win 65535
7. 11:12:14.497034 IP 192.168.0.1.3579 > 192.168.0.20.www: . ack 1 win 65535
8. 11:12:14.496975 IP 192.168.0.1.3579 > 192.168.0.20.www: P 0:454(454) ack 1 win 65535
9. 11:12:14.497110 IP 192.168.0.1.3579 > 192.168.0.20.www: P 0:454(454) ack 1 win 65535
10. 11:12:14.600835 IP 192.168.0.1.3579 > 192.168.0.20.www: . ack 367 win 65169
11. 11:12:14.600883 IP 192.168.0.1.3579 > 192.168.0.20.www: . ack 367 win 65169
12. 11:12:32.695592 IP 192.168.0.1.3579 > 192.168.0.20.www: . ack 368 win 65169
13. 11:12:32.696973 IP 192.168.0.1.3579 > 192.168.0.20.www: . ack 368 win 65169
14. 11:12:33.626790 IP 192.168.0.1.3579 > 192.168.0.20.www: R 454:454(0) ack 368 win 0
15. 11:12:33.626845 IP 192.168.0.1.3579 > 192.168.0.20.www: R 454:454(0) ack 368 win 0
16. 11:12:43.782567 IP 192.168.0.1.3581 > 192.168.0.20.www: S 2172478390:2172478390(0) win 65535 <mss 1460,nop,nop,sackOK>
17. 11:12:43.782879 IP 192.168.0.1.3581 > 192.168.0.20.www: S 2172478390:2172478390(0) win 65535 <mss 1460,nop,nop,sackOK>
18. 11:12:43.783658 IP 192.168.0.1.3581 > 192.168.0.20.www: . ack 430936399 win 65535
19. 11:12:43.783747 IP 192.168.0.1.3581 > 192.168.0.20.www: . ack 1 win 65535
20. 11:12:43.783684 IP 192.168.0.1.3581 > 192.168.0.20.www: P 0:454(454) ack 1 win 65535
21. 11:12:43.784286 IP 192.168.0.1.3581 > 192.168.0.20.www: P 0:454(454) ack 1 win 65535
22. 11:12:43.819822 IP 192.168.0.1.3581 > 192.168.0.20.www: . ack 367 win 65169
23. 11:12:43.819881 IP 192.168.0.1.3581 > 192.168.0.20.www: . ack 367 win 65169
24. 11:13:01.135798 IP 192.168.0.1.3581 > 192.168.0.20.www: . ack 368 win 65169
25. 11:13:01.135872 IP 192.168.0.1.3581 > 192.168.0.20.www: . ack 368 win 65169

复制代码

Realserver1:

1. root@Realserver1:/etc/init.d# tcpdump src 192.168.0.10
2. tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
3. listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes
4. 13:45:55.017984 IP 192.168.0.10.1024 > dns-chj.sh.cncnet.net.domain:  46347+ PTR? 20.0.168.192.in-addr.arpa. (43)
5. 13:45:55.033165 IP 192.168.0.10.1024 > dns-chj.sh.cncnet.net.domain:  14949+ PTR? 1.0.168.192.in-addr.arpa. (42)
6. 13:45:55.049390 IP 192.168.0.10.ssh > 192.168.0.1.3248: P 4234787174:4234787338(164) ack 295370700 win 9648
7. 13:45:55.052651 IP 192.168.0.10.ssh > 192.168.0.1.3248: P 164:328(164) ack 1 win 9648
8. 13:45:55.059150 IP 192.168.0.10.ssh > 192.168.0.1.3248: P 328:460(132) ack 1 win 9648
9. 13:45:55.062662 IP 192.168.0.10.ssh > 192.168.0.1.3248: P 460:576(116) ack 1 win 9648
10. 13:45:55.065992 IP 192.168.0.10.ssh > 192.168.0.1.3248: P 576:708(132) ack 1 win 9648
11. 13:45:55.067946 IP 192.168.0.10.ssh > 192.168.0.1.3248: P 708:840(132) ack 1 win 9648
12. 13:45:55.142332 IP 192.168.0.10.ssh > 192.168.0.1.3248: P 840:956(116) ack 1 win 9648
13. 13:45:55.142343 IP 192.168.0.10.ssh > 192.168.0.1.3248: P 956:1072(116) ack 1 win 9648
14. 13:45:59.417299 arp who-has 192.168.0.13 tell 192.168.0.10
15. 13:46:13.879055 IP 192.168.0.10.ssh > 192.168.0.1.3248: P 1072:1188(116) ack 1 win 9648
16. 13:46:13.885049 IP 192.168.0.10.ssh > 192.168.0.1.3248: P 1188:1304(116) ack 1 win 9648
17. 13:46:14.970845 IP 192.168.0.10.ssh > 192.168.0.1.3248: P 1304:1436(132) ack 1 win 9648
18. 13:46:14.973078 IP 192.168.0.10.ssh > 192.168.0.1.3248: P 1436:1568(132) ack 1 win 9648
19. 13:46:25.379295 IP 192.168.0.10.ssh > 192.168.0.1.3248: P 1568:1732(164) ack 1 win 9648
20. 13:46:25.381318 IP 192.168.0.10.ssh > 192.168.0.1.3248: P 1732:1896(164) ack 1 win 9648
21. 13:46:25.386048 IP 192.168.0.10.ssh > 192.168.0.1.3248: P 1896:2028(132) ack 1 win 9648
22. 13:46:25.390993 IP 192.168.0.10.ssh > 192.168.0.1.3248: P 2028:2144(116) ack 1 win 9648
23. 13:46:25.397327 IP 192.168.0.10.ssh > 192.168.0.1.3248: P 2144:2276(132) ack 1 win 9648
24. 13:46:25.398270 IP 192.168.0.10.ssh > 192.168.0.1.3248: P 2276:2408(132) ack 1 win 9648
25. 13:46:25.441275 IP 192.168.0.10.ssh > 192.168.0.1.3248: P 2408:2524(116) ack 1 win 9648
26. 13:46:25.443290 IP 192.168.0.10.ssh > 192.168.0.1.3248: P 2524:2640(116) ack 1 win 9648
27. 13:46:29.592310 arp who-has Realserver1 tell 192.168.0.10
28. 13:46:41.591921 IP 192.168.0.10.ssh > 192.168.0.1.3248: P 2640:2756(116) ack 1 win 9648
29. 13:46:41.595049 IP 192.168.0.10.ssh > 192.168.0.1.3248: P 2756:2872(116) ack 1 win 9648
30. 13:47:50.542438 IP 192.168.0.10.netbios-ns > 192.168.0.255.netbios-ns: NBT UDP PACKET(137): QUERY; REQUEST; BROADCAST
31. 13:47:50.551414 arp reply 192.168.0.10 is-at 00:0c:29:77:76:c7 (oui Unknown)

复制代码

Realserver2:

1. root@Realserver2:/etc/init.d# tcpdump src 192.168.0.10
2. tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
3. listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes
4. 14:15:09.051809 IP 192.168.0.10.1024 > dns-chj.sh.cncnet.net.domain:  46347+ PTR? 20.0.168.192.in-addr.arpa. (43)
5. 14:15:09.070910 IP 192.168.0.10.1024 > dns-chj.sh.cncnet.net.domain:  14949+ PTR? 1.0.168.192.in-addr.arpa. (42)
6. 14:15:09.098295 IP 192.168.0.10.ssh > 192.168.0.1.3248: P 4234787174:4234787338(164) ack 295370700 win 9648
7. 14:15:09.098340 IP 192.168.0.10.ssh > 192.168.0.1.3248: P 164:328(164) ack 1 win 9648
8. 14:15:09.098384 IP 192.168.0.10.ssh > 192.168.0.1.3248: P 328:460(132) ack 1 win 9648
9. 14:15:09.098597 IP 192.168.0.10.ssh > 192.168.0.1.3248: P 460:576(116) ack 1 win 9648
10. 14:15:09.098655 IP 192.168.0.10.ssh > 192.168.0.1.3248: P 576:708(132) ack 1 win 9648
11. 14:15:09.102670 IP 192.168.0.10.ssh > 192.168.0.1.3248: P 708:840(132) ack 1 win 9648
12. 14:15:09.195099 IP 192.168.0.10.ssh > 192.168.0.1.3248: P 840:956(116) ack 1 win 9648
13. 14:15:09.195115 IP 192.168.0.10.ssh > 192.168.0.1.3248: P 956:1072(116) ack 1 win 9648
14. 14:15:12.290227 arp who-has Realserver2 tell 192.168.0.10
15. 14:15:25.805998 IP 192.168.0.10.ssh > 192.168.0.1.3248: P 1072:1188(116) ack 1 win 9648
16. 14:15:25.812478 IP 192.168.0.10.ssh > 192.168.0.1.3248: P 1188:1304(116) ack 1 win 9648
17. 14:15:27.440328 IP 192.168.0.10.ssh > 192.168.0.1.3248: P 1304:1436(132) ack 1 win 9648
18. 14:15:27.440328 IP 192.168.0.10.ssh > 192.168.0.1.3248: P 1436:1568(132) ack 1 win 9648
19. 14:15:37.461602 IP 192.168.0.10.ssh > 192.168.0.1.3248: P 1568:1732(164) ack 1 win 9648
20. 14:15:37.466608 IP 192.168.0.10.ssh > 192.168.0.1.3248: P 1732:1896(164) ack 1 win 9648
21. 14:15:37.469582 IP 192.168.0.10.ssh > 192.168.0.1.3248: P 1896:2028(132) ack 1 win 9648
22. 14:15:37.474732 IP 192.168.0.10.ssh > 192.168.0.1.3248: P 2028:2144(116) ack 1 win 9648
23. 14:15:37.479656 IP 192.168.0.10.ssh > 192.168.0.1.3248: P 2144:2276(132) ack 1 win 9648
24. 14:15:37.482315 IP 192.168.0.10.ssh > 192.168.0.1.3248: P 2276:2408(132) ack 1 win 9648
25. 14:15:37.542983 IP 192.168.0.10.ssh > 192.168.0.1.3248: P 2408:2524(116) ack 1 win 9648
26. 14:15:37.545591 IP 192.168.0.10.ssh > 192.168.0.1.3248: P 2524:2640(116) ack 1 win 9648
27. 14:15:43.928563 arp who-has 192.168.0.12 tell 192.168.0.10
28. 14:15:55.998471 IP 192.168.0.10.ssh > 192.168.0.1.3248: P 2640:2756(116) ack 1 win 9648
29. 14:15:56.002589 IP 192.168.0.10.ssh > 192.168.0.1.3248: P 2756:2872(116) ack 1 win 9648
30. 14:17:04.177823 IP 192.168.0.10.netbios-ns > 192.168.0.255.netbios-ns: NBT UDP PACKET(137): QUERY; REQUEST; BROADCAST
31. 14:17:04.185172 arp reply 192.168.0.10 is-at 00:0c:29:77:76:c7 (oui Unknown)

复制代码

LVS/DR方式能够正常工作

Answer 5

当我做这个动作:从IE里输入:http://192.168.0.20访问web服务时,在Loadbalance机器上监视虚拟IP

1. root@Loadbalance:/etc/init.d# tcpdump dst 192.168.0.20
2. tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
3. listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes
4. 11:01:32.495079 IP 192.168.0.1.3478 > 192.168.0.20.www: S 1562695239:1562695239(0) win 65535 <mss 1460,nop,nop,sackOK>
5. 11:01:35.114333 IP 192.168.0.1.3478 > 192.168.0.20.www: S 1562695239:1562695239(0) win 65535 <mss 1460,nop,nop,sackOK>
6. 11:01:38.467705 IP 192.168.0.1.3478 > 192.168.0.20.www: S 1562695239:1562695239(0) win 65535 <mss 1460,nop,nop,sackOK>
7. 11:04:24.610365 arp who-has 192.168.0.20 tell 192.168.0.1
8. 11:04:24.611726 IP 192.168.0.1.3479 > 192.168.0.20.www: S 2187440212:2187440212(0) win 65535 <mss 1460,nop,nop,sackOK>
9. 11:04:26.425108 IP 192.168.0.1.3479 > 192.168.0.20.www: S 2187440212:2187440212(0) win 65535 <mss 1460,nop,nop,sackOK>
10. 11:04:32.180984 IP 192.168.0.1.3479 > 192.168.0.20.www: S 2187440212:2187440212(0) win 65535 <mss 1460,nop,nop,sackOK>

复制代码

在RS上监视Loadbalance的IP动作

1. root@Realserver1:/home# tcpdump src 192.168.0.10
2. tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
3. listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes
4. 13:37:57.111593 IP 192.168.0.10 > 192.168.0.13: IP 192.168.0.1.3479 > 192.168.0.20.www: S 2187440212:2187440212(0) win 65535 <mss 1460,nop,nop,sackOK> (ipip-proto-4)
5. 13:37:57.118789 IP 192.168.0.10.ssh > 192.168.0.1.3248: P 4234777010:4234777110(100) ack 295366668 win 8576
6. 13:37:57.120787 IP 192.168.0.10.ssh > 192.168.0.1.3248: P 100:264(164) ack 1 win 8576
7. 13:37:57.122823 arp reply 192.168.0.10 is-at 00:0c:29:77:76:c7 (oui Unknown)
8. 13:37:59.430390 IP 192.168.0.10 > 192.168.0.13: IP 192.168.0.1.3479 > 192.168.0.20.www: S 2187440212:2187440212(0) win 65535 <mss 1460,nop,nop,sackOK> (ipip-proto-4)
9. 13:37:59.430476 IP 192.168.0.10.ssh > 192.168.0.1.3248: P 264:428(164) ack 1 win 8576
10. 13:38:02.742802 arp who-has 192.168.0.13 tell 192.168.0.10
11. 13:38:05.031489 IP 192.168.0.10 > 192.168.0.13: IP 192.168.0.1.3479 > 192.168.0.20.www: S 2187440212:2187440212(0) win 65535 <mss 1460,nop,nop,sackOK> (ipip-proto-4)
12. 13:38:05.034128 IP 192.168.0.10.ssh > 192.168.0.1.3248: P 428:592(164) ack 1 win 8576
13. 13:38:22.508823 IP 192.168.0.10.netbios-dgm > 192.168.0.255.netbios-dgm: NBT UDP PACKET(138)

复制代码

不明白像这样的信息是什么意思:
13:37:59.430476 IP 192.168.0.10.ssh > 192.168.0.1.3248: P 264:428(164) ack 1 win 8576

192.168.0.1是主机上的虚拟网卡IP,其他4台机器都是VM ware虚拟的

[ 本帖最后由 Linuxcn.com 于 2006-7-4 14:37 编辑 ]

Answer 6

顶了

Answer 7

原帖由 crazycash 于 2006-7-3 17:10 发表
在win XP下从IE访问虚拟IP:http://192.168.0.12／13，能够正常访问到测试的web页面么？
从loadbalance能够正常ping通192.168.0.2么？
使用tcpdump命令看看有没有包被转发过来。

另外可以把192.168.0.2配在tunl0:0上，这样如果有其他服务也作loadbalance，就不会引起冲突。

我开始也是想把虚拟IP在RS上配置到tunl0:0上来的， 但是配的时候它说设备不存在

Answer 8

在win XP下从IE访问虚拟IP:http://192.168.0.12／13，能够正常访问到测试的web页面么？
从loadbalance能够正常ping通192.168.0.2么？
使用tcpdump命令看看有没有包被转发过来。

另外可以把192.168.0.2配在tunl0:0上，这样如果有其他服务也作loadbalance，就不会引起冲突。

Answer 9

小顶一下