istio部署在virtual machine 与k8s 通讯问题
部署需求：
肉网将 istio 安装在 vm 上，实现网格服务，共三台机器。网络插件是 Flannel
master: 192.168.3.58
vm1: 192.168.3.120
vm2: 192.168.3.56
参考官方教程：https://istio.io/latest/docs/...
安装k8s集群工具 kubeadm，将安装了负载均衡器 metallb，使用的 Layer 2 模式(教程：https://mp.weixin.qq.com/s/Z4...
# k8s-master
root@sxf-virtual-machine:/home/sxf# ksvc -A
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
default kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 168m
istio-system istio-eastwestgateway LoadBalancer 10.106.10.60 192.168.3.251 15021:32430/TCP,15443:30670/TCP,15012:31969/TCP,15017:30699/TCP 135m
istio-system istio-ingressgateway LoadBalancer 10.106.3.144 192.168.3.252 15021:30271/TCP,80:31320/TCP,443:30527/TCP 157m
istio-system istiod ClusterIP 10.101.237.7 <none> 15010/TCP,15012/TCP,443/TCP,15014/TCP 162m
kube-system kube-dns ClusterIP 10.96.0.10 <none> 53/UDP,53/TCP,9153/TCP 168m
# 查看代理信息
root@sxf-virtual-machine:/home/sxf# istioctl proxy-status
NAME CDS LDS EDS RDS ISTIOD VERSION
helloworld-v1-776f57d5f6-9pxn7.sample SYNCED SYNCED SYNCED SYNCED istiod-5bdf585695-l2845 1.10.3
helloworld-v2-54df5f84b-gfqtz.sample SYNCED SYNCED SYNCED SYNCED istiod-5bdf585695-l2845 1.10.3
istio-eastwestgateway-6fd76487d6-2hndb.istio-system SYNCED SYNCED SYNCED NOT SENT istiod-5bdf585695-l2845 1.10.3
istio-ingressgateway-697dc5b889-b54nn.istio-system SYNCED SYNCED SYNCED NOT SENT istiod-5bdf585695-l2845 1.10.3
vm1.vm SYNCED SYNCED SYNCED SYNCED istiod-5bdf585695-l2845 1.10.0
vm2.vm SYNCED SYNCED SYNCED SYNCED istiod-5bdf585695-l2845 1.10.0
# 在master宿主机通过ip可以访问helloworld服务,但域名不可以，提示无法解析（这里应该是正常的，只有pod之间才可以通过域名访问)
root@sxf-virtual-machine:/home/sxf# curl 10.111.123.197:5000/hello
Hello version: v2, instance: helloworld-v2-54df5f84b-gfqtz
root@sxf-virtual-machine:/home/sxf# curl helloworld.sample.svc:5000/hello
curl: (6) Could not resolve host: helloworld.sample.svc网关服务 istio-eastwestgateway 的LB 地址为 192.168.3.251。目前可以看到两台vm的信息。
root@sxf-virtual-machine:/home/sxf# k get wg -A
NAMESPACE NAME AGE
vm myapp 133m
root@sxf-virtual-machine:/home/sxf# k get we -A
NAMESPACE NAME AGE ADDRESS
vm myapp-192.168.3.120 111m 192.168.3.120
vm myapp-192.168.3.56 94m 192.168.3.56在教程中的 "验证安装istio是否成功" 步骤中，istio.log日志正常。但服务curl helloworld.sampel.svc 无法访问，但可以通过dns查找到服务对应的ip地址 10.111.123.197
# vm1 无法访问
root@vm1:/home/sxf# curl helloworld.sample.svc:5000/hello
upstream connect error or disconnect/reset before headers. reset reason: connection failure
# 解析到的信息，ip是正确的
root@vm1:/home/sxf# nslookup helloworld.sample.svc
Server: 127.0.0.53
Address: 127.0.0.53#53
Name: helloworld.sample.svc
Address: 10.111.123.197
# 192.168.3.1是网关地址， 3.2是公司内部的BGP网关
root@vm1:/home/sxf# ping helloworld.sample.svc
PING helloworld.sample.svc (10.111.123.197) 56(84) bytes of data.
From _gateway (192.168.3.1): icmp_seq=3 Redirect Network(New nexthop: 192.168.3.2 (192.168.3.2))
# 端口是通的
root@vm1:/home/sxf# telnet helloworld.sample.svc 5000
Trying 10.111.123.197...
Connected to helloworld.sample.svc.
Escape character is '^]'.
以下是在vm1上的istio域名解析日志
2021-08-04T08:39:41.968114Z debug dns request ;; opcode: QUERY, status: NOERROR, id: 61474
;; flags: rd; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1
;; QUESTION SECTION:
;helloworld.sample.svc. IN AAAA
;; ADDITIONAL SECTION:
;; OPT PSEUDOSECTION:
; EDNS: version 0; flags: ; udp: 1200
protocol=udp edns=true id=ae0dc6d2-d888-473e-b873-136263c1d714
2021-08-04T08:39:41.968371Z debug dns response for hostname "helloworld.sample.svc." (found=true): ;; opcode: QUERY, status: NOERROR, id: 61474
;; flags: qr aa rd; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0
;; QUESTION SECTION:
;helloworld.sample.svc. IN AAAA
protocol=udp edns=true id=ae0dc6d2-d888-473e-b873-136263c1d714
2021-08-04T08:39:41.968115Z debug dns request ;; opcode: QUERY, status: NOERROR, id: 48365
;; flags: rd; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1
;; QUESTION SECTION:
;helloworld.sample.svc. IN A
;; ADDITIONAL SECTION:
;; OPT PSEUDOSECTION:
; EDNS: version 0; flags: ; udp: 1200
protocol=udp edns=true id=e89119b5-37b5-4434-aad1-b3b0dfbf15f0
2021-08-04T08:39:41.968786Z debug dns response for hostname "helloworld.sample.svc." (found=true): ;; opcode: QUERY, status: NOERROR, id: 48365
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0
;; QUESTION SECTION:
;helloworld.sample.svc. IN A
;; ANSWER SECTION:
helloworld.sample.svc. 30 IN A 10.111.123.197
protocol=udp edns=true id=e89119b5-37b5-4434-aad1-b3b0dfbf15f0可以看到几个关键点 found=true 和 helloworld.sample.svc. 30 IN A 10.111.123.197, 所以这里dns 也应该是没有问题的。
master 主机信息
root@sxf-virtual-machine:/home/sxf# ip route
default via 192.168.3.1 dev ens160 proto dhcp metric 100
10.244.0.0/24 dev cni0 proto kernel scope link src 10.244.0.1
169.254.0.0/16 dev ens160 scope link metric 1000
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown
192.168.3.0/24 dev ens160 proto kernel scope link src 192.168.3.58 metric 100
# hosts
root@sxf-virtual-machine:/home/sxf# more /etc/hosts
127.0.0.1 localhost
127.0.1.1 sxf-virtual-machine
# The following lines are desirable for IPv6 capable hosts
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
# nameserver
root@sxf-virtual-machine:/home/sxf# more /etc/resolv.conf
nameserver 127.0.0.53
options edns0
search anee.com.cn
vm1主机信息
# hosts
root@vm1:/home/sxf# more /etc/hosts
127.0.0.1 localhost
127.0.1.1 vm1
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
192.168.3.251 istiod.istio-system.svc
# nameserver
root@vm1:/home/sxf# more /etc/resolv.conf
nameserver 127.0.0.53
options edns0
# 路由信息
root@vm1:/home/sxf# ip route
default via 192.168.3.1 dev ens36 proto static metric 20100
169.254.0.0/16 dev ens36 scope link metric 1000
192.168.3.0/24 dev ens36 proto kernel scope link src 192.168.3.120 metric 100
root@vm1:/home/sxf# route -n
内核 IP 路由表
目标 网关 子网掩码 标志 跃点 引用 使用 接口
0.0.0.0 192.168.3.1 0.0.0.0 UG 20100 0 0 ens36
169.254.0.0 0.0.0.0 255.255.0.0 U 1000 0 0 ens36
192.168.3.0 0.0.0.0 255.255.255.0 U 100 0 0 ens36
# 网卡信息
root@vm1:/home/sxf# ifconfig
ens36: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.3.120 netmask 255.255.255.0 broadcast 192.168.3.255
inet6 fe80::8e95:5567:8fa4:e0e7 prefixlen 64 scopeid 0x20<link>
ether 00:0c:29:e7:86:b1 txqueuelen 1000 (以太网)
RX packets 2164253 bytes 206952534 (206.9 MB)
RX errors 0 dropped 24275 overruns 0 frame 0
TX packets 112319 bytes 18330399 (18.3 MB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10<host>
loop txqueuelen 1000 (本地环回)
RX packets 26287 bytes 13980478 (13.9 MB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 26287 bytes 13980478 (13.9 MB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
# iptables
root@vm1:/home/sxf# iptables-save
# Generated by iptables-save v1.6.1 on Wed Aug 4 17:21:36 2021
*filter
:INPUT ACCEPT [29263:14149342]
:FORWARD DROP [0:0]
:OUTPUT ACCEPT [14082:6893008]
COMMIT
# Completed on Wed Aug 4 17:21:36 2021
# Generated by iptables-save v1.6.1 on Wed Aug 4 17:21:36 2021
*mangle
:PREROUTING ACCEPT [313262:101209145]
:INPUT ACCEPT [308840:100700735]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [101662:27585942]
:POSTROUTING ACCEPT [101762:27615155]
COMMIT
# Completed on Wed Aug 4 17:21:36 2021
# Generated by iptables-save v1.6.1 on Wed Aug 4 17:21:36 2021
*nat
:PREROUTING ACCEPT [3954:385133]
:INPUT ACCEPT [3911:377509]
:OUTPUT ACCEPT [536:37403]
:POSTROUTING ACCEPT [657:46151]
:ISTIO_INBOUND - [0:0]
:ISTIO_IN_REDIRECT - [0:0]
:ISTIO_OUTPUT - [0:0]
:ISTIO_REDIRECT - [0:0]
-A PREROUTING -p tcp -j ISTIO_INBOUND
-A OUTPUT -d 192.168.3.58/32 -p udp -m udp --dport 53 -j REDIRECT --to-ports 15053
-A OUTPUT -p tcp -j ISTIO_OUTPUT
-A OUTPUT -p udp -m udp --dport 53 -m owner --uid-owner 123 -j RETURN
-A OUTPUT -p udp -m udp --dport 53 -m owner --gid-owner 123 -j RETURN
-A OUTPUT -d 127.0.0.53/32 -p udp -m udp --dport 53 -j REDIRECT --to-ports 15053
-A ISTIO_INBOUND -p tcp -m tcp --dport 15008 -j RETURN
-A ISTIO_INBOUND -p tcp -m tcp --dport 22 -j RETURN
-A ISTIO_INBOUND -p tcp -m tcp --dport 15090 -j RETURN
-A ISTIO_INBOUND -p tcp -m tcp --dport 15021 -j RETURN
-A ISTIO_INBOUND -p tcp -j ISTIO_IN_REDIRECT
-A ISTIO_IN_REDIRECT -p tcp -j REDIRECT --to-ports 15006
-A ISTIO_OUTPUT -s 127.0.0.6/32 -o lo -j RETURN
-A ISTIO_OUTPUT ! -d 127.0.0.1/32 -o lo -p tcp -m tcp ! --dport 53 -m owner --uid-owner 123 -j ISTIO_IN_REDIRECT
-A ISTIO_OUTPUT -o lo -p tcp -m tcp ! --dport 53 -m owner ! --uid-owner 123 -j RETURN
-A ISTIO_OUTPUT -m owner --uid-owner 123 -j RETURN
-A ISTIO_OUTPUT ! -d 127.0.0.1/32 -o lo -m owner --gid-owner 123 -j ISTIO_IN_REDIRECT
-A ISTIO_OUTPUT -o lo -p tcp -m tcp ! --dport 53 -m owner ! --gid-owner 123 -j RETURN
-A ISTIO_OUTPUT -m owner --gid-owner 123 -j RETURN
-A ISTIO_OUTPUT -d 127.0.0.53/32 -p tcp -m tcp --dport 53 -j REDIRECT --to-ports 15053
-A ISTIO_OUTPUT -d 127.0.0.1/32 -j RETURN
-A ISTIO_OUTPUT -j ISTIO_REDIRECT
-A ISTIO_REDIRECT -p tcp -j REDIRECT --to-ports 15001
COMMIT
# Completed on Wed Aug 4 17:21:36 2021如果你对这篇内容有疑问，欢迎到本站社区发帖提问 参与讨论，获取更多帮助，或者扫码二维码加入 Web 技术交流群。
绑定邮箱获取回复消息
由于您还没有绑定你的真实邮箱，如果其他用户或者作者回复了您的评论，将不能在第一时间通知您！
发布评论
评论（2）
手动在vm1 上添加了一条路由规则可以了，但按文档介绍会自动更新这个规则的，并不需要手动添加。
将网络段 10.244.0.0/16 的所有请求转到 192.168.3.58 ip
这个网段是创建k8s是指定的pod的cird
看起来像是网络问题，在master和vm上trace一下路由是否有什么不一样的地方？