求助：配置 High Availability Storage with GlusterFS 有误

Question

下面的配置是按照官方文档《High Availability Storage with GlusterFS》做的
servier端（其中的一台，其他的和这个差不多）配置如下：
volume mailspool-ds
        type storage/posix
        option directory /home/export/mailspool
end-volume

volume mailspool-ns
        type storage/posix
        option directory /home/export/mailspool-ns
end-volume

volume mailspool-santa1-ds
        type protocol/client
        option transport-type tcp/client
        option remote-host 192.168.1.81
        option remote-subvolume mailspool-ds
end-volume

volume mailspool-santa1-ns
        type protocol/client
        option transport-type tcp/client
        option remote-host 192.168.1.81
        option remote-subvolume mailspool-ns
end-volume

volume mailspool-santa2-ds
        type protocol/client
        option transport-type tcp/client
        option remote-host 192.168.1.82
        option remote-subvolume mailspool-ds
end-volume

volume mailspool-santa2-ns
        type protocol/client
        option transport-type tcp/client
        option remote-host 192.168.1.82
        option remote-subvolume mailspool-ns
end-volume

volume mailspool-santa3-ds
        type protocol/client
        option transport-type tcp/client
        option remote-host 192.168.1.83
        option remote-subvolume mailspool-ds
end-volume

volume mailspool-santa3-ns
        type protocol/client
        option transport-type tcp/client
        option remote-host 192.168.1.83
        option remote-subvolume mailspool-ns
end-volume

volume mailspool-ns-afr
        type cluster/afr
        subvolumes mailspool-ns mailspool-santa1-ns mailspool-santa2-ns mailspool-santa3-ns
        option replicate *:4
end-volume

volume mailspool-ds-afr
        type cluster/afr
        subvolumes mailspool-ds mailspool-santa1-ds mailspool-santa2-ds mailspool-santa3-ds
        option replicate *:4
end-volume

volume mailspool-unify
        type cluster/unify
        subvolumes mailspool-ds-afr
        option namespace mailspool-ns-afr
        option scheduler rr
end-volume

volume mailspool
        type performance/io-threads
        option thread-count 8
        option cache-size 64MB
        subvolumes mailspool-unify
end-volume

volume server
        type protocol/server
        option transport-type tcp/server
        subvolumes mailspool
        option auth.ip.mailspool-ds.allow 192.168.1.*,127.0.0.1
        option auth.ip.mailspool-ns.allow 192.168.1.*,127.0.0.1
        option auth.ip.mailspool.allow *
end-volume

client端配置如下：
volume santa
        type protocol/client
        option transport-type tcp/client
        option remote-host roundrobin.gluster.local
        option remote-subvolume mailspool
end-volume

volume writeback
        type performance/write-behind
        option aggregate-size 131072
        subvolumes santa
end-volume

volume readahead
        type performance/read-ahead
        option page-size 65536
        option page-count 16
        subvolumes writeback
end-volume

client 端的日志如下：
2008-02-21 17:40:29 D [common-utils.c:165:gf_resolve_ip] resolver: DNS cache not present, freshly probing hostname: roundrobin.gluster.local
2008-02-21 17:40:29 E [tcp-client.c:135:tcp_connect] santa: error: not in progress - trace: Network is unreachable
2008-02-21 17:40:30 D [tcp-client.c:70:tcp_connect] santa: socket fd = 6
2008-02-21 17:40:30 D [tcp-client.c:88:tcp_connect] santa: finalized on port `1023'
2008-02-21 17:40:30 D [tcp-client.c:109:tcp_connect] santa: defaulting remote-port to 6996
2008-02-21 17:40:30 D [common-utils.c:165:gf_resolve_ip] resolver: DNS cache not present, freshly probing hostname: roundrobin.gluster.local
2008-02-21 17:40:30 E [tcp-client.c:135:tcp_connect] santa: error: not in progress - trace: Network is unreachable
2008-02-21 17:40:30 W [client-protocol.c:343:client_protocol_xfer] santa: not connected at the moment to submit frame type(0) op(34)
2008-02-21 17:40:30 E [fuse-bridge.c:431:fuse_entry_cbk] glusterfs-fuse: 2: / => -1 (107)
2008-02-21 17:40:30 D [fuse-bridge.c:437:fuse_entry_cbk] glusterfs-fuse: unlinking stale dentry for `/'
2008-02-21 17:40:30 D [tcp-client.c:70:tcp_connect] santa: socket fd = 6
2008-02-21 17:40:30 D [tcp-client.c:88:tcp_connect] santa: finalized on port `1023'
2008-02-21 17:40:30 D [tcp-client.c:109:tcp_connect] santa: defaulting remote-port to 6996
2008-02-21 17:40:30 D [common-utils.c:165:gf_resolve_ip] resolver: DNS cache not present, freshly probing hostname: roundrobin.gluster.local
2008-02-21 17:40:30 E [tcp-client.c:135:tcp_connect] santa: error: not in progress - trace: Network is unreachable
2008-02-21 17:40:30 W [client-protocol.c:343:client_protocol_xfer] santa: not connected at the moment to submit frame type(0) op(34)
2008-02-21 17:40:30 E [fuse-bridge.c:431:fuse_entry_cbk] glusterfs-fuse: 2: / => -1 (107)
2008-02-21 17:40:30 D [inode.c:308:__destroy_inode] fuse/inode: destroy inode(0) [@0x8097f30]
2008-02-21 17:40:30 D [client-protocol.c:4462:client_protocol_reconnect] santa: attempting reconnect
2008-02-21 17:40:30 D [tcp-client.c:70:tcp_connect] santa: socket fd = 6
2008-02-21 17:40:30 D [tcp-client.c:88:tcp_connect] santa: finalized on port `1023'
2008-02-21 17:40:30 D [tcp-client.c:109:tcp_connect] santa: defaulting remote-port to 6996
2008-02-21 17:40:30 D [common-utils.c:165:gf_resolve_ip] resolver: DNS cache not present, freshly probing hostname: roundrobin.gluster.local
2008-02-21 17:40:30 E [tcp-client.c:135:tcp_connect] santa: error: not in progress - trace: Network is unreachable
2008-02-21 17:40:32 D [client-protocol.c:4462:client_protocol_reconnect] santa: attempting reconnect
2008-02-21 17:40:32 D [tcp-client.c:70:tcp_connect] santa: socket fd = 6
2008-02-21 17:40:32 D [tcp-client.c:88:tcp_connect] santa: finalized on port `1023'
2008-02-21 17:40:32 D [tcp-client.c:109:tcp_connect] santa: defaulting remote-port to 6996
2008-02-21 17:40:32 D [common-utils.c:165:gf_resolve_ip] resolver: DNS cache not present, freshly probing hostname: roundrobin.gluster.local
2008-02-21 17:40:32 E [tcp-client.c:135:tcp_connect] santa: error: not in progress - trace: Network is unreachable
2008-02-21 17:40:32 D [fuse-bridge.c:1994:fuse_statfs] glusterfs-fuse: 3: STATFS
2008-02-21 17:40:32 D [tcp-client.c:70:tcp_connect] santa: socket fd = 6
2008-02-21 17:40:32 D [tcp-client.c:88:tcp_connect] santa: finalized on port `1023'
2008-02-21 17:40:32 D [tcp-client.c:109:tcp_connect] santa: defaulting remote-port to 6996
2008-02-21 17:40:32 D [common-utils.c:165:gf_resolve_ip] resolver: DNS cache not present, freshly probing hostname: roundrobin.gluster.local
2008-02-21 17:40:32 E [tcp-client.c:135:tcp_connect] santa: error: not in progress - trace: Network is unreachable
2008-02-21 17:40:32 W [client-protocol.c:343:client_protocol_xfer] santa: not connected at the moment to submit frame type(0) op(15)
2008-02-21 17:40:32 E [fuse-bridge.c:1966:fuse_statfs_cbk] glusterfs-fuse: 3: ERR => -1 (107)
2008-02-21 17:40:35 D [client-protocol.c:4462:client_protocol_reconnect] santa: attempting reconnect
2008-02-21 17:40:35 D [tcp-client.c:70:tcp_connect] santa: socket fd = 6
2008-02-21 17:40:35 D [tcp-client.c:88:tcp_connect] santa: finalized on port `1023'
2008-02-21 17:40:35 D [tcp-client.c:109:tcp_connect] santa: defaulting remote-port to 6996
2008-02-21 17:40:35 D [common-utils.c:165:gf_resolve_ip] resolver: DNS cache not present, freshly probing hostname: roundrobin.gluster.local
2008-02-21 17:40:35 E [tcp-client.c:135:tcp_connect] santa: error: not in progress - trace: Network is unreachable
2008-02-21 17:40:40 D [client-protocol.c:4462:client_protocol_reconnect] santa: attempting reconnect
2008-02-21 17:40:40 D [tcp-client.c:70:tcp_connect] santa: socket fd = 6
2008-02-21 17:40:40 D [tcp-client.c:88:tcp_connect] santa: finalized on port `1023'
2008-02-21 17:40:40 D [tcp-client.c:109:tcp_connect] santa: defaulting remote-port to 6996
2008-02-21 17:40:40 D [common-utils.c:165:gf_resolve_ip] resolver: DNS cache not present, freshly probing hostname: roundrobin.gluster.local
2008-02-21 17:40:40 E [tcp-client.c:135:tcp_connect] santa: error: not in progress - trace: Network is unreachable
2008-02-21 17:40:43 D [fuse-bridge.c:1994:fuse_statfs] glusterfs-fuse: 4: STATFS
2008-02-21 17:40:43 D [tcp-client.c:70:tcp_connect] santa: socket fd = 6
2008-02-21 17:40:43 D [tcp-client.c:88:tcp_connect] santa: finalized on port `1023'
2008-02-21 17:40:43 D [tcp-client.c:109:tcp_connect] santa: defaulting remote-port to 6996
2008-02-21 17:40:43 D [common-utils.c:165:gf_resolve_ip] resolver: DNS cache not present, freshly probing hostname: roundrobin.gluster.local
2008-02-21 17:40:43 E [tcp-client.c:135:tcp_connect] santa: error: not in progress - trace: Network is unreachable
2008-02-21 17:40:43 W [client-protocol.c:343:client_protocol_xfer] santa: not connected at the moment to submit frame type(0) op(15)
2008-02-21 17:40:43 E [fuse-bridge.c:1966:fuse_statfs_cbk] glusterfs-fuse: 4: ERR => -1 (107)
2008-02-21 17:40:48 D [client-protocol.c:4462:client_protocol_reconnect] santa: attempting reconnect
2008-02-21 17:40:48 D [tcp-client.c:70:tcp_connect] santa: socket fd = 6
2008-02-21 17:40:48 D [tcp-client.c:88:tcp_connect] santa: finalized on port `1023'
2008-02-21 17:40:48 D [tcp-client.c:109:tcp_connect] santa: defaulting remote-port to 6996
2008-02-21 17:40:48 D [common-utils.c:165:gf_resolve_ip] resolver: DNS cache not present, freshly probing hostname: roundrobin.gluster.local
2008-02-21 17:40:49 E [tcp-client.c:135:tcp_connect] santa: error: not in progress - trace: Network is unreachable
2008-02-21 17:41:02 D [client-protocol.c:4462:client_protocol_reconnect] santa: attempting reconnect
2008-02-21 17:41:02 D [tcp-client.c:70:tcp_connect] santa: socket fd = 6
2008-02-21 17:41:02 D [tcp-client.c:88:tcp_connect] santa: finalized on port `1023'
2008-02-21 17:41:02 D [tcp-client.c:109:tcp_connect] santa: defaulting remote-port to 6996
2008-02-21 17:41:02 D [common-utils.c:165:gf_resolve_ip] resolver: DNS cache not present, freshly probing hostname: roundrobin.gluster.local

根据日志，问题应该出在client端
volume santa
        type protocol/client
        option transport-type tcp/client
        option remote-host roundrobin.gluster.local
        option remote-subvolume mailspool
end-volume

的option remote-host roundrobin.gluster.local 这一行。 这里有些不明白“roundrobin.gluster.local” 是不是某台server的名字？ 而且还要在这台server上配置DNS？

Answer 1

volume santa
        type protocol/client
        option transport-type tcp/client
        option remote-host roundrobin.gluster.local
        option remote-subvolume mailspool
end-volume

可以看出 roundrobin.gluster.local是导出mailspool 这个卷的主机名

Answer 2

但是其他三台server上 也有导出mailspool 这个卷，是不是指定任意一台就行了，还有是不是还要在这台server上配置DNS呢

Answer 3

原帖由 yman 于 2008-2-22 10:39 发表
但是其他三台server上 也有导出mailspool 这个卷，是不是指定任意一台就行了，还有是不是还要在这台server上配置DNS呢

不需要DNS，如果你主机地址都用IP来指定。

从配置文件来看，任意指定一台有导出mailspool这个卷的server IP地址就可以了，如果你想要用主机名替代IP，可以配置DNS或者/etc/hosts

Answer 4

已经改用server1的IP地址了
volume santa
        type protocol/client
        option transport-type tcp/client
        option remote-host 192.168.1.80
        option remote-subvolume mailspool
end-volume
但是还是不行，用df命令时 还是说：df: “/mnt/cluster”: 传输端点尚未连接
日志错误和上面一样

Answer 5

从你的配置文件可以看出mailspool这个卷是由好几个卷组合而成的，所以如果其中有一个卷配置有问题，就会导致挂载出现问。

还有，如果你都改了IP，日志肯定是不一样的了，请仔细分析的日志，客户端和服务端的都分析一下。glusterfs的结构不是很复杂，配置也比较简单，只要逻辑关系清楚了，不难解决问题。

Answer 6

resolver:dns cache not present, freshly probing hostname:127.0.0.1
resolver:returing ip: 127.0.0.1[0] for hostname:127.0.0.1
resolver:flushing dns cache
connect on 6 in progress(non-blocking)
non-blocking connect() returned : 111(connection refused)