Issue with k8s and IPv6 environment

[lanzhiwang]

Original discussion here: https://groups.google.com/g/rabbitmq-users/c/6RJ0XOq7Pug

I want to verify that rabbimq works properly on ipv6 network protocol stack.
I'm k8s v1.23.8 deployed rabbimq cluster operator (https://github.com/rabbitmq/cluster-operator/releases/tag/v2.1.0),
Then deploy rabbitmq:3.10.2-management, which is the default for operator.
By default rabbitmq works properly

[middleware-develop-6852]$ kubectl -n rabbitmq-system get pods
NAME                                         READY   STATUS    RESTARTS   AGE
my-rabbitmq-server-0                         1/1     Running   0          37m
my-rabbitmq-server-1                         1/1     Running   0          37m
my-rabbitmq-server-2                         1/1     Running   0          37m

I checked the rabbitmq configuration and the DNS related configuration as follows

[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- ls -al /etc/rabbitmq
total 4
drwxrwxrwx 1 rabbitmq rabbitmq  20 May 20  2022 .
drwxr-xr-x 1 root     root      38 Jan  7 12:13 ..
drwxrwxrwx 1 rabbitmq rabbitmq 106 Jan  7 12:13 conf.d
-rw-r--r-- 1 root     root      43 May 20  2022 enabled_plugins

[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- ls -al /etc/rabbitmq/conf.d
total 16
drwxrwxrwx 1 rabbitmq rabbitmq 106 Jan  7 12:13 .
drwxrwxrwx 1 rabbitmq rabbitmq  20 May 20  2022 ..
-rw-rw-r-- 1 rabbitmq rabbitmq 491 May 20  2022 10-defaults.conf
-rw-r--r-- 1 root     root     459 Jan  7 12:12 10-operatorDefaults.conf
-rw-r--r-- 1 root     root      96 Jan  7 12:12 11-default_user.conf
-rw-r--r-- 1 root     root      51 Jan  7 12:12 90-userDefinedConfiguration.conf

[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- cat /etc/rabbitmq/enabled_plugins
[rabbitmq_management,rabbitmq_prometheus].

[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- cat /etc/rabbitmq/conf.d/10-defaults.conf
## DEFAULT SETTINGS ARE NOT MEANT TO BE TAKEN STRAIGHT INTO PRODUCTION
## see https://www.rabbitmq.com/configure.html for further information
## on configuring RabbitMQ

## allow access to the guest user from anywhere on the network
## https://www.rabbitmq.com/access-control.html#loopback-users
## https://www.rabbitmq.com/production-checklist.html#users
loopback_users.guest = false

## Send all logs to stdout/TTY. Necessary to see logs when running via
## a container
log.console = true

[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- cat /etc/rabbitmq/conf.d/10-operatorDefaults.conf
queue_master_locator                       = min-masters
disk_free_limit.absolute                   = 2GB
cluster_partition_handling                 = pause_minority
cluster_formation.peer_discovery_backend   = rabbit_peer_discovery_k8s
cluster_formation.k8s.host                 = kubernetes.default
cluster_formation.k8s.address_type         = hostname
cluster_formation.target_cluster_size_hint = 3
cluster_name                               = my-rabbitmq

[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- cat /etc/rabbitmq/conf.d/11-default_user.conf
default_user = default_user_HHzAT0LOPmQrBZDDbtK
default_pass = rtfO1IwUgiaNHG-LPe0r1inbNlZ9WJu9

[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- cat /etc/rabbitmq/conf.d/90-userDefinedConfiguration.conf
total_memory_available_override_value = 1717986919

[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- cat /etc/hosts
# Kubernetes-managed hosts file.
127.0.0.1	localhost
::1	localhost ip6-localhost ip6-loopback
fe00::0	ip6-localnet
fe00::0	ip6-mcastprefix
fe00::1	ip6-allnodes
fe00::2	ip6-allrouters
10.3.203.129	my-rabbitmq-server-0.my-rabbitmq-nodes.rabbitmq-system.svc.cluster.local	my-rabbitmq-server-0
fd00:10:16::cb81	my-rabbitmq-server-0.my-rabbitmq-nodes.rabbitmq-system.svc.cluster.local	my-rabbitmq-server-0

[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- cat /etc/resolv.conf
nameserver 10.4.0.10
search rabbitmq-system.svc.cluster.local svc.cluster.local cluster.local
options ndots:5

[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- hostname
my-rabbitmq-server-0

[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- nslookup my-rabbitmq-server-0.my-rabbitmq-nodes.rabbitmq-system.svc.cluster.local
Server:		10.4.0.10
Address:	10.4.0.10#53

Name:	my-rabbitmq-server-0.my-rabbitmq-nodes.rabbitmq-system.svc.cluster.local
Address: 10.3.203.129

[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- nslookup my-rabbitmq-server-0.my-rabbitmq-nodes
Server:		10.4.0.10
Address:	10.4.0.10#53

Name:	my-rabbitmq-server-0.my-rabbitmq-nodes.rabbitmq-system.svc.cluster.local
Address: 10.3.203.129

[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- nslookup my-rabbitmq-server-0
Server:		10.4.0.10
Address:	10.4.0.10#53

** server can't find my-rabbitmq-server-0: NXDOMAIN

command terminated with exit code 1

[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- rabbitmqctl eval 'inet:gethostbyname("my-rabbitmq-server-0.my-rabbitmq-nodes.rabbitmq-system.svc.cluster.local", inet).'
{ok,{hostent,"my-rabbitmq-server-0.my-rabbitmq-nodes.rabbitmq-system.svc.cluster.local",
             ["my-rabbitmq-server-0"],
             inet,4,
             [{10,3,203,129}]}}

[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- rabbitmqctl eval 'inet:gethostbyname("my-rabbitmq-server-0.my-rabbitmq-nodes", inet).'
{ok,{hostent,"my-rabbitmq-server-0.my-rabbitmq-nodes.rabbitmq-system.svc.cluster.local",
             [],inet,4,
             [{10,3,203,129}]}}

[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- rabbitmqctl eval 'inet:gethostbyname("my-rabbitmq-server-0", inet).'
{ok,{hostent,"my-rabbitmq-server-0.my-rabbitmq-nodes.rabbitmq-system.svc.cluster.local",
             ["my-rabbitmq-server-0"],
             inet,4,
             [{10,3,203,129}]}}

[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- rabbitmqctl eval 'inet:gethostbyname("my-rabbitmq-server-0.my-rabbitmq-nodes.rabbitmq-system.svc.cluster.local", inet6).'
{ok,{hostent,"my-rabbitmq-server-0.my-rabbitmq-nodes.rabbitmq-system.svc.cluster.local",
             [],inet6,16,
             [{64768,16,22,0,0,0,0,52097}]}}

[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- rabbitmqctl eval 'inet:gethostbyname("my-rabbitmq-server-0.my-rabbitmq-nodes", inet6).'
{error,nxdomain}

[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- rabbitmqctl eval 'inet:gethostbyname("my-rabbitmq-server-0", inet6).'
{ok,{hostent,"my-rabbitmq-server-0.my-rabbitmq-nodes.rabbitmq-system.svc.cluster.local",
             [],inet6,16,
             [{64768,16,22,0,0,0,0,52097}]}}

[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- rabbitmqctl eval 'inet_res:resolve("my-rabbitmq-server-0.my-rabbitmq-nodes.rabbitmq-system.svc.cluster.local", in, a).'
{ok,{dns_rec,{dns_header,2,true,query,true,false,true,false,false,0},
             [{dns_query,"my-rabbitmq-server-0.my-rabbitmq-nodes.rabbitmq-system.svc.cluster.local",
                         a,in,false}],
             [{dns_rr,"my-rabbitmq-server-0.my-rabbitmq-nodes.rabbitmq-system.svc.cluster.local",
                      a,in,0,21,
                      {10,3,203,129},
                      undefined,[],false}],
             [],[]}}

[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- rabbitmqctl eval 'inet_res:resolve("my-rabbitmq-server-0.my-rabbitmq-nodes", in, a).'
{error,nxdomain}

[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- rabbitmqctl eval 'inet_res:resolve("my-rabbitmq-server-0", in, a).'
{error,nxdomain}

[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- rabbitmqctl eval 'inet_res:resolve("my-rabbitmq-server-0.my-rabbitmq-nodes.rabbitmq-system.svc.cluster.local", in, aaaa).'
{ok,{dns_rec,{dns_header,5,true,query,true,false,true,false,false,0},
             [{dns_query,"my-rabbitmq-server-0.my-rabbitmq-nodes.rabbitmq-system.svc.cluster.local",
                         aaaa,in,false}],
             [],
             [{dns_rr,"cluster.local",soa,in,0,30,
                      {"ns.dns.cluster.local","hostmaster.cluster.local",
                       1673094992,7200,1800,86400,30},
                      undefined,[],false}],
             []}}

[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- rabbitmqctl eval 'inet_res:resolve("my-rabbitmq-server-0.my-rabbitmq-nodes", in, aaaa).'
{error,nxdomain}

[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- rabbitmqctl eval 'inet_res:resolve("my-rabbitmq-server-0", in, aaaa).'
{error,nxdomain}

---

The second verification is that the /etc/rabbitmq-erl_inetrc file is added to the image,
but is not configured to enable ipv6

[middleware-develop-6852]$ cat /etc/rabbitmq/erl_inetrc
{inet6, true}.
{resolv_conf, "/etc/resolv.conf"}.
{lookup, [dns]}.

rabbitmq will also work, and the rest of the configuration is the same as before

[middleware-develop-6852]$ kubectl -n rabbitmq-system get pods
NAME                                         READY   STATUS    RESTARTS   AGE
my-rabbitmq-server-0                         1/1     Running   0          104s
my-rabbitmq-server-1                         1/1     Running   0          104s
my-rabbitmq-server-2                         1/1     Running   0          104s

[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- cat /etc/hosts
# Kubernetes-managed hosts file.
127.0.0.1	localhost
::1	localhost ip6-localhost ip6-loopback
fe00::0	ip6-localnet
fe00::0	ip6-mcastprefix
fe00::1	ip6-allnodes
fe00::2	ip6-allrouters
10.3.203.202	my-rabbitmq-server-0.my-rabbitmq-nodes.rabbitmq-system.svc.cluster.local	my-rabbitmq-server-0
fd00:10:16::cbca	my-rabbitmq-server-0.my-rabbitmq-nodes.rabbitmq-system.svc.cluster.local	my-rabbitmq-server-0

[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- cat /etc/resolv.conf
nameserver 10.4.0.10
search rabbitmq-system.svc.cluster.local svc.cluster.local cluster.local
options ndots:5

[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- hostname
my-rabbitmq-server-0

[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- nslookup my-rabbitmq-server-0.my-rabbitmq-nodes.rabbitmq-system.svc.cluster.local
Server:		10.4.0.10
Address:	10.4.0.10#53

Name:	my-rabbitmq-server-0.my-rabbitmq-nodes.rabbitmq-system.svc.cluster.local
Address: 10.3.203.202

[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- nslookup my-rabbitmq-server-0.my-rabbitmq-nodes
Server:		10.4.0.10
Address:	10.4.0.10#53

Name:	my-rabbitmq-server-0.my-rabbitmq-nodes.rabbitmq-system.svc.cluster.local
Address: 10.3.203.202

[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- nslookup my-rabbitmq-server-0
Server:		10.4.0.10
Address:	10.4.0.10#53

** server can't find my-rabbitmq-server-0: NXDOMAIN

command terminated with exit code 1

[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- rabbitmqctl eval 'inet:gethostbyname("my-rabbitmq-server-0.my-rabbitmq-nodes.rabbitmq-system.svc.cluster.local", inet).'
{ok,{hostent,"my-rabbitmq-server-0.my-rabbitmq-nodes.rabbitmq-system.svc.cluster.local",
             ["my-rabbitmq-server-0"],
             inet,4,
             [{10,3,203,202}]}}

[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- rabbitmqctl eval 'inet:gethostbyname("my-rabbitmq-server-0.my-rabbitmq-nodes", inet).'
{ok,{hostent,"my-rabbitmq-server-0.my-rabbitmq-nodes.rabbitmq-system.svc.cluster.local",
             [],inet,4,
             [{10,3,203,202}]}}

[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- rabbitmqctl eval 'inet:gethostbyname("my-rabbitmq-server-0", inet).'
{ok,{hostent,"my-rabbitmq-server-0.my-rabbitmq-nodes.rabbitmq-system.svc.cluster.local",
             ["my-rabbitmq-server-0"],
             inet,4,
             [{10,3,203,202}]}}

[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- rabbitmqctl eval 'inet:gethostbyname("my-rabbitmq-server-0.my-rabbitmq-nodes.rabbitmq-system.svc.cluster.local", inet6).'
{ok,{hostent,"my-rabbitmq-server-0.my-rabbitmq-nodes.rabbitmq-system.svc.cluster.local",
             [],inet6,16,
             [{64768,16,22,0,0,0,0,52170}]}}

[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- rabbitmqctl eval 'inet:gethostbyname("my-rabbitmq-server-0.my-rabbitmq-nodes", inet6).'
{error,nxdomain}

[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- rabbitmqctl eval 'inet:gethostbyname("my-rabbitmq-server-0", inet6).'
{ok,{hostent,"my-rabbitmq-server-0.my-rabbitmq-nodes.rabbitmq-system.svc.cluster.local",
             [],inet6,16,
             [{64768,16,22,0,0,0,0,52170}]}}

[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- rabbitmqctl eval 'inet_res:resolve("my-rabbitmq-server-0.my-rabbitmq-nodes.rabbitmq-system.svc.cluster.local", in, a).'
{ok,{dns_rec,{dns_header,1,true,query,true,false,true,false,false,0},
             [{dns_query,"my-rabbitmq-server-0.my-rabbitmq-nodes.rabbitmq-system.svc.cluster.local",
                         a,in,false}],
             [{dns_rr,"my-rabbitmq-server-0.my-rabbitmq-nodes.rabbitmq-system.svc.cluster.local",
                      a,in,0,30,
                      {10,3,203,202},
                      undefined,[],false}],
             [],[]}}

[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- rabbitmqctl eval 'inet_res:resolve("my-rabbitmq-server-0.my-rabbitmq-nodes", in, a).'
{error,nxdomain}

[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- rabbitmqctl eval 'inet_res:resolve("my-rabbitmq-server-0", in, a).'
{error,nxdomain}

[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- rabbitmqctl eval 'inet_res:resolve("my-rabbitmq-server-0.my-rabbitmq-nodes.rabbitmq-system.svc.cluster.local", in, aaaa).'
{ok,{dns_rec,{dns_header,4,true,query,true,false,true,false,false,0},
             [{dns_query,"my-rabbitmq-server-0.my-rabbitmq-nodes.rabbitmq-system.svc.cluster.local",
                         aaaa,in,false}],
             [],
             [{dns_rr,"cluster.local",soa,in,0,30,
                      {"ns.dns.cluster.local","hostmaster.cluster.local",
                       1673097281,7200,1800,86400,30},
                      undefined,[],false}],
             []}}

[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- rabbitmqctl eval 'inet_res:resolve("my-rabbitmq-server-0.my-rabbitmq-nodes", in, aaaa).'
{error,nxdomain}

[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- rabbitmqctl eval 'inet_res:resolve("my-rabbitmq-server-0", in, aaaa).'
{error,nxdomain}

---

The third verification is that you are adding the /etc/rabbitmq-erl_inetrc file to the image and enabling ipv6

In this case, rabbitmq does not work properly. The rabbitmq logs are as follows：

2023-01-07 13:30:08.123689+00:00 [info] <0.219.0> Configured peer discovery backend: rabbit_peer_discovery_k8s
2023-01-07 13:30:08.124245+00:00 [info] <0.219.0> Will try to lock with peer discovery backend rabbit_peer_discovery_k8s
2023-01-07 13:30:08.127846+00:00 [warning] <0.315.0> Description: "Authenticity is not established by certificate path validation"
2023-01-07 13:30:08.127846+00:00 [warning] <0.315.0>      Reason: "Option {verify, verify_peer} and cacertfile/cacerts is missing"
2023-01-07 13:30:08.127846+00:00 [warning] <0.315.0>
2023-01-07 13:30:08.185568+00:00 [error] <0.219.0> Failed to fetch a list of nodes from Kubernetes API: {failed_connect,[{to_address,{"kubernetes.default",443}},
2023-01-07 13:30:08.185568+00:00 [error] <0.219.0>                  {inet6,[inet6],nxdomain}]}
2023-01-07 13:30:08.188142+00:00 [warning] <0.316.0> Description: "Authenticity is not established by certificate path validation"
2023-01-07 13:30:08.188142+00:00 [warning] <0.316.0>      Reason: "Option {verify, verify_peer} and cacertfile/cacerts is missing"
2023-01-07 13:30:08.188142+00:00 [warning] <0.316.0>
2023-01-07 13:30:08.219100+00:00 [error] <0.219.0> Failed to lock with peer discovery backend rabbit_peer_discovery_k8s: "{failed_connect,[{to_address,{\"kubernetes.default\",443}},\n                 {inet6,[inet6],nxdomain}]}"
2023-01-07 13:30:38.220353+00:00 [info] <0.219.0> Will try to lock with peer discovery backend rabbit_peer_discovery_k8s
2023-01-07 13:30:38.223474+00:00 [warning] <0.365.0> Description: "Authenticity is not established by certificate path validation"
2023-01-07 13:30:38.223474+00:00 [warning] <0.365.0>      Reason: "Option {verify, verify_peer} and cacertfile/cacerts is missing"
2023-01-07 13:30:38.223474+00:00 [warning] <0.365.0>
2023-01-07 13:30:38.259894+00:00 [error] <0.219.0> Failed to fetch a list of nodes from Kubernetes API: {failed_connect,[{to_address,{"kubernetes.default",443}},
2023-01-07 13:30:38.259894+00:00 [error] <0.219.0>                  {inet6,[inet6],nxdomain}]}
2023-01-07 13:30:38.262612+00:00 [warning] <0.366.0> Description: "Authenticity is not established by certificate path validation"
2023-01-07 13:30:38.262612+00:00 [warning] <0.366.0>      Reason: "Option {verify, verify_peer} and cacertfile/cacerts is missing"
2023-01-07 13:30:38.262612+00:00 [warning] <0.366.0>
2023-01-07 13:30:38.301716+00:00 [error] <0.219.0> Failed to lock with peer discovery backend rabbit_peer_discovery_k8s: "{failed_connect,[{to_address,{\"kubernetes.default\",443}},\n                 {inet6,[inet6],nxdomain}]}"
2023-01-07 13:31:08.302540+00:00 [info] <0.219.0> Will try to lock with peer discovery backend rabbit_peer_discovery_k8s
2023-01-07 13:31:08.308443+00:00 [warning] <0.415.0> Description: "Authenticity is not established by certificate path validation"
2023-01-07 13:31:08.308443+00:00 [warning] <0.415.0>      Reason: "Option {verify, verify_peer} and cacertfile/cacerts is missing"
2023-01-07 13:31:08.308443+00:00 [warning] <0.415.0>
2023-01-07 13:31:08.345506+00:00 [error] <0.219.0> Failed to fetch a list of nodes from Kubernetes API: {failed_connect,[{to_address,{"kubernetes.default",443}},
2023-01-07 13:31:08.345506+00:00 [error] <0.219.0>                  {inet6,[inet6],nxdomain}]}
2023-01-07 13:31:08.350786+00:00 [warning] <0.416.0> Description: "Authenticity is not established by certificate path validation"
2023-01-07 13:31:08.350786+00:00 [warning] <0.416.0>      Reason: "Option {verify, verify_peer} and cacertfile/cacerts is missing"
2023-01-07 13:31:08.350786+00:00 [warning] <0.416.0>
2023-01-07 13:31:08.387389+00:00 [error] <0.219.0> Failed to lock with peer discovery backend rabbit_peer_discovery_k8s: "{failed_connect,[{to_address,{\"kubernetes.default\",443}},\n                 {inet6,[inet6],nxdomain}]}"

[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- ls -al /etc/rabbitmq
total 12
drwxrwxrwx 1 rabbitmq rabbitmq  45 Jan  7 13:29 .
drwxr-xr-x 1 root     root      38 Jan  7 13:29 ..
drwxrwxrwx 1 rabbitmq rabbitmq 106 Jan  7 13:29 conf.d
-rw-r--r-- 1 root     root      43 May 20  2022 enabled_plugins
-rw-r--r-- 1 rabbitmq rabbitmq  67 Jan  7 11:05 erl_inetrc
-rw-r--r-- 1 root     root     148 Jan  7 13:29 rabbitmq-env.conf

[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- cat /etc/rabbitmq/erl_inetrc
{inet6, true}.
{resolv_conf, "/etc/resolv.conf"}.
{lookup, [dns]}.

[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- cat /etc/rabbitmq/rabbitmq-env.conf
RABBITMQ_SERVER_ADDITIONAL_ERL_ARGS="-kernel inetrc '/etc/rabbitmq/erl_inetrc' -proto_dist inet6_tcp"
RABBITMQ_CTL_ERL_ARGS="-proto_dist inet6_tcp"

[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- cat /etc/hosts
# Kubernetes-managed hosts file.
127.0.0.1	localhost
::1	localhost ip6-localhost ip6-loopback
fe00::0	ip6-localnet
fe00::0	ip6-mcastprefix
fe00::1	ip6-allnodes
fe00::2	ip6-allrouters
10.3.203.238	my-rabbitmq-server-0.my-rabbitmq-nodes.rabbitmq-system.svc.cluster.local	my-rabbitmq-server-0
fd00:10:16::cbee	my-rabbitmq-server-0.my-rabbitmq-nodes.rabbitmq-system.svc.cluster.local	my-rabbitmq-server-0

[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- cat /etc/resolv.conf
nameserver 10.4.0.10
search rabbitmq-system.svc.cluster.local svc.cluster.local cluster.local
options ndots:5

[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- hostname
my-rabbitmq-server-0

[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- nslookup my-rabbitmq-server-0.my-rabbitmq-nodes.rabbitmq-system.svc.cluster.local
Server:		10.4.0.10
Address:	10.4.0.10#53

Name:	my-rabbitmq-server-0.my-rabbitmq-nodes.rabbitmq-system.svc.cluster.local
Address: fd00:10:16::cbee

[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- nslookup my-rabbitmq-server-0.my-rabbitmq-nodes
Server:		10.4.0.10
Address:	10.4.0.10#53

Name:	my-rabbitmq-server-0.my-rabbitmq-nodes.rabbitmq-system.svc.cluster.local
Address: fd00:10:16::cbee

[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- nslookup my-rabbitmq-server-0
Server:		10.4.0.10
Address:	10.4.0.10#53

** server can't find my-rabbitmq-server-0: NXDOMAIN

command terminated with exit code 1

[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- rabbitmqctl eval 'inet:gethostbyname("my-rabbitmq-server-0.my-rabbitmq-nodes.rabbitmq-system.svc.cluster.local", inet).'
{ok,{hostent,"my-rabbitmq-server-0.my-rabbitmq-nodes.rabbitmq-system.svc.cluster.local",
             [],inet,4,
             [{127,0,0,1}]}}

[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- rabbitmqctl eval 'inet:gethostbyname("my-rabbitmq-server-0.my-rabbitmq-nodes", inet).'
{error,nxdomain}

[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- rabbitmqctl eval 'inet:gethostbyname("my-rabbitmq-server-0", inet).'
{ok,{hostent,"my-rabbitmq-server-0",[],inet,4,[{127,0,0,1}]}}

[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- rabbitmqctl eval 'inet:gethostbyname("my-rabbitmq-server-0.my-rabbitmq-nodes.rabbitmq-system.svc.cluster.local", inet6).'
{ok,{hostent,"my-rabbitmq-server-0.my-rabbitmq-nodes.rabbitmq-system.svc.cluster.local",
             [],inet6,16,
             [{64768,16,22,0,0,0,0,52206}]}}

[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- rabbitmqctl eval 'inet:gethostbyname("my-rabbitmq-server-0.my-rabbitmq-nodes", inet6).'
{ok,{hostent,"my-rabbitmq-server-0.my-rabbitmq-nodes.rabbitmq-system.svc.cluster.local",
             [],inet6,16,
             [{64768,16,22,0,0,0,0,52206}]}}

[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- rabbitmqctl eval 'inet:gethostbyname("my-rabbitmq-server-0", inet6).'
{ok,{hostent,"my-rabbitmq-server-0",[],inet6,16,[{0,0,0,0,0,0,0,1}]}}

[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- rabbitmqctl eval 'inet_res:resolve("my-rabbitmq-server-0.my-rabbitmq-nodes.rabbitmq-system.svc.cluster.local", in, a).'
{ok,{dns_rec,{dns_header,5,true,query,true,false,true,false,false,0},
             [{dns_query,"my-rabbitmq-server-0.my-rabbitmq-nodes.rabbitmq-system.svc.cluster.local",
                         a,in,false}],
             [],
             [{dns_rr,"cluster.local",soa,in,0,30,
                      {"ns.dns.cluster.local","hostmaster.cluster.local",
                       1673098842,7200,1800,86400,30},
                      undefined,[],false}],
             []}}

[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- rabbitmqctl eval 'inet_res:resolve("my-rabbitmq-server-0.my-rabbitmq-nodes", in, a).'
{error,nxdomain}

[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- rabbitmqctl eval 'inet_res:resolve("my-rabbitmq-server-0", in, a).'
{error,nxdomain}

[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- rabbitmqctl eval 'inet_res:resolve("my-rabbitmq-server-0.my-rabbitmq-nodes.rabbitmq-system.svc.cluster.local", in, aaaa).'
{ok,{dns_rec,{dns_header,24,true,query,true,false,true,false,false,0},
             [{dns_query,"my-rabbitmq-server-0.my-rabbitmq-nodes.rabbitmq-system.svc.cluster.local",
                         aaaa,in,false}],
             [{dns_rr,"my-rabbitmq-server-0.my-rabbitmq-nodes.rabbitmq-system.svc.cluster.local",
                      aaaa,in,0,30,
                      {64768,16,22,0,0,0,0,52206},
                      undefined,[],false}],
             [],[]}}

[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- rabbitmqctl eval 'inet_res:resolve("my-rabbitmq-server-0.my-rabbitmq-nodes", in, aaaa).'
{error,nxdomain}

[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- rabbitmqctl eval 'inet_res:resolve("my-rabbitmq-server-0", in, aaaa).'
{error,nxdomain}

What is the problem and how should I solve it? Or what verification do I need to do

[lukebakken]

Please note that I edited your question to use GitHub's <details> feature to make it much easier to read. Another alternative is to attach output to files and add the files to your questions/comments. Please do NOT paste large amounts of text.

---

This command fails:

[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- nslookup my-rabbitmq-server-0
Server:		10.4.0.10
Address:	10.4.0.10#53

** server can't find my-rabbitmq-server-0: NXDOMAIN

Since that fails, it means that DNS resolution for short node names (names like my-rabbitmq-server-0) will fail. I'm assuming you have your Erlang nodes set up to use short node names, so when Erlang starts it does not know how to look node names up in DNS.

It looks like the full host name is my-rabbitmq-server-0.my-rabbitmq-nodes.rabbitmq-system.svc.cluster.local, but your search domains do not include my-rabbitmq-nodes:

[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- cat /etc/resolv.conf
nameserver 10.4.0.10
search rabbitmq-system.svc.cluster.local svc.cluster.local cluster.local
options ndots:5

You should start by fixing your search domains so that the nslookup my-rabbitmq-server-0 command succeeds.

[lanzhiwang]

my-rabbitmq-server-0 does not resolve because of the k8s configuration, as long as you add the relevant entry in the /etc/resolv.conf file will work properly

[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- cat /etc/resolv.conf
nameserver 10.4.0.10
search rabbitmq-system.svc.cluster.local svc.cluster.local cluster.local my-rabbitmq-nodes.rabbitmq-system.svc.cluster.local
options ndots:5


[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- nslookup my-rabbitmq-server-0
Server:		10.4.0.10
Address:	10.4.0.10#53

Name:	my-rabbitmq-server-0.my-rabbitmq-nodes.rabbitmq-system.svc.cluster.local
Address: fd00:10:16::d844

rabbitmq still has the same problem

2023-01-09 02:09:34.281352+00:00 [warning] <0.314.0> Description: "Authenticity is not established by certificate path validation"
2023-01-09 02:09:34.281352+00:00 [warning] <0.314.0>      Reason: "Option {verify, verify_peer} and cacertfile/cacerts is missing"
2023-01-09 02:09:34.281352+00:00 [warning] <0.314.0>
2023-01-09 02:09:34.323661+00:00 [error] <0.219.0> Failed to fetch a list of nodes from Kubernetes API: {failed_connect,[{to_address,{"kubernetes.default",443}},
2023-01-09 02:09:34.323661+00:00 [error] <0.219.0>                  {inet6,[inet6],nxdomain}]}
2023-01-09 02:09:34.326403+00:00 [warning] <0.315.0> Description: "Authenticity is not established by certificate path validation"
2023-01-09 02:09:34.326403+00:00 [warning] <0.315.0>      Reason: "Option {verify, verify_peer} and cacertfile/cacerts is missing"
2023-01-09 02:09:34.326403+00:00 [warning] <0.315.0>
2023-01-09 02:09:34.365858+00:00 [error] <0.219.0> Failed to lock with peer discovery backend rabbit_peer_discovery_k8s: "{failed_connect,[{to_address,{\"kubernetes.default\",443}},\n                 {inet6,[inet6],nxdomain}]}"
2023-01-09 02:10:04.367376+00:00 [info] <0.219.0> Will try to lock with peer discovery backend rabbit_peer_discovery_k8s
2023-01-09 02:10:04.370728+00:00 [warning] <0.364.0> Description: "Authenticity is not established by certificate path validation"
2023-01-09 02:10:04.370728+00:00 [warning] <0.364.0>      Reason: "Option {verify, verify_peer} and cacertfile/cacerts is missing"
2023-01-09 02:10:04.370728+00:00 [warning] <0.364.0>
2023-01-09 02:10:04.406398+00:00 [error] <0.219.0> Failed to fetch a list of nodes from Kubernetes API: {failed_connect,[{to_address,{"kubernetes.default",443}},
2023-01-09 02:10:04.406398+00:00 [error] <0.219.0>                  {inet6,[inet6],nxdomain}]}
2023-01-09 02:10:04.408751+00:00 [warning] <0.365.0> Description: "Authenticity is not established by certificate path validation"
2023-01-09 02:10:04.408751+00:00 [warning] <0.365.0>      Reason: "Option {verify, verify_peer} and cacertfile/cacerts is missing"
2023-01-09 02:10:04.408751+00:00 [warning] <0.365.0>
2023-01-09 02:10:04.436222+00:00 [error] <0.219.0> Failed to lock with peer discovery backend rabbit_peer_discovery_k8s: "{failed_connect,[{to_address,{\"kubernetes.default\",443}},\n                 {inet6,[inet6],nxdomain}]}"
2023-01-09 02:10:34.437378+00:00 [info] <0.219.0> Will try to lock with peer discovery backend rabbit_peer_discovery_k8s

[lanzhiwang]

The k8s api also works

[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- curl --header "Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IjFDNWJySzRnc0dVdFF1V2FzTTNrN2doM0R1d1lnOHZUamhrVzhFUVdVM3cifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJyYWJiaXRtcS1zeXN0ZW0iLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlY3JldC5uYW1lIjoibXktcmFiYml0bXEtc2VydmVyLXRva2VuLXg0Y3ZrIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQubmFtZSI6Im15LXJhYmJpdG1xLXNlcnZlciIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6ImNjNDExNzZlLTA0YmItNGU0NC04MTdkLWMyMjQyNmUwOTI4OSIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDpyYWJiaXRtcS1zeXN0ZW06bXktcmFiYml0bXEtc2VydmVyIn0.BY0rSJmtCnC-XP5PZFTJuzDEIOGlvN4urT7GYJwejKm4VO6PjSyDYldRJreDCfCm9KOhK64Il1yxwZOyrRFJlRzNe_piK09APygXOhejMAnQ2TMOs9e7k9DJzU_nAoMc_C115rSd9TG2l-6ZrLzVsTsuR0pAOQilc0LCTXdxoPww2K36tzD03_PSTSNonfXJ1pYUFeIrIMybZfFfkFjJS9Zy0O3gBqgpBo14r0Cp8ZsfvTWFci7NAYvr18J2MksNSor21YAEnjs5IkniCWwDJAtHyf0eRWsw0OVBuiUEf0uOzH9gz5Ue13uobtWJJIvJucs6uVYNQ8L-dtvCGKWJIw" --insecure -X GET https://kubernetes.default:443/api/v1/namespaces/rabbitmq-system/endpoints/my-rabbitmq-nodes
{
  "kind": "Endpoints",
  "apiVersion": "v1",
  "metadata": {
    "name": "my-rabbitmq-nodes",
    "namespace": "rabbitmq-system",
    }
  },
  "subsets": [
    {
      "addresses": [
        {
          "ip": "fd00:10:16::d844",
          "hostname": "my-rabbitmq-server-0",
          "nodeName": "192.168.132.210",
          "targetRef": {
            "kind": "Pod",
            "namespace": "rabbitmq-system",
            "name": "my-rabbitmq-server-0",
            "uid": "fb84dc20-bc39-44fa-a299-f6a6b6be8ef3",
            "resourceVersion": "268988988"
          }
        },
        {
          "ip": "fd00:10:16::d845",
          "hostname": "my-rabbitmq-server-1",
          "nodeName": "192.168.132.210",
          "targetRef": {
            "kind": "Pod",
            "namespace": "rabbitmq-system",
            "name": "my-rabbitmq-server-1",
            "uid": "345159da-ae6b-439e-9739-7b7f9d818e5b",
            "resourceVersion": "268989059"
          }
        },
        {
          "ip": "fd00:10:16::d846",
          "hostname": "my-rabbitmq-server-2",
          "nodeName": "192.168.132.210",
          "targetRef": {
            "kind": "Pod",
            "namespace": "rabbitmq-system",
            "name": "my-rabbitmq-server-2",
            "uid": "7594db95-a55c-447e-a198-30f4cb035905",
            "resourceVersion": "268989112"
          }
        }
      ],
      "ports": [
        {
          "name": "cluster-rpc",
          "port": 25672,
          "protocol": "TCP"
        },
        {
          "name": "epmd",
          "port": 4369,
          "protocol": "TCP"
        }
      ]
    }
  ]
}

[simonwu-sn]

The workaround works:

apiVersion: rabbitmq.com/v1beta1
kind: RabbitmqCluster
metadata:
  name: hello-world
spec:
  replicas: 1
  override:
    statefulSet:
      spec:
        template:
          spec:
            initContainers:
            - name: ipv6-init
              image: "docker.io/busybox:1.33.1"
              imagePullPolicy: IfNotPresent
              volumeMounts:
              - name: ipv6-cfg
                mountPath: /ipv6
              command: ['sh', '-c', 'echo "{inet6, true}." > /ipv6/erl_inetrc']
            volumes:
            - name: ipv6-cfg
              emptyDir: {}
            containers:
              - name: rabbitmq
                env:
                  - name: RABBITMQ_SERVER_ADDITIONAL_ERL_ARGS
                    value: "-kernel inetrc '/ipv6/erl_inetrc' -proto_dist inet6_tcp"
                  - name: RABBITMQ_CTL_ERL_ARGS
                    value: "-proto_dist inet6_tcp"
                volumeMounts:
                - name: ipv6-cfg
                  mountPath: /ipv6

[michaelklishin]

That configures both inter-node configuration and CLI tools to use IPv6, so seems correct to me.