db_clickhouse unreachable from gprofiler-ps-ch-rest-service

[joshuaoc-intel]

I am running into an issue where it seems like the frontend cannot connect to the backend. When I run the diagnostic I get the error:

[ERROR] container unreachable from gprofiler-ps-ch-rest-service
Error response from daemon: No such container: db_clickhouse
[ERROR] container unreachable from db_clickhouse

When I look into it further and pull the error logs I find this error for ports 8123, 9000, 9004, 9005, 9009:

Application: Listen [0.0.0.0]:8123 failed: Poco::Exception. Code: 1000, e.code() = 98, Net Exception: Address already in use: 0.0.0.0:8123 (version 22.8.21.38 (official build)). If it is an IPv6 or IPv4 address and your host has disabled IPv6 or IPv4, then consider to specify not disabled IPv4 or IPv6 address to listen in <listen_host> element of configuration file. Example for disabled IPv6: <listen_host>0.0.0.0</listen_host> . Example for disabled IPv4: <listen_host>::</listen_host>

Any insight?

[leonid-granulate]

Looks like clickhouse failed to start.
Which command you used to start the stack?
Also the commend you used to fetch logs can be useful

[joshuaoc-intel]

Used to start stack: docker-compose --profile with-clickhouse up -d --build
Used to fetch logs: docker cp gprofiler-ps-agents-logs-backend:/logs ./logs_from_container

[leonid-granulate]

Hi, tnx for your response.
Which os and arch you are running?

[joshuaoc-intel]

c5.2xl on AWS running Ubuntu 22 and 6.5 kernel

[leonid-granulate]

It looks like the problem is starting clickhouse. Let's troubleshoot:

Bring down the existing containers and remove volumes:

docker-compose --profile with-clickhouse down -v

Ensure no containers are left:

docker ps -a

Start only clickhouse:

docker-compose --profile with-clickhouse up -d db_clickhouse

If it fails, check:
Container logs:

docker-compose logs db_clickhouse

Used ports:

sudo lsof -i -P -n | grep LISTEN

If you uncommented the debug section in the Docker Compose file, comment it again and retry:

docker-compose --profile with-clickhouse up -d db_clickhouse

Please follow these steps and let us know the results.

[joshuaoc-intel]

docker-compose logs db_clickhouse

Attaching to gprofiler-ps-clickhouse
gprofiler-ps-clickhouse | /entrypoint.sh: create new user 'dbuser' instead 'default'
gprofiler-ps-clickhouse | Processing configuration file '/etc/clickhouse-server/config.xml'.
gprofiler-ps-clickhouse | Merging configuration file '/etc/clickhouse-server/config.d/docker_related_config.xml'.
gprofiler-ps-clickhouse | Logging trace to /var/log/clickhouse-server/clickhouse-server.log
gprofiler-ps-clickhouse | Logging errors to /var/log/clickhouse-server/clickhouse-server.err.log
gprofiler-ps-clickhouse | Processing configuration file '/etc/clickhouse-server/config.xml'.
gprofiler-ps-clickhouse | Merging configuration file '/etc/clickhouse-server/config.d/docker_related_config.xml'.
gprofiler-ps-clickhouse | Saved preprocessed configuration to '/var/lib/clickhouse/preprocessed_configs/config.xml'.
gprofiler-ps-clickhouse | Processing configuration file '/etc/clickhouse-server/users.xml'.
gprofiler-ps-clickhouse | Merging configuration file '/etc/clickhouse-server/users.d/default-user.xml'.
gprofiler-ps-clickhouse | Saved preprocessed configuration to '/var/lib/clickhouse/preprocessed_configs/users.xml'.
gprofiler-ps-clickhouse |
gprofiler-ps-clickhouse | /entrypoint.sh: running /docker-entrypoint-initdb.d/create_schema.sql
gprofiler-ps-clickhouse |
gprofiler-ps-clickhouse |
gprofiler-ps-clickhouse | Processing configuration file '/etc/clickhouse-server/config.xml'.
gprofiler-ps-clickhouse | Merging configuration file '/etc/clickhouse-server/config.d/docker_related_config.xml'.
gprofiler-ps-clickhouse | Logging trace to /var/log/clickhouse-server/clickhouse-server.log
gprofiler-ps-clickhouse | Logging errors to /var/log/clickhouse-server/clickhouse-server.err.log
gprofiler-ps-clickhouse | Processing configuration file '/etc/clickhouse-server/config.xml'.
gprofiler-ps-clickhouse | Merging configuration file '/etc/clickhouse-server/config.d/docker_related_config.xml'.
gprofiler-ps-clickhouse | Saved preprocessed configuration to '/var/lib/clickhouse/preprocessed_configs/config.xml'.
gprofiler-ps-clickhouse | Processing configuration file '/etc/clickhouse-server/users.xml'.
gprofiler-ps-clickhouse | Merging configuration file '/etc/clickhouse-server/users.d/default-user.xml'.
gprofiler-ps-clickhouse | Saved preprocessed configuration to '/var/lib/clickhouse/preprocessed_configs/users.xml'.

sudo lsof -i -P -n | grep LISTEN

sshd       1518          ubuntu    7u  IPv6  18020      0t0  TCP [::1]:6010 (LISTEN)
sshd       1518          ubuntu    9u  IPv4  18021      0t0  TCP 127.0.0.1:6010 (LISTEN)
sshd       4994            root    3u  IPv4  27366      0t0  TCP *:22 (LISTEN)
sshd       4994            root    4u  IPv6  27368      0t0  TCP *:22 (LISTEN)
systemd-r 14828 systemd-resolve   14u  IPv4  39109      0t0  TCP 127.0.0.53:53 (LISTEN)
container 15303            root   10u  IPv4  35355      0t0  TCP 127.0.0.1:32947 (LISTEN)

docker cp gprofiler-ps-clickhouse:/var/log/clickhouse-server/clickhouse-server.err.log ./logs_from_container
cat logs_from_container

2024.06.03 19:38:30.414632 [ 52 ] {} <Warning> Access(local directory): File /var/lib/clickhouse/access/users.list doesn't exist
2024.06.03 19:38:30.414657 [ 52 ] {} <Warning> Access(local directory): Recovering lists in directory /var/lib/clickhouse/access/
2024.06.03 19:38:30.679081 [ 52 ] {} <Warning> Application: Listen [0.0.0.0]:8123 failed: Poco::Exception. Code: 1000, e.code() = 98, Net Exception: Address already in use: 0.0.0.0:8123 (version 22.8.21.38 (official build)). If it is an IPv6 or IPv4 address and your host has disabled IPv6 or IPv4, then consider to specify not disabled IPv4 or IPv6 address to listen in <listen_host> element of configuration file. Example for disabled IPv6: <listen_host>0.0.0.0</listen_host> . Example for disabled IPv4: <listen_host>::</listen_host>
2024.06.03 19:38:30.679175 [ 52 ] {} <Warning> Application: Listen [0.0.0.0]:9000 failed: Poco::Exception. Code: 1000, e.code() = 98, Net Exception: Address already in use: 0.0.0.0:9000 (version 22.8.21.38 (official build)). If it is an IPv6 or IPv4 address and your host has disabled IPv6 or IPv4, then consider to specify not disabled IPv4 or IPv6 address to listen in <listen_host> element of configuration file. Example for disabled IPv6: <listen_host>0.0.0.0</listen_host> . Example for disabled IPv4: <listen_host>::</listen_host>
2024.06.03 19:38:30.679259 [ 52 ] {} <Warning> Application: Listen [0.0.0.0]:9004 failed: Poco::Exception. Code: 1000, e.code() = 98, Net Exception: Address already in use: 0.0.0.0:9004 (version 22.8.21.38 (official build)). If it is an IPv6 or IPv4 address and your host has disabled IPv6 or IPv4, then consider to specify not disabled IPv4 or IPv6 address to listen in <listen_host> element of configuration file. Example for disabled IPv6: <listen_host>0.0.0.0</listen_host> . Example for disabled IPv4: <listen_host>::</listen_host>
2024.06.03 19:38:30.679336 [ 52 ] {} <Warning> Application: Listen [0.0.0.0]:9005 failed: Poco::Exception. Code: 1000, e.code() = 98, Net Exception: Address already in use: 0.0.0.0:9005 (version 22.8.21.38 (official build)). If it is an IPv6 or IPv4 address and your host has disabled IPv6 or IPv4, then consider to specify not disabled IPv4 or IPv6 address to listen in <listen_host> element of configuration file. Example for disabled IPv6: <listen_host>0.0.0.0</listen_host> . Example for disabled IPv4: <listen_host>::</listen_host>
2024.06.03 19:38:30.679461 [ 52 ] {} <Warning> Application: Listen [0.0.0.0]:9009 failed: Poco::Exception. Code: 1000, e.code() = 98, Net Exception: Address already in use: 0.0.0.0:9009 (version 22.8.21.38 (official build)). If it is an IPv6 or IPv4 address and your host has disabled IPv6 or IPv4, then consider to specify not disabled IPv4 or IPv6 address to listen in <listen_host> element of configuration file. Example for disabled IPv6: <listen_host>0.0.0.0</listen_host> . Example for disabled IPv4: <listen_host>::</listen_host>
2024.06.03 19:38:33.795477 [ 1 ] {} <Warning> Application: Listen [::]:8123 failed: Poco::Exception. Code: 1000, e.code() = 0, DNS error: EAI: Address family for hostname not supported (version 22.8.21.38 (official build)). If it is an IPv6 or IPv4 address and your host has disabled IPv6 or IPv4, then consider to specify not disabled IPv4 or IPv6 address to listen in <listen_host> element of configuration file. Example for disabled IPv6: <listen_host>0.0.0.0</listen_host> . Example for disabled IPv4: <listen_host>::</listen_host>
2024.06.03 19:38:33.795590 [ 1 ] {} <Warning> Application: Listen [::]:9000 failed: Poco::Exception. Code: 1000, e.code() = 0, DNS error: EAI: Address family for hostname not supported (version 22.8.21.38 (official build)). If it is an IPv6 or IPv4 address and your host has disabled IPv6 or IPv4, then consider to specify not disabled IPv4 or IPv6 address to listen in <listen_host> element of configuration file. Example for disabled IPv6: <listen_host>0.0.0.0</listen_host> . Example for disabled IPv4: <listen_host>::</listen_host>
2024.06.03 19:38:33.795691 [ 1 ] {} <Warning> Application: Listen [::]:9004 failed: Poco::Exception. Code: 1000, e.code() = 0, DNS error: EAI: Address family for hostname not supported (version 22.8.21.38 (official build)). If it is an IPv6 or IPv4 address and your host has disabled IPv6 or IPv4, then consider to specify not disabled IPv4 or IPv6 address to listen in <listen_host> element of configuration file. Example for disabled IPv6: <listen_host>0.0.0.0</listen_host> . Example for disabled IPv4: <listen_host>::</listen_host>
2024.06.03 19:38:33.795790 [ 1 ] {} <Warning> Application: Listen [::]:9005 failed: Poco::Exception. Code: 1000, e.code() = 0, DNS error: EAI: Address family for hostname not supported (version 22.8.21.38 (official build)). If it is an IPv6 or IPv4 address and your host has disabled IPv6 or IPv4, then consider to specify not disabled IPv4 or IPv6 address to listen in <listen_host> element of configuration file. Example for disabled IPv6: <listen_host>0.0.0.0</listen_host> . Example for disabled IPv4: <listen_host>::</listen_host>
2024.06.03 19:38:33.831964 [ 1 ] {} <Warning> Application: Listen [::]:9009 failed: Poco::Exception. Code: 1000, e.code() = 0, DNS error: EAI: Address family for hostname not supported (version 22.8.21.38 (official build)). If it is an IPv6 or IPv4 address and your host has disabled IPv6 or IPv4, then consider to specify not disabled IPv4 or IPv6 address to listen in <listen_host> element of configuration file. Example for disabled IPv6: <listen_host>0.0.0.0</listen_host> . Example for disabled IPv4: <listen_host>::</listen_host>

[joshuaoc-intel]

diagnostic.sh is giving a false positive. The containers are actually connecting to each other even though the error is throwing. I suggest an update to the script