Unable to use Docker Compose support when mixing dedicated and shared services

[sivaprasadreddy]

I have a microservices-based mono-repo with multiple spring-boot applications. I would like to use Spring Boot Docker Compose support to start the dependent services. There are few services (ex: db) that are specific to each service and there are some services (ex: RabbitMQ) shared by multiple services.

All the microservices dependencies are defined in a single compose.yml fil. If I start any one microservice it starts all the services defined in the compose.yml file, which is ok for me.

version: "3.8"
services:
  catalog-db:
    image: postgres:16-alpine
    ...
  orders-db:
    image: postgres:16-alpine
    ...
  rabbitmq:
    image: rabbitmq:3.12.11-management
    ...

But the problem is, when I have multiple databases defined for different services, Flyway initialization fails.

Parameter 0 of method flyway in org.springframework.boot.autoconfigure.flyway.FlywayAutoConfiguration$FlywayConfiguration required a single bean, but 2 were found:
	- flywayConnectionDetailsForCatalogDb: defined in unknown location
	- flywayConnectionDetailsForOrdersDb: defined in unknown location

Is there a way to configure at the microservice level (using spring.docker.compose.* properties) to take only certain services into account?

[sivaprasadreddy]

Instead of putting all the services in a single compose.yml file, I tried to split them into separate files.

common.yml

version: "3.8"
services:
  rabbitmq:
    image: rabbitmq:3.12.11-management
    ...

catalog.yml

version: "3.8"
services:
  catalog-db:
    image: postgres:16-alpine
    ...

orders.yml

version: "3.8"
include:
  - common.yml
services:
  orders-db:
    image: postgres:16-alpine
    ...

Now, in the application.properties file I configured which file to use as follows:

spring.docker.compose.file=compose-files/orders.yml
spring.docker.compose.lifecycle-management=start_only

Now, the problem is, that the first spring-boot app (say order-service) that I started works as expected. But when I start the second application (say catalog-service) I see the following message due to which the required services are not being started.

s.b.d.c.l.DockerComposeLifecycleManager : Using Docker Compose file '/path/compose-files/catalog.yml'
s.b.d.c.l.DockerComposeLifecycleManager : There are already Docker Compose services running, skipping startup

[scottfrederick]

I tried to split them into separate files.

Another option would be to keep the services in one file, use Docker Compose profiles, and modify your application.properties to control when individual services are started.

I see the following message due to which the required services are not being started.

This behavior is mentioned in the documentation:

If the Docker Compose services are already running when starting the application, Spring Boot will only create the service connection beans for each supported container. It will not call docker compose up again and it will not call docker compose stop when the application is shutdown.

There is some discussion on why we chose to implement things this way in #38398.

You can set spring.docker.compose.lifecycle-management=none to prevent this message. You will have to start the service manually (for example, with docker compose up), but Spring Boot will create the service connections as expected.

An additional mode like the one suggested in #39749, where docker compose up would be called without first checking to see what services are running, could help this situation. I will close this issue in favor of that issue.

[sach429]

@scottfrederick

Another option would be to keep the services in one file, use Docker Compose profiles, and modify your application.properties to control when individual services are started.

Do you mean to say, if I create a single file

version: "3.8"
services:
  catalog-db:
    image: postgres:16-alpine
    profiles:
      - catalog
    ...
  orders-db:
    image: postgres:16-alpine
    profiles:
      - order
    ...
  rabbitmq:
    image: rabbitmq:3.12.11-management
    ...

And then specify in the application.properties of Catalog service and Order service spring.docker.compose.profiles.active=catalog and spring.docker.compose.profiles.active=order respectively, then there will be two DB instances running but Catalog service and Order Service will only create one DB ConnectionDetails instead of two?

[bclozel]

@sach429 that is correct.

[sach429]

@sach429 that is correct.

@bclozel I don't think it's working that way. Let me clarify how I am running my applications. I have two spring boot apps and I am running them directly from IntelliJ one after another.

For eg. this is my setup and my intention is to have Catalog and Order have their own separate DBs but share the same Redis server

Catalog:
application.properties
spring.docker.compose.profiles.active=catalog,redis
compose.yml

version: "3.8"
services:
  catalog-db:
    image: postgres:16-alpine
    profiles:
      - catalog
    ...
  redis:
    image: redis:latest
    profiles:
      - redis
    .....

Order:
application.properties
spring.docker.compose.profiles.active=order
compose.yml

version: "3.8"
services:
  orders-db:
    image: postgres:16-alpine
    profiles:
      - order
    ...
  redis:
    image: redis:latest
    profiles:
      - redis
    .....

I am running Catalog first, followed by Order. When I start Order, it detects the already running postgres container and eventually created two ConnectionDetails instances instead of one. I put breakpoints in the code, DefaultDockerCompose, list contains all the running containers in the host machine and seems like the code is designed that way to detect all the existing running containers and eventually creating a ConnectionDetails instance as long as they match any of the supported services.

It does not matter if I merge the two compose files into one and refer to the same file in two Projects. I believe the profiles value is being passed as a flag when starting the docker services. So it's executing docker compose --profile catalog --profile redis up and docker compose --profile order when running Catalog and Order services respectively. It's not used to filter an existing running container.

[bclozel]

@sach429 Sorry I missed the ConnectionDetails bit in your initial comment. I don't understand what you're asking here. Is this behavior incorrect? Do you think this does not work for a multi-module project? If you believe there is a bug, could you create a new issue?

[sach429]

@bclozel I am not sure if it's a bug because I don't fully understand if the way it is working now is by design or not

My requirement is simple - Two Spring Boot apps and each talking to it's own DB. When I run the apps one after another - App1 starts it creates DB container#1 and thereby one ConnectionsDetails instance, App2 starts creates DB container#2 and also detects DB container#1 thereby creating two ConnectionsDetails instances.

This causes an issue in code which expects one ConnectionDetails instance.

Is there a way that when App2 starts it ignores DB Container#1. I don't think profile helps here and reading the documentation it seems like the use of profile is to prevent certain services from being started rather than ignoring already running containers.

Essentially what I need is App2 to ignore connecting to DB container#1. The feature to ignore is available to certain extent but it's not practical in this case. The documentation states that we can specify a label like below

labels:
      org.springframework.boot.ignore: true

But, it's not useful here, because there's no way to tell that only App2 should ignore, Since both are spring boot apps, either both of them will ignore or none of them.

[mhalbritter]

My requirement is simple - Two Spring Boot apps and each talking to it's own DB.

Why don't you use two separate Docker Compose files for that?

sivaprasadreddy had the problem that it's currently not possible to use the Docker Compose support if you have two services with a separate database for each one AND a shared component (rabbitmq in that example).

But if your usecase doesn't involve a shared component, it should be possible.

[mhalbritter]

I looked into the original problem a bit and tried:

• Multiple profiles: Doesn't work, as we don't filter the running containers for profiles (which doesn't seem to be possible. Some of the docker compose commands are profile aware, but not ps. Even docker inspect <id> doesn't return the profile the container has been assigned). On start of the 2nd application it complains about multiple JDBC connection details.
• Separate files, using include: to include the shared component: While now every application has their own database, the shared component is then started twice.
• Separate files, using multiple files for spring.docker.compose.file: same as with the include: directive.

[mhalbritter]

What could work is:

Run docker compose --profile <profile> config --format=json. This returns the config for the services filtered by profile in JSON format:

{
  "name": "sb-40139",
  "networks": {
    "default": {
      "name": "sb-40139_default",
      "ipam": {}
    }
  },
  "services": {
    "order-db": {
      "profiles": [
        "order"
      ],
      "command": null,
      "entrypoint": null,
      "environment": {
        "POSTGRES_DB": "mydatabase",
        "POSTGRES_PASSWORD": "secret",
        "POSTGRES_USER": "myuser"
      },
      "image": "postgres:17",
      "networks": {
        "default": null
      },
      "ports": [
        {
          "mode": "ingress",
          "target": 5432,
          "protocol": "tcp"
        }
      ]
    },
    "redis": {
      "command": null,
      "entrypoint": null,
      "image": "redis:7",
      "networks": {
        "default": null
      },
      "ports": [
        {
          "mode": "ingress",
          "target": 6379,
          "protocol": "tcp"
        }
      ]
    }
  }
}

Then run docker compose ps --format=json, which returns all running containers for the project (doesn't do profile filtering):

// ...
{
  "Command": "\"docker-entrypoint.s…\"",
  "CreatedAt": "2024-12-09 13:53:46 +0100 CET",
  "ExitCode": 0,
  "Health": "",
  "ID": "eae6fa2df78e",
  "Image": "postgres:17",
  "Labels": "...",
  "LocalVolumes": "1",
  "Mounts": "577790079dbe43…",
  "Name": "sb-40139-order-db-1",
  "Names": "sb-40139-order-db-1",
  "Networks": "sb-40139_default",
  "Ports": "0.0.0.0:32812->5432/tcp, :::32812->5432/tcp",
  "Project": "sb-40139",
  "Publishers": [
    {
      "URL": "0.0.0.0",
      "TargetPort": 5432,
      "PublishedPort": 32812,
      "Protocol": "tcp"
    },
    {
      "URL": "::",
      "TargetPort": 5432,
      "PublishedPort": 32812,
      "Protocol": "tcp"
    }
  ],
  "RunningFor": "8 minutes ago",
  "Service": "order-db",
  "Size": "0B",
  "State": "running",
  "Status": "Up 8 minutes"
}
// ...

Now we can look in the config from above (under the services object) using the docker compose ps Service field. If it's not in the config, we totally ignore the running container.

This way, when specifying e.g. spring.docker.compose.profiles.active=order it should only return running containers belonging to the order profile.

[mhalbritter]

I also filed an issue on Docker compose side: docker/compose#12361

[sach429]

My requirement is simple - Two Spring Boot apps and each talking to it's own DB.

Why don't you use two separate Docker Compose files for that?

sivaprasadreddy had the problem that it's currently not possible to use the Docker Compose support if you have two services with a separate database for each one AND a shared component (rabbitmq in that example).

But if your usecase doesn't involve a shared component, it should be possible.

Thank you for looking into it. My use case is described in detail here . I am using two docker compose files and there's a common service. But I believe it doesn't matter if there's a common service or not. As long as two spring boot apps are using a common type of service which will create same type of ConnectionDetails more than once, it will cause this issue.

[sach429]

What could work is:

Run docker compose --profile <profile> config --format=json. This returns the config for the services filtered by profile in JSON format:

{
  "name": "sb-40139",
  "networks": {
    "default": {
      "name": "sb-40139_default",
      "ipam": {}
    }
  },
  "services": {
    "order-db": {
      "profiles": [
        "order"
      ],
      "command": null,
      "entrypoint": null,
      "environment": {
        "POSTGRES_DB": "mydatabase",
        "POSTGRES_PASSWORD": "secret",
        "POSTGRES_USER": "myuser"
      },
      "image": "postgres:17",
      "networks": {
        "default": null
      },
      "ports": [
        {
          "mode": "ingress",
          "target": 5432,
          "protocol": "tcp"
        }
      ]
    },
    "redis": {
      "command": null,
      "entrypoint": null,
      "image": "redis:7",
      "networks": {
        "default": null
      },
      "ports": [
        {
          "mode": "ingress",
          "target": 6379,
          "protocol": "tcp"
        }
      ]
    }
  }
}

Then run docker compose ps --format=json, which returns all running containers for the project (doesn't do profile filtering):

// ...
{
  "Command": "\"docker-entrypoint.s…\"",
  "CreatedAt": "2024-12-09 13:53:46 +0100 CET",
  "ExitCode": 0,
  "Health": "",
  "ID": "eae6fa2df78e",
  "Image": "postgres:17",
  "Labels": "...",
  "LocalVolumes": "1",
  "Mounts": "577790079dbe43…",
  "Name": "sb-40139-order-db-1",
  "Names": "sb-40139-order-db-1",
  "Networks": "sb-40139_default",
  "Ports": "0.0.0.0:32812->5432/tcp, :::32812->5432/tcp",
  "Project": "sb-40139",
  "Publishers": [
    {
      "URL": "0.0.0.0",
      "TargetPort": 5432,
      "PublishedPort": 32812,
      "Protocol": "tcp"
    },
    {
      "URL": "::",
      "TargetPort": 5432,
      "PublishedPort": 32812,
      "Protocol": "tcp"
    }
  ],
  "RunningFor": "8 minutes ago",
  "Service": "order-db",
  "Size": "0B",
  "State": "running",
  "Status": "Up 8 minutes"
}
// ...

Now we can look in the config from above (under the services object) using the docker compose ps Service field. If it's not in the config, we totally ignore the running container.

This way, when specifying e.g. spring.docker.compose.profiles.active=order it should only return running containers belonging to the order profile.

Couldn't an easier solution be achieved using labels instead. For eg. you could apply label org.springframwork.boot.service-name=orderdb and org.springframwork.boot.service-name=catalogdb to the two postgres containers for order and catalog services respectively and use a property like spring.docker.compose.labels.active=!catalogdb OR spring.docker.compose.labels.active=orderdb?

[bclozel]

@sach429 this looks like supporting profiles in docker ps through custom labels. I think docker/compose#12361 is much more promis in because this means support for the entire docker compose ecosystem.

[mhalbritter]

Update on Docker Compose side: They redirected me to docker/compose#11737, and using --orphans=false should get us the behavior we want. I'll give that a try.

[mhalbritter]

--orphans=false did the trick. I've also added an integration test which tests that this indeed works.