HTTPS Support for ECS Load Balancers

[jdconley]

I'd like to be able to specify a TLS certificate from the AWS ACM that would configure a listener on the load balancer port 443 and appropriate inbound security rule. I do this on my own today after running docker compose up in the ecs context.

docker-compose.yml would look something like:

version: "3.8"
services:
    web:
        image: my_image_that_listens_for_http_on_port_8080
        x-aws-loadbalancer_certificate: arn:aws:acm:us-east-2:858246198183:certificate/[uuid]
        ports:
            - "443:8080"

[ndeloof]

Issue doing so is that we can't use 443:8080 port translation for service-to-service communication (services won't pass throught a load-balancer to connect to each other)
Having port mapping support for external communication but not for internal one would be error prone.

I have no idea yet how we could balance this with a reasonable and portable workaround.

[jdconley]

Issue doing so is that we can't use 443:8080 port translation for service-to-service communication (services won't pass throught a load-balancer to connect to each other)
Having port mapping support for external communication but not for internal one would be error prone.

I have no idea yet how we could balance this with a reasonable and portable workaround.

True, this short hand would be troublesome. This would be quite useful in AWS where the load balancer that compose-cli uses already supports HTTPS TLS with just a couple of CloudFormation configurations. Azure Front Door is similar.

Just a straw man here. Without modifying ports itself, though that might be ideal, like how swarm added "ingress", maybe something like this? And in this configuration the "published" version of the port would take precedence in CloudFormation ELB configuration, but the internal "8080" port would still be exposed and available to the host network.

version: "3.8"
services:
  web:
    image: my_image_that_listens_for_http_on_port_8080
    x-external-ingress:
      - driver: ecs
        target: 8080
        published: 443
        protocol: https
        certificate: arn:aws:acm:us-east-2:858246198183:certificate/[uuid]
    ports: 
      - "8080"

[flaviostutz]

Issue doing so is that we can't use 443:8080 port translation for service-to-service communication (services won't pass throught a load-balancer to connect to each other)
Having port mapping support for external communication but not for internal one would be error prone.

I have no idea yet how we could balance this with a reasonable and portable workaround.

@ndeloof, proposal at #777 would have the same issue regarding to service-to-service communication you mentioned?

[ndeloof]

@flaviostutz afaik AWS load balancers only apply to external incoming traffic, so #777 would also not be supported for service-to-service routing. I wonder AWS AppMesh would offer more flexibility, and/or we could expose a higher level abstraction for "routing" that could be implemented by AppMesh but also others, which would make a huge benefits for compose-spec as same compose file could then be deployed both on AWS and local development (using Envoy/Træffic/xxx)

[flaviostutz]

@ndeloof I've been studying AppMesh with ECS and I think there are tons of advantages for us to integrate AppMesh for both ingress and service-to-service routing. By doing this we can enable the usage of other native AWS capabilities like service calls tracing (xray), monitoring (CloudWatch), path based routing and weighted service calls too.

For both ingress and service-to-service routing, AWS indicates this architecture:

https://aws.amazon.com/blogs/containers/app-mesh-integration-with-aws-alb-ingress-controller/

Here we can see a great evolutionary example on CloudFormation templates for ECS + AppMesh:
https://github.com/aws/aws-app-mesh-examples/tree/master/walkthroughs/howto-ecs-basics/deploy

Doing this kind of deployment without compose would be very hard (CloudFormation file gets very complex because there are tons of items to configure), but by using compose cli this will be transparent for users and we'll enable great tools with no additional costs at AWS (as AppMesh is deployed as "side cars" there will be no additional charge on doing this).

What do you think in implementing compose-cli using the templates based on the examples/diagrams above?

related #694

[ndeloof]

I haven't looked into details on AppMesh but this indeed seems to me to be the right way to introduce more flexibility and act as implementation backend for compose-spec/compose-spec#111

Does app mesh come with extra costs (so we should only use it when required)?

Need to sync with AWS team on best approach and get their recommendations

[ndeloof]

@flaviostutz as you already proposed compose-spec/compose-spec#111 and App Mesh could be our best option to implement this by ECS integration, would you mind creating a Feature Request on
https://github.com/docker/roadmap to get Docker dig into compose support for service mesh?
This would bring some product visibility to this and help drive a global effort on compose-spec / compose-cli / docker compose. cc @nebuk89

[flaviostutz]

Sure, @ndeloof! How do I do this? Is it by simply creating a new issue there with the proposal?

[ndeloof]

yes, and add links to your issues across relevant repositories

[flaviostutz]

I haven't looked into details on AppMesh but this indeed seems to me to be the right way to introduce more flexibility and act as implementation backend for compose-spec/compose-spec#111

Does app mesh come with extra costs (so we should only use it when required)?

Need to sync with AWS team on best approach and get their recommendations

No extra costs are added because the containers used for the mesh proxy/utilities are "sidecars" and the Mesh service is not billed. Probably there will be a small increment in Task CPU/Mem usage for the proxy routing.

[flaviostutz]

Issue doing so is that we can't use 443:8080 port translation for service-to-service communication (services won't pass throught a load-balancer to connect to each other)
Having port mapping support for external communication but not for internal one would be error prone.
I have no idea yet how we could balance this with a reasonable and portable workaround.

True, this short hand would be troublesome. This would be quite useful in AWS where the load balancer that compose-cli uses already supports HTTPS TLS with just a couple of CloudFormation configurations. Azure Front Door is similar.

Just a straw man here. Without modifying ports itself, though that might be ideal, like how swarm added "ingress", maybe something like this? And in this configuration the "published" version of the port would take precedence in CloudFormation ELB configuration, but the internal "8080" port would still be exposed and available to the host network.

version: "3.8"
services:
  web:
    image: my_image_that_listens_for_http_on_port_8080
    x-external-ingress:
      - driver: ecs
        target: 8080
        published: 443
        protocol: https
        certificate: arn:aws:acm:us-east-2:858246198183:certificate/[uuid]
    ports: 
      - "8080"

Using a certificate spec inside the service/port means we'd have one LB/listener for each service, which could increase AWS costs (>15.00/mon per service), so we could employ an ingress routing by domain/path, so that we could have only one LB distributing ingress requests to the services on the stack (just like we do with Traefik and Caddy). See compose-spec/compose-spec#111 for a discussion on this topic

We could use the top level attribute x-aws-loadbalancer_https_certificate that will be used on the "stack" LB. There is a pending PR with this feature at #871. You can try this fork while we discuss further about adding AppMesh capabilities to compose cli.

[ndeloof]

Using a certificate spec inside the service/port means we'd have one LB/listener for each service

Nope, by compose-spec this just means each service MIGHT have it's own configuration, which makes perfect sense. Then AWS implementation, which uses a single Load Balancer, could add a restriction that all exposed service MUST use the same certificate.

I don't like the idea to introduce x-aws-loadbalancer_https_certificate, we already added way too much x-aws- attributes that will never be part of the compose-spec. IMHO custom extension should be reserved for experiment with features that we expect could be implemented on other platforms and then be adopted by the spec. x-aws-pull_credentials is such an attribute.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale]

This issue has been automatically closed because it had not recent activity during the stale period.