Implementing LB forwarding rules by URL to ECS tasks

[flaviostutz]

What I did

Created custom attributes x-aws-loadbalancer_urls and x-aws-loadbalancer_https_certificate. The objective is to configure the access scheme/domains/apis right for the URLs used to access the services in the docker-compose.yml itself so that "releases" to multiple environments are more easily done.

According to "x-aws-loadbalancer_urls", new listeners and rules are automatically added to the LB/listeners so that according to scheme (HTTP/HTTPS), Host header and Request path, the request will be forwarded to the desired TargetGroup of the service. If any url is "https", it will use the "x-aws-loadbalancer_https_certificate" parameter in order to create a HTTPS listener for the balancer. A default HTTP->HTTPS redirect rule is created on the HTTP listener for URLs that are present only in HTTPS form.

To avoid re-creating unnecessary resources during updates, the algorithm takes care to reproduce the same CloudFormation template bytes for parts that don't need to be changed by using ordering.

To avoid collisions in "normalized" names, I added a CRC32 string to the "normalize" function. This is specially needed because as we have URLs and paths on "rules" element names that needs to be normalized, the strings would get too long for names and simply truncating then would cause collisions (apart from that case, we could have collision with service names that have small differences in names).

I fixed a small bug on tty that happens on small screens/long names combinations that was arising during our tests.

I tried to implement this with minimum impact to current LB published port code base as an alternative that will only take place if the user specifies x-aws-loadbalancer_urls. The default behavior hasn't changed.

I tested some different cases using both the published port and the URL specification scenarios with some of our real cases over the weekend.

This PR is part of an effort for us to migrate our current self managed Swarm Clusters on VMs to ECS. Today we use Traefik/Caddy labels in docker-compose.yml to configure URLs for each environment and it makes our operation very agile.

Related issue
fixes #777

(not mandatory) A picture of a cute animal, if possible in relation with what you did

[flaviostutz]

Some tests:

docker-compose.yml:

x-aws-loadbalancer: arn:aws:elasticloadbalancing:us-west-2:aaaaaaaa:loadbalancer/app/poc2/aaaaaa
x-aws-loadbalancer_https_certificate: arn:aws:acm:us-west-2:bbbbbbb:certificate/bbbbbb

services:
  professores-frontend:
    image: professores-frontend:0.20.0
    ports:
      - target: 80
        protocol: tcp
        x-aws-protocol: http
        x-aws-loadbalancer_urls: https://professores.poc2.stutz.com.br http://professores1.poc2.stutz.com.br
...

docker compose up

Just after deploy, access https://professores.poc2.stutz.com.br

[ndeloof]

I'd prefer we don't get more x-aws-* custom extensions to tweak load balancer.

Short terms one should just provide a load balancer and configure it by more specific tools. Longer terms, the Routing proposal could be used to express how to route traffic to service and external visibility (domain name, path, certificate, ...) then we can translate such a generic definition into AWS-specific components.

[ndeloof]

I proposed some docs addition to clarify use of an external load balancer: docker/docs#11771
unfortunately, certificate must be set on listeners, not load balancer, so one can't get them pre-configured before running compose up.

[ndeloof]

I understand the intent here but I don't like we polute the logical resource name and make them unpredictable (for a humain brain, which can't compute CRCs :P)
I don't expect conflicts to be such a common thing, and we could detect those before conversion

[flaviostutz]

I had a lot of naming collisions when adding path based rules because the path names are too long for being used as the resource name without truncating it, and by truncating/removing non alphabetic chars the chance of a collision was high. We could enhance this mechanism by verifying if there is, among all rules, a "real" collision and then add a counter to the resource name when adding path based routing to ECS here...

[ndeloof]

This PR is mixing lot's of propopsed improvements,

• logical resource naming conflict resolution
• support for Listener certifcate by ARN
• support for Listener URLs
• a default action to send 404 (not sure I fully understand this part)
• path based routing
• http to https redirection

for all those we need to establish product direction and alignment with local developer experience: what does it mean to have path based routing on ECS if you can't test for local deployment on your workstation?

this make it highly inspirational but low chance to get merged as-is
If you don't mind, I'll cherry-pick code from it into feature-focussed PRs, in sync with documentation updates and relase schedule.

[flaviostutz]

This PR is mixing lot's of propopsed improvements,

• logical resource naming conflict resolution
• support for Listener certifcate by ARN
• support for Listener URL_s_
• a default action to send 404 (not sure I fully understand this part)
• path based routing
• http to https redirection

for all those we need to establish product direction and alignment with local developer experience: what does it mean to have path based routing on ECS if you can't test for local deployment on your workstation?

this make it highly inspirational but low chance to get merged as-is
If you don't mind, I'll cherry-pick code from it into feature-focussed PRs, in sync with documentation updates and relase schedule.

I know there is lots of changes here. I don't mind spliting this into multiple discussions/PRs.

[ndeloof]

closing as obsolete
routing support require more discussions on modeling and actual implementaiton
and cloudformation overlays offer an actionable solution for users