Skip to content

Amazon SES mangles non-ASCII message bodies if using open/click tracking #115

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
varche1 opened this issue Jul 20, 2018 · 5 comments
Closed

Amazon SES mangles non-ASCII message bodies if using open/click tracking #115

varche1 opened this issue Jul 20, 2018 · 5 comments
Assignees
@medmunds
Labels
bug esp:Amazon SES

Comments

@varche1
Copy link

varche1 commented Jul 20, 2018

Hello!

I've got encoding problems for Amazon SES with SES Event Publishing for non-ASCII characters:
email1

I found out that this problem only shows up when using AMAZON_SES_CONFIGURATION_SET_NAME (event tracking with SES Event Publishing)

I also found out that the email content (RawMessage -> Data) is identical both when using AMAZON_SES_CONFIGURATION_SET_NAME and without it.

The problem is the way Amazon SES with SES Event Publishing treat with 'utf-8' encoding email content if Content-Transfer-Encoding: 8bit:

Content-Type: multipart/alternative;
 boundary="===============4882781876866556742=="
MIME-Version: 1.0
Subject: Subject
From: [email protected]
To: [email protected]
Date: Fri, 20 Jul 2018 11:30:06 -0000
Message-ID: <20180720113006.12882.39747@localhost>

--===============4882781876866556742==
MIME-Version: 1.0
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: 8bit

Это text body
--===============4882781876866556742==
MIME-Version: 1.0
Content-Type: text/html; charset="utf-8"
Content-Transfer-Encoding: 8bit

<html>Это html <b>body</b></html>
--===============4882781876866556742==--

Content-Transfer-Encoding: base64 works well, but how Anymail can use it?

How to reproduce:

from django.core.mail import send_mail, get_connection
connection = get_connection("anymail.backends.amazon_ses.EmailBackend")

send_mail(
    "Subject",
    "Это text body",
    "[email protected]",
    ["[email protected]"],
    html_message="<html>Это html <b>body</b></html>",
    connection=connection
)

By the way https://docs.aws.amazon.com/ses/latest/DeveloperGuide/send-email-raw.html use Content-Transfer-Encoding: base64

  • Anymail version: 3.0
  • ESP (Mailgun, SendGrid, etc.): Amazon SES
  • Versions of Django, requests, python: Django 1.10.8; requests 2.13.0; Python 2.7.12
  • Exact error message and/or stack trace: -
@medmunds
Copy link
Contributor

Huh, very weird. Thanks for verifying the RawMessage sent to SES is the same in both cases. FWIW, that message looks valid, which makes this seem like an SES bug.

It might be helpful to compare the "show original" version of the two messages as received in your email client, to see what Amazon SES is mangling when using a configuration set.

Django (and therefore Anymail) uses Python's underlying email package to generate the raw mime message. I can't think offhand of an easy way to convince it to use some other transfer encoding on particular MIME parts, though my memory is that the Python 2.7 email package tends to favor Content-Transfer-Encoding: 8bit in cases where the Python 3 version uses base64 or quoted-printable instead. (8bit can sometimes cause problems with older email clients or gateways, but it really should be supported at the sending end by any reasonably-modern service.)

It's going to be several days before I'd be able to take a closer look at this. It might be worth opening an issue with Amazon or on StackOverflow; you could remove Anymail and Django from the equation by capturing isolating the boto3 SendEmailRaw params similar to how you captured the RawMessage Data above.

@medmunds
Copy link
Contributor

OK, I've tracked this down to what appears to be an Amazon SES bug: if you are using open or click tracking, when SES rewrites your message to add the tracking pixels/links, it mangles any Content-Transfer-Encoding: 8bit text parts.

Here are two immediate workarounds:

  • If you don't need open or click tracking, don't enable these options in your SES configuration set. (SES won't rewrite your message, and everything else will work correctly.)
  • If you do need open or click tracking, ensure that both the plaintext and html message bodies have at least one line longer than 998 characters—it can be a line of spaces or an HTML comment. (The long lines trick Django into forcing Content-Transfer-Encoding: quoted-printable, and SES's message rewriting handles this correctly.)

I think I've also found a way to work around the SES bug in Anymail's backend, but it will require more testing. (I'll report the bug to Amazon first; obviously it would be preferable for them to fix SES.)

@medmunds medmunds changed the title Encoding problems for Amazon SES with SES Event Publishing Amazon SES mangles non-ASCII message bodies if using open/click tracking Jul 31, 2018
@medmunds medmunds self-assigned this Jul 31, 2018
@varche1
Copy link
Author

varche1 commented Jul 31, 2018

Thank you to possible workarounds! It works, but any way, all of this is a hack or limits the possibilities. Waiting for SES will fix this bug

@medmunds
Copy link
Contributor

Here's the bug report to SES: https://forums.aws.amazon.com/thread.jspa?threadID=287048

@medmunds
Copy link
Contributor

Workaround released in Anymail v4.0.

@medmunds medmunds mentioned this issue Apr 12, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@medmunds medmunds

Labels
bug esp:Amazon SES
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@medmunds @varche1