UnicodeEncodeError: 'ascii' codec can't encode character [with ugettext_lazy]

[madisvain]

Seems like anymail does not handle unicode correctly.

UnicodeEncodeError: 'ascii' codec can't encode character u'\u0142' in position 0: ordinal not in range(128)
  File "django/core/handlers/base.py", line 149, in get_response
    response = self.process_exception_by_middleware(e, request)
  File "django/core/handlers/base.py", line 147, in get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "django/utils/decorators.py", line 149, in _wrapped_view
    response = view_func(request, *args, **kwargs)
  File "gsmtasks/users/views.py", line 250, in password_reset
    form.save()
  File "gsmtasks/users/forms.py", line 174, in save
    msg.send()
  File "django/core/mail/message.py", line 292, in send
    return self.get_connection(fail_silently).send_messages([self])
  File "anymail/backends/base.py", line 86, in send_messages
    sent = self._send(message)
  File "anymail/backends/base_requests.py", line 56, in _send
    return super(AnymailRequestsBackend, self)._send(message)
  File "anymail/backends/base.py", line 116, in _send
    response = self.post_to_esp(payload, message)
  File "anymail/backends/base_requests.py", line 69, in post_to_esp
    response = self.session.request(**params)
  File "opbeat/instrumentation/packages/base.py", line 63, in __call__
    args, kwargs)
  File "opbeat/instrumentation/packages/base.py", line 222, in call_if_sampling
    return self.call(module, method, wrapped, instance, args, kwargs)
  File "opbeat/instrumentation/packages/requests.py", line 38, in call
    return wrapped(*args, **kwargs)
  File "requests/sessions.py", line 461, in request
    prep = self.prepare_request(req)
  File "requests/sessions.py", line 394, in prepare_request
    hooks=merge_hooks(request.hooks, self.hooks),
  File "requests/models.py", line 298, in prepare
    self.prepare_body(data, files, json)
  File "requests/models.py", line 452, in prepare_body
    body = self._encode_params(data)
  File "requests/models.py", line 97, in _encode_params
    return urlencode(result, doseq=True)
  File "python2.7/urllib.py", line 1357, in urlencode
    l.append(k + '=' + quote_plus(str(elt)))

[medmunds]

Anymail is meant to handle Unicode, but there may be some cases we missed.

• Which ESP are you using?
• What EmailMessage attr was that Unicode character in? (If you're not sure, seeing the code where you build the EmailMessage would be helpful to narrow things down.)

[madisvain]

Hei - I'll be looking into this with our team also hopefully.

• We are using Mailgun as an ESP currently.
• It seems to be the subject - I have attached the Sentry error here.
  

So the test string would probably be "Wymagane zresetowanie hasła do GSMtasks" - it's Polish character I suppose.

[medmunds]

Hmm, I'm having trouble reproducing the problem. This seems to work with the Mailgun backend:

# -*- coding: utf-8 -*-
from django.core.mail.message import EmailMessage
msg = EmailMessage(
    u"Wymagane zresetowanie hasła do GSMtasks",
    u"Kliknij aby zresetować hasło",
    '[email protected]', ['[email protected]'])
msg.send()

What version of requests do you have installed, and what version of python? (I've just tested this on python 2.7.11 with both requests 2.9.1 and 2.11.1.)

[medmunds]

Looking at the requests source, it seems like the only way you'd see this error (in python 2.x) is if your subject were somehow a unicode string, but isinstance(subject, unicode) returned False.

Is there any chance the subject is a Django ugettext_lazy translation string? If so, you could call django.utils.encoding.force_text on it before placing it in the EmailMessage subject.

(I'm trying to figure out whether Anymail should be calling force_text for you before handing off to requests. I have this vague memory of a similar problem with the from_email, which we ended up fixing in Django's EmailMessage code.)

[medmunds]

@madisvain were you able to track this down to a Django lazy translation string, or otherwise reproduce it? As I mentioned above, Anymail+Mailgun seems to handle the unicode characters just fine.

[anderspetersson]

@medmunds I had the same problem and it was due to a lazy translation string, force_text fixed it.

[medmunds]

@anderspetersson thanks for the confirmation; I updated the issue title to reflect this.

There are actually some potentially-serious problems with using ugettext_lazy but not explicitly forcing the result to non-lazy text somewhere. Even if you're not seeing encoding errors, you could end up with unexpected results. The issue isn't specific to Anymail. I posted to the django-dev list seeking clarification; will see what comes out of the discussion there.

[medmunds]

OK, I've become convinced that as a package with "Django" in its name, one of django-anymail's responsibilities is handling ugettext_lazy objects for its users. Anymail is the bridge between the Django world and non-Django ESP APIs, so it needs to handle converting Django lazy strings for consumption by those APIs.

The fix needs to go into Anymail somewhere in the base backend, because it applies to every EmailMessage attribute that might be text. (Or that might contain text -- consider message.metadata = {'dept': ugettext_lazy("Sales")}.) And it's relevant for every ESP, whether we're using requests or calling the ESP's own python API package.