Sydent returns a 500 error when email address or other email fields contain non-ascii characters

[ara4n]

Traceback (most recent call last):
  File "/home/vectoris/venv/local/lib/python2.7/site-packages/twisted/web/http.py", line 1705, in dataReceived
    finishCallback(data[contentLength:])
  File "/home/vectoris/venv/local/lib/python2.7/site-packages/twisted/web/http.py", line 2121, in _finishRequestBody
    self.allContentReceived()
  File "/home/vectoris/venv/local/lib/python2.7/site-packages/twisted/web/http.py", line 2196, in allContentReceived
    req.requestReceived(command, path, version)
  File "/home/vectoris/venv/local/lib/python2.7/site-packages/twisted/web/http.py", line 920, in requestReceived
    self.process()
--- <exception caught here> ---
  File "/home/vectoris/venv/local/lib/python2.7/site-packages/twisted/web/server.py", line 199, in process
    self.render(resrc)
  File "/home/vectoris/venv/local/lib/python2.7/site-packages/twisted/web/server.py", line 259, in render
    body = resrc.render(self)
  File "/home/vectoris/venv/local/lib/python2.7/site-packages/twisted/web/resource.py", line 250, in render
    return m(request)
  File "sydent/http/servlets/store_invite_servlet.py", line 104, in render_POST
    sendEmail(self.sydent, "email.invite_template", address, substitutions)
  File "sydent/util/emailutils.py", line 54, in sendEmail
    allSubstitutions[k] = v.decode('utf8')
  File "/home/vectoris/venv/lib/python2.7/encodings/utf_8.py", line 16, in decode
    return codecs.utf_8_decode(input, errors, True)
exceptions.UnicodeEncodeError: 'ascii' codec can't encode character u'\u201c' in position 1: ordinal not in range(128)


Traceback (most recent call last):
  File "/home/vectoris/venv/local/lib/python2.7/site-packages/twisted/web/http.py", line 1705, in dataReceived
    finishCallback(data[contentLength:])
  File "/home/vectoris/venv/local/lib/python2.7/site-packages/twisted/web/http.py", line 2121, in _finishRequestBody
    self.allContentReceived()
  File "/home/vectoris/venv/local/lib/python2.7/site-packages/twisted/web/http.py", line 2196, in allContentReceived
    req.requestReceived(command, path, version)
  File "/home/vectoris/venv/local/lib/python2.7/site-packages/twisted/web/http.py", line 920, in requestReceived
    self.process()
--- <exception caught here> ---
  File "/home/vectoris/venv/local/lib/python2.7/site-packages/twisted/web/server.py", line 199, in process
    self.render(resrc)
  File "/home/vectoris/venv/local/lib/python2.7/site-packages/twisted/web/server.py", line 259, in render
    body = resrc.render(self)
  File "/home/vectoris/venv/local/lib/python2.7/site-packages/twisted/web/resource.py", line 250, in render
    return m(request)
  File "sydent/http/servlets/store_invite_servlet.py", line 104, in render_POST
    sendEmail(self.sydent, "email.invite_template", address, substitutions)
  File "sydent/util/emailutils.py", line 67, in sendEmail
    raise EmailAddressException()
sydent.util.emailutils.EmailAddressException:

[anoadragon453]

I wonder if switching to python3 would solve these unicode issues?

[richvdh]

as of 687d774 (#198), it now returns a 500 rather than hanging, which is a minor improvement I guess.

[richvdh]

I wonder if switching to python3 would solve these unicode issues?

If we do it right, yes, it would probably get fixed as a side-effect of doing so.
https://github.com/matrix-org/sydent/pull/239/files#diff-2aea5cf83e41d5063d3c49a8c859b992R60 looks like it is sorting out the line causing the first exception here.

[DMRobertson]

This symptom is still visible today, see #458. That issue also has an encoding problem, but it comes later (from within smtplib). I'll close this in favour of #458.