Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Encoding::UndefinedConversionError - "\xD7" from ASCII-8BIT to UTF-8: #339

Closed
4refael opened this issue May 11, 2017 · 9 comments
Closed

Comments

@4refael
Copy link

4refael commented May 11, 2017

Starting MailCatcher
==> smtp://127.0.0.1:1025
==> http://127.0.0.1:1080
==> SMTP: Received message from '<[email protected]>' (909 bytes)
*** Error sending message through websocket: {"id"=>1, "sender"=>"<noreply@examp
le.com>", "recipients"=>["<[email protected]>"], "subject"=>"\xD7\x94\xD7\
xA8\xD7\xA9\xD7 \xD7\x94 \xD7\x91\xD7 \xD7\xAA\xD7\xA8 sugar10.local"
, "source"=>"To: [email protected]\r\nSubject: \xD7\x94\xD7\xA8\xD7\xA9\xD
7 \xD7\x94 \xD7\x91\xD7 \xD7\xAA\xD7\xA8 sugar10.local\r\nX-PHP-Origi
nating-Script: 0:Email.php\r\nUser-Agent: CodeIgniter\r\nDate: Thu, 11 May 2017
13:17:14 +0300\r\nFrom: \"test\" <[email protected]>\r\nReturn-Path: <noreply@
example.com>\r\nReply-To: \"[email protected]\" <[email protected]>\r\nX-Sen
der: [email protected]\r\nX-Mailer: xampp\r\nX-Priority: 3 (Normal)\r\nMessage
-ID: <[email protected]>\r\nMime-Version: 1.0\r\nContent-Type: text/plai
n; charset=utf-8\r\nContent-Transfer-Encoding: 8bit\r\n\r\n\xD7\xA9\xD7\x9C\xD7\
x95\xD7\x9D sadasdas,\r\n\r\n\xD7\xAA\xD7\x95\xD7\x93\xD7\x94 \xD7\xA9\xD7\xA0\x
D7\xA8\xD7\xA9\xD7\x9E\xD7\xAA \xD7\x91\xD7\x90\xD7\xAA\xD7\xA8 sugar
10.local!\r\n\r\n\r\n\xD7\xA2\xD7\x9C \xD7\x9E\xD7\xA0\xD7\xAA \xD7\x9C\xD7\x94\
xD7\xA9\xD7\x9C\xD7\x99\xD7\x9D \xD7\x90\xD7\xAA \xD7\xAA\xD7\x94\xD7\x9C\xD7\x9
9\xD7\x9A \xD7\x94\xD7\x94\xD7\xA8\xD7\xA9\xD7\x9E\xD7\x94, \xD7\xA2\xD7\x9C\xD7
\x99\xD7\x9A \xD7\x9C\xD7\x90\xD7\x9E\xD7\xAA\r\n\xD7\x90\xD7\xAA \xD7\x94\xD7\x
93\xD7\x95\xD7\x90\"\xD7\x9C \xD7\xA9\xD7\x9C\xD7\x9A \xD7\x91\xD7\x9C\xD7\x97\x
D7\x99\xD7\xA6\xD7\x94 \xD7\xA2\xD7\x9C \xD7\x94\xD7\xA7\xD7\x99\xD7\xA9\xD7\x95
\xD7\xA8 \xD7\x94\xD7\x91\xD7\x90:\r\n\r\nPlease follow the link to confirm your
 registration\r\nhttp://sugar10.local/users/confirm/ca2c2c7034\r\n\r\
n\r\n\xD7\x91\xD7\x91\xD7\xA8\xD7\x9B\xD7\x94,\r\n\xD7\xA6\xD7\x95\xD7\x95\xD7\x
AA \xD7\x90\xD7\xAA\xD7\xA8 \xD7\xA9\xD7\x95\xD7\x92\xD7\xA8 \xD7\x93\xD7\x93\xD
7\x99\r\n\r\n", "size"=>"909", "type"=>"text/plain", "created_at"=>#<DateTime: 2
017-05-11T10:17:17+00:00 ((2457885j,37037s,0n),+0s,2299161j)>}
    Exception: "\xD7" from ASCII-8BIT to UTF-8
    Backtrace:
       C:/Ruby23-x64/lib/ruby/2.3.0/json/common.rb:224:in `encode'
       C:/Ruby23-x64/lib/ruby/2.3.0/json/common.rb:224:in `generate'
       C:/Ruby23-x64/lib/ruby/2.3.0/json/common.rb:224:in `generate'
       mailcatcher (0.6.5) lib/mail_catcher/web/application.rb:69:in `block (3 l
evels) in <class:Application>'
       eventmachine (1.0.9.1) lib/em/channel.rb:43:in `block (3 levels) in push'

       eventmachine (1.0.9.1) lib/em/channel.rb:43:in `each'
       eventmachine (1.0.9.1) lib/em/channel.rb:43:in `block (2 levels) in push'

       eventmachine (1.0.9.1) lib/em/channel.rb:43:in `each'
       eventmachine (1.0.9.1) lib/em/channel.rb:43:in `block in push'
       eventmachine (1.0.9.1) lib/eventmachine.rb:256:in `schedule'
       eventmachine (1.0.9.1) lib/em/channel.rb:43:in `push'
       mailcatcher (0.6.5) lib/mail_catcher/mail.rb:57:in `block in add_message'

       eventmachine (1.0.9.1) lib/eventmachine.rb:981:in `block in run_deferred_
callbacks'
       eventmachine (1.0.9.1) lib/eventmachine.rb:978:in `times'
       eventmachine (1.0.9.1) lib/eventmachine.rb:978:in `run_deferred_callbacks
'
       eventmachine (1.0.9.1) lib/eventmachine.rb:193:in `run_machine'
       eventmachine (1.0.9.1) lib/eventmachine.rb:193:in `run'
       mailcatcher (0.6.5) lib/mail_catcher.rb:173:in `run!'
       mailcatcher (0.6.5) bin/mailcatcher:5:in `<top (required)>'
       C:/Ruby23-x64/bin/mailcatcher:22:in `load'
       C:/Ruby23-x64/bin/mailcatcher:22:in `<main>'
    Please submit this as an issue at http://github.com/sj26/mailcatcher/issues
2017-05-11 13:17:21 - Encoding::UndefinedConversionError - "\xD7" from ASCII-8BI
T to UTF-8:
        C:/Ruby23-x64/lib/ruby/2.3.0/json/common.rb:224:in `encode'
        C:/Ruby23-x64/lib/ruby/2.3.0/json/common.rb:224:in `generate'
        C:/Ruby23-x64/lib/ruby/2.3.0/json/common.rb:224:in `generate'
        C:/Ruby23-x64/lib/ruby/gems/2.3.0/gems/mailcatcher-0.6.5/lib/mail_catche
r/web/application.rb:81:in `block in <class:Application>'
        C:/Ruby23-x64/lib/ruby/gems/2.3.0/gems/sinatra-1.4.8/lib/sinatra/base.rb
:1611:in `call'
        C:/Ruby23-x64/lib/ruby/gems/2.3.0/gems/sinatra-1.4.8/lib/sinatra/base.rb
:1611:in `block in compile!'
        C:/Ruby23-x64/lib/ruby/gems/2.3.0/gems/sinatra-1.4.8/lib/sinatra/base.rb
:975:in `block (3 levels) in route!'
        C:/Ruby23-x64/lib/ruby/gems/2.3.0/gems/sinatra-1.4.8/lib/sinatra/base.rb
:994:in `route_eval'
        C:/Ruby23-x64/lib/ruby/gems/2.3.0/gems/sinatra-1.4.8/lib/sinatra/base.rb
:975:in `block (2 levels) in route!'
        C:/Ruby23-x64/lib/ruby/gems/2.3.0/gems/sinatra-1.4.8/lib/sinatra/base.rb
:1015:in `block in process_route'
        C:/Ruby23-x64/lib/ruby/gems/2.3.0/gems/sinatra-1.4.8/lib/sinatra/base.rb
:1013:in `catch'
        C:/Ruby23-x64/lib/ruby/gems/2.3.0/gems/sinatra-1.4.8/lib/sinatra/base.rb
:1013:in `process_route'
        C:/Ruby23-x64/lib/ruby/gems/2.3.0/gems/sinatra-1.4.8/lib/sinatra/base.rb
:973:in `block in route!'
        C:/Ruby23-x64/lib/ruby/gems/2.3.0/gems/sinatra-1.4.8/lib/sinatra/base.rb
:972:in `each'
        C:/Ruby23-x64/lib/ruby/gems/2.3.0/gems/sinatra-1.4.8/lib/sinatra/base.rb
:972:in `route!'
        C:/Ruby23-x64/lib/ruby/gems/2.3.0/gems/sinatra-1.4.8/lib/sinatra/base.rb
:1085:in `block in dispatch!'
        C:/Ruby23-x64/lib/ruby/gems/2.3.0/gems/sinatra-1.4.8/lib/sinatra/base.rb
:1067:in `block in invoke'
        C:/Ruby23-x64/lib/ruby/gems/2.3.0/gems/sinatra-1.4.8/lib/sinatra/base.rb
:1067:in `catch'
        C:/Ruby23-x64/lib/ruby/gems/2.3.0/gems/sinatra-1.4.8/lib/sinatra/base.rb
:1067:in `invoke'
        C:/Ruby23-x64/lib/ruby/gems/2.3.0/gems/sinatra-1.4.8/lib/sinatra/base.rb
:1082:in `dispatch!'
        C:/Ruby23-x64/lib/ruby/gems/2.3.0/gems/sinatra-1.4.8/lib/sinatra/base.rb
:907:in `block in call!'
        C:/Ruby23-x64/lib/ruby/gems/2.3.0/gems/sinatra-1.4.8/lib/sinatra/base.rb
:1067:in `block in invoke'
        C:/Ruby23-x64/lib/ruby/gems/2.3.0/gems/sinatra-1.4.8/lib/sinatra/base.rb
:1067:in `catch'
        C:/Ruby23-x64/lib/ruby/gems/2.3.0/gems/sinatra-1.4.8/lib/sinatra/base.rb
:1067:in `invoke'
        C:/Ruby23-x64/lib/ruby/gems/2.3.0/gems/sinatra-1.4.8/lib/sinatra/base.rb
:907:in `call!'
        C:/Ruby23-x64/lib/ruby/gems/2.3.0/gems/sinatra-1.4.8/lib/sinatra/base.rb
:895:in `call'
        C:/Ruby23-x64/lib/ruby/gems/2.3.0/gems/rack-protection-1.5.3/lib/rack/pr
otection/xss_header.rb:18:in `call'
        C:/Ruby23-x64/lib/ruby/gems/2.3.0/gems/rack-protection-1.5.3/lib/rack/pr
otection/path_traversal.rb:16:in `call'
        C:/Ruby23-x64/lib/ruby/gems/2.3.0/gems/rack-protection-1.5.3/lib/rack/pr
otection/json_csrf.rb:18:in `call'
        C:/Ruby23-x64/lib/ruby/gems/2.3.0/gems/rack-protection-1.5.3/lib/rack/pr
otection/base.rb:49:in `call'
        C:/Ruby23-x64/lib/ruby/gems/2.3.0/gems/rack-protection-1.5.3/lib/rack/pr
otection/base.rb:49:in `call'
        C:/Ruby23-x64/lib/ruby/gems/2.3.0/gems/rack-protection-1.5.3/lib/rack/pr
otection/frame_options.rb:31:in `call'
        C:/Ruby23-x64/lib/ruby/gems/2.3.0/gems/rack-1.6.6/lib/rack/nulllogger.rb
:9:in `call'
        C:/Ruby23-x64/lib/ruby/gems/2.3.0/gems/rack-1.6.6/lib/rack/head.rb:13:in
 `call'
        C:/Ruby23-x64/lib/ruby/gems/2.3.0/gems/sinatra-1.4.8/lib/sinatra/base.rb
:182:in `call'
        C:/Ruby23-x64/lib/ruby/gems/2.3.0/gems/sinatra-1.4.8/lib/sinatra/base.rb
:2013:in `call'
        C:/Ruby23-x64/lib/ruby/gems/2.3.0/gems/sinatra-1.4.8/lib/sinatra/base.rb
:1487:in `block in call'
        C:/Ruby23-x64/lib/ruby/gems/2.3.0/gems/sinatra-1.4.8/lib/sinatra/base.rb
:1787:in `synchronize'
        C:/Ruby23-x64/lib/ruby/gems/2.3.0/gems/sinatra-1.4.8/lib/sinatra/base.rb
:1487:in `call'
        C:/Ruby23-x64/lib/ruby/gems/2.3.0/gems/rack-1.6.6/lib/rack/urlmap.rb:66:
in `block in call'
        C:/Ruby23-x64/lib/ruby/gems/2.3.0/gems/rack-1.6.6/lib/rack/urlmap.rb:50:
in `each'
        C:/Ruby23-x64/lib/ruby/gems/2.3.0/gems/rack-1.6.6/lib/rack/urlmap.rb:50:
in `call'
        C:/Ruby23-x64/lib/ruby/gems/2.3.0/gems/rack-1.6.6/lib/rack/builder.rb:15
3:in `call'
        C:/Ruby23-x64/lib/ruby/gems/2.3.0/gems/mailcatcher-0.6.5/lib/mail_catche
r/web.rb:19:in `call'
        C:/Ruby23-x64/lib/ruby/gems/2.3.0/gems/thin-1.5.1/lib/thin/connection.rb
:81:in `block in pre_process'
        C:/Ruby23-x64/lib/ruby/gems/2.3.0/gems/thin-1.5.1/lib/thin/connection.rb
:79:in `catch'
        C:/Ruby23-x64/lib/ruby/gems/2.3.0/gems/thin-1.5.1/lib/thin/connection.rb
:79:in `pre_process'
        C:/Ruby23-x64/lib/ruby/gems/2.3.0/gems/thin-1.5.1/lib/thin/connection.rb
:54:in `process'
        C:/Ruby23-x64/lib/ruby/gems/2.3.0/gems/thin-1.5.1/lib/thin/connection.rb
:39:in `receive_data'
        C:/Ruby23-x64/lib/ruby/gems/2.3.0/gems/eventmachine-1.0.9.1/lib/eventmac
hine.rb:193:in `run_machine'
        C:/Ruby23-x64/lib/ruby/gems/2.3.0/gems/eventmachine-1.0.9.1/lib/eventmac
hine.rb:193:in `run'
        C:/Ruby23-x64/lib/ruby/gems/2.3.0/gems/mailcatcher-0.6.5/lib/mail_catche
r.rb:173:in `run!'
        C:/Ruby23-x64/lib/ruby/gems/2.3.0/gems/mailcatcher-0.6.5/bin/mailcatcher
:5:in `<top (required)>'
        C:/Ruby23-x64/bin/mailcatcher:22:in `load'
        C:/Ruby23-x64/bin/mailcatcher:22:in `<main>'

@jgrau
Copy link

jgrau commented May 11, 2017

Yup. I am having the same issue.

@caugner
Copy link

caugner commented Jun 1, 2017

Similar issue here, running mailcatcher using Ruby 2.3.3 (x64) on Windows 10:

    Exception: "\xC3" from ASCII-8BIT to UTF-8
    Backtrace:
       json (2.1.0) lib/json/common.rb:224:in `encode'
       json (2.1.0) lib/json/common.rb:224:in `generate'
       json (2.1.0) lib/json/common.rb:224:in `generate'
       mailcatcher (0.6.5) lib/mail_catcher/web/application.rb:69:in `block (3 levels) in <class:Application>'
       eventmachine (1.0.9.1) lib/em/channel.rb:43:in `block (3 levels) in push'
       eventmachine (1.0.9.1) lib/em/channel.rb:43:in `each'
       eventmachine (1.0.9.1) lib/em/channel.rb:43:in `block (2 levels) in push'
       eventmachine (1.0.9.1) lib/em/channel.rb:43:in `each'
       eventmachine (1.0.9.1) lib/em/channel.rb:43:in `block in push'
       eventmachine (1.0.9.1) lib/eventmachine.rb:256:in `schedule'
       eventmachine (1.0.9.1) lib/em/channel.rb:43:in `push'
       mailcatcher (0.6.5) lib/mail_catcher/mail.rb:57:in `block in add_message'
       eventmachine (1.0.9.1) lib/eventmachine.rb:981:in `block in run_deferred_callbacks'
       eventmachine (1.0.9.1) lib/eventmachine.rb:978:in `times'
       eventmachine (1.0.9.1) lib/eventmachine.rb:978:in `run_deferred_callbacks'
       eventmachine (1.0.9.1) lib/eventmachine.rb:193:in `run_machine'
       eventmachine (1.0.9.1) lib/eventmachine.rb:193:in `run'
       mailcatcher (0.6.5) lib/mail_catcher.rb:173:in `run!'
       mailcatcher (0.6.5) bin/mailcatcher:5:in `<top (required)>'
       C:/Ruby23-x64/bin/mailcatcher:22:in `load'
       C:/Ruby23-x64/bin/mailcatcher:22:in `<main>'

@fnxnet
Copy link

fnxnet commented Aug 9, 2017

Same here on ruby 2.1.5p273 (2014-11-13) [x86_64-linux-gnu]

Exception: "\xC5" from ASCII-8BIT to UTF-8
    Backtrace:
       /usr/lib/ruby/2.1.0/json/common.rb:223:in `encode'
       /usr/lib/ruby/2.1.0/json/common.rb:223:in `generate'
       /usr/lib/ruby/2.1.0/json/common.rb:223:in `generate'
       mailcatcher (0.6.5) lib/mail_catcher/web/application.rb:69:in `block (3 levels) in <class:Application>'
       eventmachine (1.0.9.1) lib/em/channel.rb:43:in `call'
       eventmachine (1.0.9.1) lib/em/channel.rb:43:in `block (3 levels) in push'
       eventmachine (1.0.9.1) lib/em/channel.rb:43:in `each'
       eventmachine (1.0.9.1) lib/em/channel.rb:43:in `block (2 levels) in push'
       eventmachine (1.0.9.1) lib/em/channel.rb:43:in `each'
       eventmachine (1.0.9.1) lib/em/channel.rb:43:in `block in push'
       eventmachine (1.0.9.1) lib/eventmachine.rb:256:in `call'
       eventmachine (1.0.9.1) lib/eventmachine.rb:256:in `schedule'
       eventmachine (1.0.9.1) lib/em/channel.rb:43:in `push'
       mailcatcher (0.6.5) lib/mail_catcher/mail.rb:57:in `block in add_message'
       eventmachine (1.0.9.1) lib/eventmachine.rb:981:in `call'
       eventmachine (1.0.9.1) lib/eventmachine.rb:981:in `block in run_deferred_callbacks'
       eventmachine (1.0.9.1) lib/eventmachine.rb:978:in `times'
       eventmachine (1.0.9.1) lib/eventmachine.rb:978:in `run_deferred_callbacks'
       eventmachine (1.0.9.1) lib/eventmachine.rb:193:in `run_machine'
       eventmachine (1.0.9.1) lib/eventmachine.rb:193:in `run'
       mailcatcher (0.6.5) lib/mail_catcher.rb:173:in `run!'
       mailcatcher (0.6.5) bin/mailcatcher:5:in `<top (required)>'
       /usr/local/bin/mailcatcher:23:in `load'
       /usr/local/bin/mailcatcher:23:in `<main>'
    Please submit this as an issue at http://github.com/sj26/mailcatcher/issues

@znz
Copy link

znz commented Aug 10, 2017

I think it is bug of mail sender, not mailcatcher.
see #322

@kdiogenes
Copy link

I use UTF-8 in from, like the following code in ActionMailer::Base:

default from: 'leão - my latin america brand <[email protected]>'

I did the following commit only to show a possible fix kdiogenes@b2520ff

What do you think about it @sj26? If you think it's acceptable I can prepare a PR.

Best regards!

@znz
Copy link

znz commented Aug 1, 2018

I think receiving mails with 8bit is root of problems.
Because SMTP server of mailcatcher does not respond with 8BITMIME to the EHLO command, it should reject mail with 8bit from bad mail clients.
I think sane mail clients re-send mail with 7bit encoding.
It can help to reduce mail clients that violate RFCs.

@kdiogenes
Copy link

kdiogenes commented Aug 1, 2018

@znz my email body is encoded with 7bit (I don't do anything special to have it, ActionMailer already send it ok), but the from header field keeps giving error, this is my from header:

From: =?UTF-8?Q?AlfaCon_-_AlfaCon_Concursos_P=C3=BAblicos_<no-reply@alf?=
 =?UTF-8?Q?a-dev.com.br:3000>?=

The body have the proper charset and encoding:

Content-Type: multipart/mixed;
 boundary="--==_mimepart_5b61a61b9d9a5_159c2b07c5d3d950208f3";
 charset=UTF-8
Content-Transfer-Encoding: 7bit


----==_mimepart_5b61a61b9d9a5_159c2b07c5d3d950208f3
Content-Type: text/html;
 charset=UTF-8
Content-Transfer-Encoding: quoted-printable

And in the body I have this string <p>Obrigado por comprar no AlfaCon Concursos P=C3=BAblicos</p>

If I remove the accent from the from header I get no exception and mailcatcher show the email properly.

So, my question is basically how I can use accents in from field?

@djomlastic
Copy link

I'm not a ruby dev, but I believe this is related. Using mailcatcher in a vagrant box that serves as a magento development server. If email contains some non-ascii characters (german umlauts, euro symbol), it doesn't get sent. Apache error log:

/var/lib/gems/2.5.0/gems/mail-2.7.1/lib/mail/message.rb:2032:in `match': invalid byte sequence in US-ASCII (ArgumentError)
	from /var/lib/gems/2.5.0/gems/mail-2.7.1/lib/mail/message.rb:2032:in `match'
	from /var/lib/gems/2.5.0/gems/mail-2.7.1/lib/mail/message.rb:2032:in `set_envelope_header'
	from /var/lib/gems/2.5.0/gems/mail-2.7.1/lib/mail/message.rb:2132:in `init_with_string'
	from /var/lib/gems/2.5.0/gems/mail-2.7.1/lib/mail/message.rb:137:in `initialize'
	from /var/lib/gems/2.5.0/gems/mail-2.7.1/lib/mail/mail.rb:51:in `new'
	from /var/lib/gems/2.5.0/gems/mail-2.7.1/lib/mail/mail.rb:51:in `new'
	from /var/lib/gems/2.5.0/gems/mailcatcher-0.7.1/bin/catchmail:59:in `<top (required)>'
	from /usr/local/bin/catchmail:23:in `load'
	from /usr/local/bin/catchmail:23:in `<main>

So match function's argument is US-ASCII encoded, which is weird. Ruby / encoding related stuff:

vagrant@syone:~$ ruby -v
ruby 2.5.1p57 (2018-03-29 revision 63029) [x86_64-linux-gnu]
vagrant@syone:~$ ruby -e 'p Encoding.default_external'
#<Encoding:UTF-8>
vagrant@syone:~$ locale
LANG=C.UTF-8
LANGUAGE=
LC_CTYPE="C.UTF-8"
LC_NUMERIC=C.UTF-8
LC_TIME=en_US.UTF-8
LC_COLLATE="C.UTF-8"
LC_MONETARY=C.UTF-8
LC_MESSAGES="C.UTF-8"
LC_PAPER=C.UTF-8
LC_NAME=C.UTF-8
LC_ADDRESS=C.UTF-8
LC_TELEPHONE=C.UTF-8
LC_MEASUREMENT=C.UTF-8
LC_IDENTIFICATION=C.UTF-8
LC_ALL=

But if I print out Encoding.default_external from catchmail , it really uses US-ASCII. Setting Encoding.default_external = en_US.UTF-8 within catchmail solves the problem, also changing line 59 within catchmail to message = Mail.new($stdin.read.force_encoding('UTF-8')) solves it.
So I ended up changing sendmail_path within php.ini to:
sendmail_path = "LANG=en_US.UTF-8 /usr/bin/env catchmail -f vagrant@syone"
That works, but I still don't understand why catchmail thinks external encoding is US-ASCII.

@sj26
Copy link
Owner

sj26 commented Feb 22, 2021

MailCatcher does not advertise itself as 8-bit capable, and so mail clients should be sending mail in ascii compatible encodings. A feature request for 8 bit transfer encodings is now here, please follow if you're interested: #445

@sj26 sj26 closed this as completed Feb 22, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

8 participants