The other day I was writing a Chrome extension that sends data to a server.

Everything was going fine until I added authentication and cookie-based sessions to the backend. Alls of a sudden the extension’s POST requests to the server did not include my cookies.

When those POSTs arrived at the server (sans cookies), the backend had no choice but to respond with an HTTP 403 Forbidden error.

After all, the only barrier that divides the unwashed masses from gloriously-authenticated users is having the right session cookie. Without those cookies, none of the requests coming from the extension went through.

Somehow I needed to figure out why Chrome didn’t send the cookies to my server’s backend. But how?

A yak is sensed in the mind first

I tried all the obvious things:

• Permutations of cookie settings and flags in the backend and the Chrome extension code, on the off-chance something was misconfigured, or the extension had triggered some strict security policy that prevented Chrome from sending the cookies ²
• Permutations of extension settings in manifest.json: broader permissions, loosened restrictions, and such forth
• Searched StackOverflow, mailing-lists, Googled for like an hour
• Read the Chrome dev docs, especially the extension XHR docs

But the problem persisted. My cookies were still missing.

There didn’t appear to be anything I could do from “client-space” to make Chrome send the cookies from the extension. And I couldn’t glean any useful bits from the documentation or elsewhere.

The tricky bit is that an extension has only partial, indirect control over whether cookies are automagically sent for requests from the extension to a domain ¹. There’s an opaque, non-queryable (AFAICT) bit of policy inside Chrome that decides whether a domain’s pre-existing cookies are sent with requests.

The only feedback that Chrome provided was the absence of cookies in the DevTools network inspector, which wasn’t useful for formulating a diagnostic…

So I figured it was time to trace the problem into the Chrome source code.

An adult yak can weigh up to 1,300 pounds

Chrome is a behemoth codebase. It has somewhere between 5 and 7 million lines of code written by many, many hands.

Contrary to popular belief, yaks (and their manure) have little odor

Luckily, I only needed to understand a few dozen lines of Chrome source to find the root of the problem.

I built Chromium on a spare Ubuntu PC, guessing it would be the simplest platform to build on (and it was ³).

With a few modifications, Bojan Komazec’s instructions for building Chromium on Ubuntu did the trick:

• The version of Ninja that comes with Ubuntu Xenial (Ninja v1.5.1) was older than the one required to build Chromium (Ninja 1.8.2), so:

  export PATH="$PATH:${HOME}/dev/depot_tools"
  

  should be:

  export PATH="${HOME}/dev/depot_tools:$PATH"
  

  so that the Chromium-bundled Ninja supercedes the Ubuntu-bundled Ninja

• To generate debug symbols, I omitted the line:

  emove_webcore_debug_symbols=true
  

  (which is a typo anyways; this should instead be: remove_webcore_debug_symbols=true if you indeed wanted to remove webcore debug symbols)

Building Chromium took 6-7 hours (!) on an old laptop with an SSD and a quad-core i7 CPU (with all CPUs pegged near 100%). I instantly regretted not putting all the Chromium source on a RAMdisk to reap the faster build time. But I will just have to live with that regret. Along with all the others.

In addition to being a source of meat, milk and textiles in Tibet, the Yak is a beast of burden

Next came the hard part - and where I hit a wall.

The first challenge was figuring out where to set a breakpoint.

The question was: out of those 5 - 7 million lines of code, which ones relate to my problem?

I searched through the Chromium source code for mentions of “cookie” and found some useful constant string literals. Then I searched further for uses of those literals, and found half-of-a-dozen candidate functions where the cookies might be injected into a request. With a handful of functions in…hand, I set to running Chrome under GDB.

The second challenge was attaching to the correct Chrome process that ran with all the correct settings.

You see, the thing about debuggers is that they’re difficult to use in the presence of complicated threading. …And multiple processes. Chrome has both complicated threading and multiple processes, plus additional complexities like GPU rendering and sandboxing.

So I struggled with getting GDB to play nice with Chromium for a bit until hitting upon the solution. The magical words to start Chromium under GDB for easier debugging were:

$ gdb -tui --args out/Default/chrome \
    --disable-seccomp-sandbox \
    --no-sandbox \
    --single-process \
    --disable-gpu

That command starts GDB with the TUI text-based interface, disables sandboxing and the GPU, and forces Chromium to run all tabs in the same process.

When GDB was ready to accept commands, I started Chromium running:

gdb$ run

Then I waited for GDB to show the prompt again, at which point I set a breakpoint in Chromium code, and told GDB to continue executing the debuggee:

gdb$ b xml_http_request.cc:998
gdb$ continue

I switched over to Chromium, did the thing that made my extension send the POST request, and the breakpoint was hit. Excellent! And I could also step through code. Eureka!

…Now I simply needed to inspect some local variables and I could figure out why the cookies weren’t sent…

gdb$ print http_body
No symbol "http_body" in current context.

Hmmmmmmmmmmmmmmm.........

All was not going to be so smooth in the land of Chromium debugging. Now I had a problem to solve.

The Dzo is an infertile yak-cattle hybrid, like a mule

So I could set breakpoints and do source-level debugging, but I could not inspect variables.

Something was amiss…

As we all learn before entering kindergarten, the .debug_info section of an ELF shared library is where the DWARF debug metadata is stored. And we also know—because it has been embedded inside our DNA as instinctual knowledge for millenia—that the DWARF metadata contains a Debugging Information Entry (DIE) for each function, variable, and parameter in the library.

And as even the smallest micro-organism could conclude: if there are no DIEs for the variables in a shared library, then GDB cannot inspect any variables.

Armed with such trivial common-place information (I don’t know why I even bother to mention it!), I set about confirming this theory. I asked GDB which .so contained the XMLHttpRequest::CreateRequest() method:

$gdb info symbol XMLHttpRequest::CreateRequest
blink::XMLHttpRequest::CreateRequest(scoped_refptr<blink::EncodedFormData>, blink::ExceptionState&) in section .text of /home/devon/Projects/DebuggingChromiumOnUbuntu/chromium/src/out/Default/./libblink_core.so

…and then “objdump-grepped” the DWARF metadata, searching for variable DIEs in the libblink_core.so library:

objdump --dwarf=info out/Default/libblink_core.so | grep DW_TAG_variable

And the theory was confirmed. There were no DW_TAG_variable DIEs in the shared library!

But there were 3 other kinds of DIEs in the library:

$ objdump --dwarf=info out/Default/libblink_core.so | grep -o DW_TAG_.* | sort | uniq
DW_TAG_compile_unit)
DW_TAG_inlined_subroutine)
DW_TAG_subprogram)

…which further confirmed the behavior we saw in GDB earlier: we could set source-level breakpoints, but could not inspect variables.

“A yak will starve unless brought to a place where there is grass”

At this point it was clear that the Chromium binaries that I compiled way back at the beginning were missing the all-important DWARF variable and parameter debug metadata. In fact, they probably were not even built using the compiler flags that generate that part of the DWARF metadata.

So all we need to do is re-compile with that metadata, right?

But how to do that? As I later found out, this is explained in the Chromium developer docs. But I didn’t know about those instructions initially, so I had to dive into the bits of Chromium that I did understand up to this point. (Also, the headings should have made it clear what this journey is all about by now…)

First, I needed to see the parameters that were passed to clang to generate debug metadata for xml_http_request.cc. I touch’ed xml_http_request.cc, to force Ninja to recompile it:

$ touch third_party/blink/renderer/core/xmlhttprequest/xml_http_request.cc

Then I used gn to figure out if there was a sub-component to recompile just xml_http_request.cc (to save time while iterating):

$ gn ls out/Default | grep -i xmlhttp
//third_party/blink/renderer/core/xmlhttprequest:xmlhttprequest

Indeed there was exactly one such sub-component:

//third_party/blink/renderer/core/xmlhttprequest:xmlhttprequest

So I told Ninja to compile only that xmlhttprequest module, and show all executed commands (-v) while compiling:

$ ninja -v -C out/Default third_party/blink/renderer/core/xmlhttprequest:xmlhttprequest

The output from ninja was quite verbose, but held the clue for why GDB could set breakpoints but not inspect variables:

ninja: Entering directory `out/Default'
[1/2] ../../third_party/llvm-build/Release+Asserts/bin/clang++ -MMD -MF obj/third_party/blink/renderer/core/xmlhttprequest/xmlhttprequest/xml_http_request.o.d -DV8_DEPRECATION_WARNINGS -DUSE_UDEV -DUSE_AURA=1 -DUSE_GLIB=1 -DUSE_NSS_CERTS=1 -DUSE_X11=1 -DFULL_SAFE_BROWSING -DSAFE_BROWSING_CSD -DSAFE_BROWSING_DB_LOCAL -DCHROMIUM_BUILD -DFIELDTRIAL_TESTING_ENABLED -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -DCR_CLANG_REVISION=\"329921-1\" -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -DCOMPONENT_BUILD -DCR_LIBCXX_REVISION=329375 -DCR_LIBCXXABI_REVISION=329629 -DCR_SYSROOT_HASH=4e7db513b0faeea8fb410f70c9909e8736f5c0ab -D_DEBUG -DDYNAMIC_ANNOTATIONS_ENABLED=1 -DWTF_USE_DYNAMIC_ANNOTATIONS=1 -D_GLIBCXX_DEBUG=1 -DBLINK_CORE_IMPLEMENTATION=1 -DGLIB_VERSION_MAX_ALLOWED=GLIB_VERSION_2_32 -DGLIB_VERSION_MIN_REQUIRED=GLIB_VERSION_2_26 -DWEBP_EXTERN=extern -DGL_GLEXT_PROTOTYPES -DUSE_GLX -DUSE_EGL -DBLINK_IMPLEMENTATION=1 -DINSIDE_BLINK -DU_USING_ICU_NAMESPACE=0 -DU_ENABLE_DYLOAD=0 -DICU_UTIL_DATA_IMPL=ICU_UTIL_DATA_FILE -DUCHAR_TYPE=uint16_t -DGOOGLE_PROTOBUF_NO_RTTI -DGOOGLE_PROTOBUF_NO_STATIC_INITIALIZER -DHAVE_PTHREAD -DPROTOBUF_USE_DLLS -DBORINGSSL_SHARED_LIBRARY -DSK_IGNORE_LINEONLY_AA_CONVEX_PATH_OPTS -DSK_HAS_PNG_LIBRARY -DSK_HAS_WEBP_LIBRARY -DSK_HAS_JPEG_LIBRARY -DSKIA_DLL -DGR_GL_IGNORE_ES3_MSAA=0 -DSK_SUPPORT_GPU=1 -DWTF_USE_DYNAMIC_ANNOTATIONS=1 -DWTF_USE_WEBAUDIO_FFMPEG=1 -DWTF_USE_DEFAULT_RENDER_THEME=1 -DLEVELDB_PLATFORM_CHROMIUM=1 -DUSE_LIBJPEG_TURBO=1 -DUSING_V8_SHARED -DV8_ENABLE_CHECKS -DUSING_V8_SHARED -DV8_ENABLE_CHECKS -DLIBXSLT_STATIC -I../.. -Igen -I../../third_party/blink/renderer -I../../third_party/blink -Igen/third_party/blink/renderer -Igen/third_party/blink -I../../third_party/libwebp/src -I../../third_party/khronos -I../../gpu -I../../third_party/libyuv/include -I../../third_party/blink/renderer -Igen/third_party/blink/renderer -I../../third_party/ced/src -I../../third_party/icu/source/common -I../../third_party/icu/source/i18n -I../../third_party/protobuf/src -I../../third_party/protobuf/src -Igen/protoc_out -I../../third_party/boringssl/src/include -I../../skia/config -I../../skia/ext -I../../third_party/skia/include/c -I../../third_party/skia/include/config -I../../third_party/skia/include/core -I../../third_party/skia/include/effects -I../../third_party/skia/include/encode -I../../third_party/skia/include/gpu -I../../third_party/skia/include/images -I../../third_party/skia/include/lazy -I../../third_party/skia/include/pathops -I../../third_party/skia/include/pdf -I../../third_party/skia/include/pipe -I../../third_party/skia/include/ports -I../../third_party/skia/include/utils -I../../third_party/skia/third_party/skcms -I../../third_party/skia/src/gpu -I../../third_party/skia/src/sksl -I../../third_party/angle/include -I../../third_party/angle/src/common/third_party/base -Igen/angle -I../../third_party/libwebm/source -I../../third_party/leveldatabase -I../../third_party/leveldatabase/src -I../../third_party/leveldatabase/src/include -I../../third_party/libjpeg_turbo -I../../third_party/blink -Igen/third_party/blink -I../../v8/include -Igen/v8/include -I../../third_party/iccjpeg -I../../third_party/libpng -I../../third_party/zlib -I../../third_party/ots/include -I../../v8/include -Igen/v8/include -I../../third_party/libxml/src/include -I../../third_party/libxml/linux/include -I../../third_party/libxslt/src -I../../third_party/snappy/src -I../../third_party/snappy/linux -fno-strict-aliasing -fmerge-all-constants --param=ssp-buffer-size=4 -fstack-protector -Wno-builtin-macro-redefined -D__DATE__= -D__TIME__= -D__TIMESTAMP__= -funwind-tables -fPIC -pipe -B../../third_party/binutils/Linux_x64/Release/bin -pthread -fcolor-diagnostics -Xclang -mllvm -Xclang -instcombine-lower-dbg-declare=0 -Xclang -mllvm -Xclang -fast-isel-sink-local-values=1 -no-canonical-prefixes -m64 -march=x86-64 -Wall -Werror -Wextra -Wimplicit-fallthrough -Wthread-safety -Wno-missing-field-initializers -Wno-unused-parameter -Wno-c++11-narrowing -Wno-covered-switch-default -Wno-unneeded-internal-declaration -Wno-inconsistent-missing-override -Wno-undefined-var-template -Wno-nonportable-include-path -Wno-address-of-packed-member -Wno-unused-lambda-capture -Wno-user-defined-warnings -Wno-enum-compare-switch -Wno-null-pointer-arithmetic -Wno-ignored-pragma-optimize -Wno-return-std-move -O0 -fno-omit-frame-pointer -fvisibility=hidden -Xclang -load -Xclang ../../third_party/llvm-build/Release+Asserts/lib/libFindBadConstructs.so -Xclang -add-plugin -Xclang find-bad-constructs -Xclang -plugin-arg-find-bad-constructs -Xclang no-realpath -Xclang -plugin-arg-find-bad-constructs -Xclang check-enum-max-value -Xclang -plugin-arg-find-bad-constructs -Xclang check-ipc -Wheader-hygiene -Wstring-conversion -Wtautological-overlap-compare -Wexit-time-destructors -Xclang -load -Xclang ../../third_party/llvm-build/Release+Asserts/lib/libBlinkGCPlugin.so -Xclang -add-plugin -Xclang blink-gc-plugin -Wglobal-constructors -g1 -isystem../../build/linux/debian_sid_amd64-sysroot/usr/include/glib-2.0 -isystem../../build/linux/debian_sid_amd64-sysroot/usr/lib/x86_64-linux-gnu/glib-2.0/include -Wno-header-guard -isystem../../build/linux/debian_sid_amd64-sysroot/usr/include/nss -isystem../../build/linux/debian_sid_amd64-sysroot/usr/include/nspr -DLIBXML_STATIC= -Wno-undefined-bool-conversion -Wno-tautological-undefined-compare -std=gnu++14 -fno-exceptions -fno-rtti -nostdinc++ -isystem../../buildtools/third_party/libc++/trunk/include -isystem../../buildtools/third_party/libc++abi/trunk/include --sysroot=../../build/linux/debian_sid_amd64-sysroot -fvisibility-inlines-hidden -c ../../third_party/blink/renderer/core/xmlhttprequest/xml_http_request.cc -o obj/third_party/blink/renderer/core/xmlhttprequest/xmlhttprequest/xml_http_request.o
[2/2] touch obj/third_party/blink/renderer/core/xmlhttprequest/xmlhttprequest.stamp

Aha! You might have missed it, but the all-important clue we’re looking for is nestled in that 6000-character-long wall of text:

-g1

The -g1 flag tells Clang to “emit debug line number tables only”.

Eureka! (yet again). That must be why the variable debug metadata was missing from the shared library.

In central China’s Gansu Province, nomads can buy insurance policies for their sheep and yaks

I edited the bottom bit of src/third_party/blink/renderer/config.gni to enable symbols for the Blink renderer module:

#if (blink_symbol_level == 2) {
#  blink_symbols_config = [ "//build/config/compiler:symbols" ]
#} else if (blink_symbol_level == 1) {
#  blink_symbols_config = [ "//build/config/compiler:minimal_symbols" ]
#} else if (blink_symbol_level == 0) {
#  blink_symbols_config = [ "//build/config/compiler:no_symbols" ]
#} else {
#  blink_symbols_config = [ "//build/config/compiler:default_symbols" ]
#}
blink_symbols_config = [ "//build/config/compiler:symbols" ]

There’s definitely a cleaner way to do that, but this was a quick hack that worked.

Just like other cows, yak spends a lot of time in re-chewing its food before final swallowing

I touch’d the xml_http_request.cc file again, then did a ninja build of just the xmlhttprequest module with the verbose option…

And indeed the -g1 switch to Clang was now replaced by the -g2 switch.

To confirm that the DWARF metadata was in fact generated, I ran objdump again. But this time I ran it on xml_http_request.dwo, instead of the shared library .so file. The .dwo file contains the debug symbols, and is generated separately from the .so file, due to the DWARF “Fission” feature:

$ objdump --dwarf=info out/Default/obj/third_party/blink/renderer/core/xmlhttprequest/xmlhttprequest/xml_http_request.dwo | grep -o "DW_TAG_[^\)]*" | sort | uniq
DW_TAG_GNU_template_parameter_pack
DW_TAG_array_type
DW_TAG_base_type
DW_TAG_class_type
DW_TAG_compile_unit
DW_TAG_const_type
DW_TAG_enumeration_type
DW_TAG_enumerator
DW_TAG_formal_parameter
DW_TAG_imported_declaration
DW_TAG_imported_module
DW_TAG_inheritance
DW_TAG_inlined_subroutine
DW_TAG_lexical_block
DW_TAG_member
DW_TAG_namespace
DW_TAG_pointer_type
DW_TAG_ptr_to_member_type
DW_TAG_reference_type
DW_TAG_restrict_type
DW_TAG_rvalue_reference_type
DW_TAG_structure_type
DW_TAG_subprogram
DW_TAG_subrange_type
DW_TAG_subroutine_type
DW_TAG_template_type_param
DW_TAG_template_value_param
DW_TAG_typedef
DW_TAG_union_type
DW_TAG_unspecified_parameters
DW_TAG_unspecified_type
DW_TAG_variable
DW_TAG_volatile_type

In fact, Clang was now emitting a ton of useful debug metadata that we didn’t see before.

To wrap things up, I compiled the top-level chromium project:

$ ninja -C out/Default chrome

And then I confirmed that the generated libblink_core.so now contained the DWARF variables metadata.

(I didn't copy the command I used or its output, so you'll have to imagine it here!)

Mating season of yaks takes place during the September. Pregnancy lasts 9 months and ends with one baby. Female gives birth every second year

Finally, my custom-built Chromium was ready to debug. I started Chromium under GDB, as before:

$ gdb -tui --args out/Default/chrome \
    --disable-seccomp-sandbox \
    --no-sandbox \
    --single-process \
    --disable-gpu

Then waited for GDB to show a prompt:

gdb$ run
gdb$ b xml_http_request.cc:998
gdb$ continue

…and when Chromium hit that breakpoint, I crossed my fingers before typing:

gdb$ print this
$1 = ((anonymous namespace)::XMLHttpRequest *) 0x301a072aa318

gdb$ print url_
warning: RTTI symbol not found for class 'blink::XMLHttpRequest'
$2 = {is_valid_ = 0x1, protocol_is_in_http_family_ = 0x1, protocol_ = {impl_ = {ptr_ = 0x3cf36f8101c0}}, parsed_ = {scheme = {begin = 0x0, len = 0x5}, username = {begin = 0x0, len = 0xffffffff}, password = {begin = 0x0, len = 0xffffffff}, host = {begin = 0x8, len = 0xd}, port = {begin = 0x0, len = 0xffffffff}, path = {begin = 0x15, len = 0xa}, query = {begin = 0x20, len = 0x7}, ref = {begin = 0x0, len = 0xffffffff}, potentially_dangling_markup = 0x0, inner_parsed_ = 0x0}, string_ = {impl_ = {ptr_ = 0x3cf36f880e48}}, inner_url_ = {__ptr_ = {<(anonymous namespace)::(anonymous namespace)::__compressed_pair_elem<blink::KURL*, 0, false>> = {__value_ = 0x0}, <(anonymous namespace)::(anonymous namespace)::__compressed_pair_elem<std::__1::default_delete<blink::KURL>, 1, true>> = {<(anonymous namespace)::(anonymous namespace)::default_delete<blink::KURL>> = {<No data fields>}, <No data fields>}, <No data fields>}}}

Success!!! GDB found the type debug metadata, and I could finally inspect Chromium variables.

Hooves of yaks are split, which facilitate movement across the rocky and icy terrains

To make it easier to print Chromium-specific types, I create a ~/.gdbinit file with some Python code to import pretty-printers bundled with the Chromium source code:

python
import sys
sys.path.insert(0, "/home/devon/Projects/DebuggingChromiumOnUbuntu/chromium/src/third_party/WebKit/Tools/gdb/")
import webkit
sys.path.insert(0, "/home/devon/Projects/DebuggingChromiumOnUbuntu/chromium/src/tools/gdb/")
import gdb_chrome

The punchline

There’s an odd thing that often happens when delving deep into a complex system [3^]. Somehow the solution appears while searching for some minute detail of the system. And the solution makes all the work seem kind of ridiculous.

But that’s just how it works. The investment isn’t wasted, it’s just not proportional to the root cause of the problem.

That’s exactly what happened in the case of this cookie issue I’d been tracking down. I found Chrome bug 796480: “Requests from content scripts doesn’t send samesite cookies”. I was searching for some esoteric thing about the Chrome source code, and luckily hit upon that semi-related thread.

As it turns out, I had indeed configured the server’s cookies as ‘samesite’.

I disabled that setting, and then everything worked. My cookies were now being sent from the browser extension to the backend -_-

In summary, I found my cookies. Chrome was hiding them.

Footnotes