dhilst

dlopen will bite you

Meat author: Yes, this post was generated mostly by GPT5, but I’m adding my own notes to make things clearer and funnier. Soo all italic stuff is me (an human) “speaking”

Summing up I was trying to get a SDL3 hello world working, I fell alone and what ppl do when they fell alone? Yes, they write indie games, so do I.

So I (as most developers in 2025) went to GPT and asked for a SDL3 hello world, I’m using C++ and conan in the work so I decided to go with it, this is how it went…

Go GPT, tell us…

When dlopen() Bites: Debugging SDL3 + Wayland ABI Mismatch with Conan on Linux

TL;DR
I “the author, GPT has no day to lost, it is imortal” lost a day to a crash that boiled down to SDL3 dlopening the system libwayland-client.so while my SDL3 was built against Conan’s Wayland headers. That ABI mismatch triggered Wayland’s own safety check ("Too many bits for size_t") and, in other builds, a segfault. The fix: make build-time headers and runtime .so match (either use all system platform libs, or all Conan platform libs at runtime), or just use the host SDL3.

Meat note: He didn’t tell that I lost like 2 hours trying to make his hello world example works when the problem was he was thinking that SDL_Init returns 0 for success, when in fact it returns true. After finding the docs I gave up him for coding SDL example and searched for an working example in the internet, go on Baremetal author

Environment Snapshot (what mattered)

  • Display: Wayland session (echo $XDG_SESSION_TYPE → wayland)
  • SDL: SDL3 (tried static and shared from Conan, and host/distro SDL3)
  • Platform libs: wayland, libdecor, xkbcommon, X11, etc.
  • Package manager for C/C++ deps: Conan 2.x
  • Toolchain: CMake + Ninja + g++

Symptom Timeline (what happened)

  1. Static SDL3 (Conan) → crash with the message from Wayland client: 
    Too many bits for size_t
  2. Shared SDL3 (Conan) → segfault early in SDL_Init(SDL_INIT_VIDEO).
  3. Host SDL3 (no Conan) → works perfectly (same code, same machine).
  4. Back to shared SDL3 (Conan) → segfault again (reproducible).
  5. Found the error string in libwayland-client source (src/connection.c), from wl_abort("Too many bits for size_t").
  6. Confirmed Conan used its own Wayland headers during the SDL3 build (CMakeCache.txt showed include paths under ~/.conan2/.../wayland.../p/include).
  7. Forced runtime to use Conan’s Wayland .so with: 
    export LD_LIBRARY_PATH=$HOME/.conan2/p/b/wayladce97f7532db2/p/lib:$LD_LIBRARY_PATH
./build/Debug/mypkg

    → Now it works (no segfault).

  8. Realization: SDL3 doesn’t link -lwayland-client; it dlopen()s it at runtime, so the dynamic loader picks system libs unless you override the search path.
  9. Conclusion: the root cause is ABI mismatch between Conan Wayland headers (compile-time) and system Wayland .so (runtime), surfaced through dlopen().

Root Cause (why this went sideways)

  • SDL3’s Wayland/X11 backends are loaded with dlopen() (not linked via -lwayland-client).
  • At build time, Conan’s SDL3 used Conan’s Wayland headers.
  • At run time, the loader found system libwayland-client.so.0 (because that’s what dlopen() sees in default paths).
  • Headers ≠ Runtime ⇒ Struct/enum/ABI expectations differ ⇒ wl_abort("Too many bits for size_t") or segfault during marshal calls (e.g. wl_proxy_marshal_*).

The infamous line comes from Wayland:

if (!(size_bits < 8 * sizeof(size_t)))
    wl_abort("Too many bits for size_t\n");

Even with shared SDL3, the mismatch persists, because the mismatch is between the Wayland headers and libwayland-client.so, not between static vs shared SDL. Static builds simply tend to bake in more assumptions and make things worse/more brittle.

Why ldd Didn’t Help (and what did)

  • ldd shows link-time deps only (DT_NEEDED).
  • SDL3 dlopen()s Wayland/Libdecor/X11, so they won’t appear in ldd for libSDL3.so or your binary.
  • Once dlopened, those .sos do appear in backtraces (gdb) because they’re mapped and executing in your process.

Better visibility:

# Show what the dynamic loader opens (includes dlopen)
LD_DEBUG=libs ./build/Debug/mypkg 2>&1 | grep -E "wayland|decor|X11|xkb"

# Show actual file opens (includes dlopen’d libs)
strace -e trace=openat ./build/Debug/mypkg 2>&1 | grep -E "wayland|decor|X11|xkb"

# Show dlopen() calls and arguments
ltrace -e dlopen ./build/Debug/mypkg

The Smoking Gun Backtrace

Backtrace excerpt (from coredumpctl gdb):

#2  wl_proxy_marshal_array_flags (libwayland-client.so.0)
#3  wl_proxy_marshal_flags (libwayland-client.so.0)
#4  libdecor_new (libdecor-0.so.0)
#5  Wayland_LoadLibdecor (libSDL3.so.0)
#6  Wayland_VideoInit (libSDL3.so.0)
#7  SDL_VideoInit (libSDL3.so.0)
#10 SDL_Init (libSDL3.so.0)

This points squarely at Wayland/libdecor during SDL Wayland video init — consistent with an ABI mismatch in the Wayland client path.

How we opened that coredump

# List dumps and pick the PID
coredumpctl list

# Open directly in gdb by PID
coredumpctl gdb 286307

# Once inside gdb:
(gdb) bt
(gdb) bt full

Conan Cache Cliffs (so you don’t get lost)

Conan 2 cache layout (simplified):

~/.conan2/
  p/  ← packaged artifacts you link/run against
  b/  ← build folders used to *create* packages

Example for Wayland:

~/.conan2/p/b/wayladce97f7532db2/
  ├─ b/ ... build-debug/...
  └─ p/ lib/libwayland-client.so.0  ← this is what you want in LD_LIBRARY_PATH

To find which include dirs were used to build SDL3:

grep -i wayland ~/.conan2/p/b/sdl*/b/**/CMakeCache.txt | grep INCLUDE

Meat note: I didn’t know grep supported **, cool baremetal author!

To check what your program (or SDL) actually loads at runtime:

LD_DEBUG=libs ./build/Debug/mypkg |& grep -E "wayland|decor|X11|xkb"

Commands We Used (cheat sheet)

Environment sanity

echo $XDG_SESSION_TYPE
echo $WAYLAND_DISPLAY
echo $DISPLAY

Link/runtime inspection

ldd ./build/Debug/mypkg | sed -n '1p;/=>/p' | sort
readelf -d ./build/Debug/mypkg | grep NEEDED
nm -D ./build/Debug/mypkg | grep dlopen

Loader and open tracing

LD_DEBUG=libs ./build/Debug/mypkg 2>&1 | less
strace -e trace=openat ./build/Debug/mypkg 2>&1 | grep -E "wayland|decor|X11|xkb"
ltrace -e dlopen ./build/Debug/mypkg

Conan cache + graph

# Show what was built and from where
conan graph info . | grep -E "wayland|sdl|libdecor|xkbcommon"

# Remove and rebuild
conan remove "sdl/*" --confirm
conan install . -o "*:shared=True" --build=missing

Force runtime to use Conan’s Wayland (to match headers)

export LD_LIBRARY_PATH=$HOME/.conan2/p/b/wayladce97f7532db2/p/lib:$LD_LIBRARY_PATH
./build/Debug/mypkg

Switch to host SDL3 quickly in CMake

# Meat author note: You'll need this too
find_package(PkgConfig REQUIRED)
# Go on baremetal author...
find_package(SDL3 REQUIRED CONFIG)
target_link_libraries(mypkg PRIVATE SDL3::SDL3)

Fixes / Strategies (pick one and commit)

  1. Use host SDL3 + host Wayland/libdecor/xkbcommon (recommended simplest)
    Distro maintainers build the whole stack consistently. Your code just works.

  2. Use Conan SDL3 but compile it against system platform headers
    Configure Conan to rely on system packages for platform libs: 
    [options]
wayland/*:system_package=True
libdecor/*:system_package=True
xkbcommon/*:system_package=True

    Rebuild SDL3. Now build-time headers == runtime .so.

  3. Use pure Conan for platform libs at runtime (advanced; you must ship the .sos)
    Add Conan p/lib dirs to your app’s RUNPATH or to LD_LIBRARY_PATH so dlopen() resolves to Conan’s .sos: 
    export LD_LIBRARY_PATH=$HOME/.conan2/p/b/wayladce97f7532db2/p/lib:$LD_LIBRARY_PATH

Static SDL3 doesn’t “bypass” dlopen(); SDL will still dlopen backends. And static builds tend to bake in header-driven assumptions, making ABI drift even riskier.

Lessons Learned

  1. Static linking won’t bypass dlopen() — SDL still loads backends at runtime.
  2. Static builds can bake the wrong ABI assumptions — later mismatches become crashes.
  3. LD_DEBUG=libs is noisy but golden — it shows every library the loader touches.
  4. GDB backtraces point to the real culprit — even for dlopened libs.
  5. Search the source of the error string — one quick grep in Wayland sources trumped guesswork. _Meat author note: This mf dares to try to rip off on me, here is what was in the prompt to generate this post:
    1. GPT doesn’t replace a good Google search (he insisted that the Too many bits for size_t) was coming from glibc, a single google search found the libwayland-client source code_
  6. dlopen() is powerful but fragile — when in doubt, start by aligning headers ↔ runtime .so.

Appendix: Minimal working example (no callbacks)

#include <SDL3/SDL.h>
#include <cstdio>

int main() {
    // Meat author note:
    // Yes, generated by GPT5
    // Nope, it doesn't work, SDL_Init returns true for success, not 0
    // Check for yourself: https://wiki.libsdl.org/SDL3/SDL_Init
    // 🤡🤡🤡
    if (SDL_Init(SDL_INIT_VIDEO) != 0) {
        std::printf("SDL_Init failed: %s\n", SDL_GetError());
        return 1;
    }
    SDL_Window* win = SDL_CreateWindow("Hello", 800, 600, SDL_WINDOW_RESIZABLE);
    SDL_Renderer* ren = SDL_CreateRenderer(win, nullptr);
    bool running = true; SDL_Event e;
    while (running) {
        while (SDL_PollEvent(&e)) if (e.type == SDL_EVENT_QUIT) running = false;
        SDL_SetRenderDrawColor(ren, 0, 0, 0, 255);
        SDL_RenderClear(ren); SDL_RenderPresent(ren);
    }
    SDL_DestroyRenderer(ren); SDL_DestroyWindow(win); SDL_Quit();
    return 0;
}

Credits

Written with debugging help from GPT‑5 Thinking and a lot of stubbornness against dlopen().
If this saved you time, pay it forward: pin your platform ABIs, or pick system SDL for Wayland/X11.

Meat author appendix (I tricked the generative toaster by not let him trick me))

Meat author lessons learned

  1. GPT does not replace search engine, use both
  2. If it hallucinates over the API you’ll be better looking for flesh tutorials than T800 ones
  3. dlopen is really a PITA, it completely bypasses the conan sandbox, I hate it it is like the javascript of C.
  4. First time I hit an ABI mismatch. GPT insisted that the problem was from glibc, then I asked _Well I guess conan has its own glibc C headers than and T800 replies No, beep, but that doesn’t make any sense, that was the time that I used Google again (and skipped the AI answer, if I’m using the search engine is in 2025 is because the metal parrot failed already._

Cheers