tracy-memory

Tracy Memory Profiling for Kit-Based Applications

Capture per-allocation CPU and GPU memory data in Tracy. Prerequisite: Tracy capture setup (see profiling skill), install-profilers for binaries.

Step 1: Environment Variables (Linux, required)

The omni.cpumemorytracking extension uses LD_PRELOAD to intercept malloc/free. Without it, Kit logs Failed to load library: 'liballocwrapper.so' and zero memory events are captured.

# Path varies by packman version
export LD_PRELOAD=~/.cache/packman/chk/allocmemwrapper/<version>/liballocwrapper.so
export TRACY_USE_LIB_UNWIND_FOR_BT=1   # libunwind-based backtrace
export TRACY_NO_SYS_TRACE=1            # reduce overhead
export TRACY_PORT="${TRACY_PORT:-8086}" # use 8087 for Isaac Sim 6.0+ when needed

Step 2: Kit Flags

# Base Tracy backend
--/app/profilerBackend=tracy
--/app/profileFromStart=true
--/profiler/enabled=true
--/plugins/carb.profiler-tracy.plugin/skipEventsOnShutdown=true
--/plugins/carb.profiler-tracy.plugin/memoryTraceStackCaptureDepth=16

# Reduce noise
--/profiler/channels/carb.tasking/enabled=false
--/profiler/channels/carb.events/enabled=false

# CPU memory tracking
--enable omni.cpumemorytracking
--/profiler/channels/cpu.memory/enabled=true
--/plugins/carb.cpumemorytracking.plugin/minCpuAllocSizeInBytesToTrack=1024

# GPU memory tracking
--/plugins/carb.memorytracking.plugin/enabled=true
--/profiler/channels/graphics.memory/enabled=true

Step 3: Unset LD_PRELOAD Before Capture Binary

Only Kit should use the interposer. The Tracy capture binary must NOT inherit it:

# After launching Kit with LD_PRELOAD in background
unset LD_PRELOAD
TRACY_CAPTURE_BIN=$(command -v capture || command -v capture-release || command -v tracy-capture)
[ -n "$TRACY_CAPTURE_BIN" ] || { echo "Missing capture/capture-release/tracy-capture binary"; exit 1; }
"$TRACY_CAPTURE_BIN" -o memtrace.tracy -f -p "${TRACY_PORT:-8086}"

Step 4: Verify — Strip Test (MANDATORY)

Do not rely on Kit log lines alone. Verify actual memory data in the output:

# Strip memory events and compare file sizes
TRACY_UPDATE_BIN=$(command -v update || command -v tracy-update)
[ -n "$TRACY_UPDATE_BIN" ] || { echo "Missing update/tracy-update binary; see install-profilers"; exit 1; }
"$TRACY_UPDATE_BIN" -s M memtrace.tracy memtrace_no_mem.tracy

# Good:   67 MB → 44 MB (~23 MB of memory data)
# Broken: 18 MB → 18 MB (zero memory data — LD_PRELOAD failed)

Kit Log Verification

Must see both lines:

[carb.cpumemorytracking] Loaded shared library: 'liballocwrapper.so'
[carb.cpumemorytracking] set memory alloc/free callbacks successfully!

If you see Failed to load library: 'liballocwrapper.so' → LD_PRELOAD path is wrong. Stop and fix before re-running.

Step 5: Analysis in Tracy

Open in Tracy GUI → Memory tab. A successful capture shows:

• Live counters: Active allocations, Memory usage
• Callstack tree grouping allocations by function
• Timeline showing allocation/free events

If Memory tab is empty: recheck LD_PRELOAD path and cpu.memory channel setting.

Debug Symbols

?? and [unknown] ??:??? in allocation callstacks = debug symbols missing. Stripped release binaries only have addresses.

To see function names and source locations:

• Build with RelWithDebInfo or keep .debug/.pdb files
• Or use debuginfod / Tracy's symbol path setting (Tools → Resolve symbol path)

Without symbols: allocation size/count/lifetime analysis still works, but code-level attribution doesn't.

Common Mistakes

• Missing LD_PRELOAD=liballocwrapper.so → extension starts but hooks never install
• Missing TRACY_USE_LIB_UNWIND_FOR_BT=1 → poor backtrace quality
• Missing GPU memory: need both carb.memorytracking.plugin/enabled=true AND graphics.memory channel
• LD_PRELOAD bleeding into capture binary → corrupt capture
• Trusting Kit log text without strip test → false positive