Skip to main content

Profiling with Datadog tracers

CubeAPM accepts continuous profiles from Datadog tracers. If your applications already send traces to CubeAPM with a Datadog tracer, you only need to turn profiling on in the tracer, and tell the agent to send profiles to CubeAPM.

Coverage​

LanguageProfile types
.NETCPU, wall time, exceptions, lock contention, timeline
GoCPU and heap by default. Block, mutex and goroutine when you turn them on
JavaCPU, allocations, wall time, GC time, lock time (only when there was lock contention)
Node.jsWall time with CPU time, heap space, timeline
PHPCPU time, wall time, allocations, exceptions, timeline
PythonCPU time, wall time, exceptions, lock wait and hold time, allocations, heap
RubyCPU time, wall time, timeline
C, C++, Rust and other native codeCPU time, allocations

For every language that has a Datadog tracer, CubeAPM also shows:

  • the endpoint breakdown: how much of a profile was spent in each HTTP route;
  • the link from a trace to its profile: open a trace, go to its Profiles tab, and see the profiles whose samples were taken while that request ran.

The profile types you see depend on what the tracer collects. Most tracers collect a subset by default and let you turn on more. Check the Datadog profiler documentation for your language.

Integration​

  1. Install Datadog tracer

    For .NET, Go, Java, Node.js, PHP, Python, and Ruby, the profiler ships along with the tracer. So, for these languages, the first step is to install the Datadog tracer.

    For C, C++, Rust and other native code, profiling works via a separate integration as described in the next step.

  2. Turn on profiling in the tracer

    Profiling is off by default in Datadog tracers. Turn it on for each application, then restart it.

    The profiler ships in the same datadog-dotnet-apm package as the tracer. On Linux x64 with the package installed in /opt/datadog, set these on the application, next to the tracer variables:

    DD_PROFILING_ENABLED=true
    CORECLR_ENABLE_PROFILING=1
    CORECLR_PROFILER={846F5F1C-F9AE-4B07-969E-05C26BC060D8}
    CORECLR_PROFILER_PATH=/opt/datadog/Datadog.Trace.ClrProfiler.Native.so
    DD_DOTNET_TRACER_HOME=/opt/datadog
    # needed by the profiler on Linux
    LD_PRELOAD=/opt/datadog/linux-x64/Datadog.Linux.ApiWrapper.x64.so

    After installing the package, run /opt/datadog/createLogPath.sh once so the tracer can write its logs.

  3. Configure Datadog Agent to forward profiles to CubeAPM

    DD_API_KEY=<your_datadog_key>
    # send runtime metrics to CubeAPM
    DD_DD_URL=http://<ip_address_of_cubeapm_server>:3130
    # send traces to CubeAPM
    DD_APM_DD_URL=http://<ip_address_of_cubeapm_server>:3130
    # send profiles to CubeAPM
    DD_APM_PROFILING_DD_URL=http://<ip_address_of_cubeapm_server>:3130/api/v2/profile
    # Accept telemetry data from applications running outside the agent host
    DD_APM_NON_LOCAL_TRAFFIC=true
    DD_DOGSTATSD_NON_LOCAL_TRAFFIC=true

    If you are not using Datadog agent, and instead sending telemetry data from Datadog tracer directly to CubeAPM, the same DD_TRACE_AGENT_URL setting used for sending traces to CubeAPM works for profiles as well, so no extra config is needed.

Troubleshooting​

What you seeCauseFix
Python profiles have no hostThe Python tracer does not send a host name with profilesFilter Python profiles by service
Go heap profile: allocations are empty in some recordsGo updates heap data only after a garbage collection. A minute with no GC has no new allocationsExpected. Look at a longer time range
A trace's Profiles tab says no profile records overlap this spanThe tab starts narrowed to the selected span, and the work ran in a child spanClick widen to the whole trace
A trace's Profiles tab is empty for a very fast requestThe profiler took no sample while that request ran. Requests of a few milliseconds often have noneExpected. Open a slower trace of the same endpoint