Prodfiler

Prodfiler only showing process name and not function names

I’ve set up prodfiler on an instance with a Bitcoin Core node.

On prodfiler.com I only see the process/binary name and not the function names. I’d expect to see the function names too.

Hey,

if this is a binary for which you have symbols somewhere, you should be able to get the stack traces symbolized by uploading symbols to the backend:

If the problem persists, please ping this thread again :slight_smile:

Cheers,
Thomas

1 Like

Thanks! That was a classic case of user-not-reading-the-docs. A note or link on the prodfiler UI would be useful. The binary I’ve been running has in fact been stripped. I’m running into this case: Prodfiler: Feature Reference - Stripped C/C++ binaries which mentions to reach out if one runs into this problem.

As I’m running prodfiler on a test system, I’m able to switch out the binary. Will try a non-stripped binary next.

Hey there,

we’ll add a note to the UI :slight_smile: – that’s good feedback, thanks!

For dealing with stripped binaries for which no debug symbols can be found anywhere: We are working on a solution for that, but it’ll likely take a few months. The idea would be that if you have a completely stripped binary with no symbols running, and you can find a similar binary for which you have symbols, we would be able to offer to "port’ symbols from the unstripped version to the stripped version. This would not be 100% reliable, but at least allow some results…

1 Like

Hi Thomas :slightly_smiling_face:

I’ve had a play with Prodfiler this afternoon, looking to profile an Istio benchmarking cluster. The istio-proxy container runs an Envoy binary which has had its symbols stripped. I’ve downloaded the “envoy with symbols” build that matches the version of Envoy published with the version of Istio I have installed. I tried uploading it twice, once with the commit ID in the name, once as just envoy, in case that mattered.

A few hours later, nothing in the flamegraph.

Would you expect this to have worked? Anything I should have done?

-c

Hey there,

Some background information on symbolization: We are building a hash of the executable and reporting this hash to the backend. For uploaded executables we also build a hash. And only if the hash of both executables match we can assign symbols.
At the moment it looks like the hash from the executables of your cluster does not match with the hash of the uploaded executable with symbols.

Cheers,
Florian

To elaborate on what @flo said: even if 2 envoy binaries originate from the same version/commit, they are generally unlikely to have been compiled exactly the same:

  • The version of the compiler might be different
  • The compilation options might be different
  • They might have “custom patches” applied - this may not apply to your case, but it is often the case in binaries from Linux distribution (Debian, Fedora, etc will apply their own patches to upstream code)

Unfortunately, as @flo implied, symbolization requires the binary on your host to be byte-identical to the one for which you uploaded symbols. This is for 2 reasons:

  1. We identify files by their content hash (as Florian said) for symbolization purposes
  2. In general, it is not possible to use symbols that were generated for a different build. Assigning symbols to instruction addresses requires precision at the byte granularity. If anything is off, the result would be non-sensical. So in the end, 1. is not really a technical limitation of Prodfiler, but more an inherent limitation of symbolizers/profilers/debuggers.

We have some ideas on how to improve the situation in the long term, but this is uncharted territory (I am not aware of anything that solves this kind of problem today). For the time being, we do require symbols do have originated from the same build that produced the deployed binary.

If there’s any way we can assist you further please let us know - hopefully we can find something that works for you!

One thing you could try is: Strip the envoy-with-symbols executable and run that? That should give you symbolized stack traces. Obviously that’s less comfortable than running the upstream container …

Thanks for the comprehensive feedback. It doesn’t look like Istio publishes a version of the proxy container with an Envoy binary that includes symbols, so it would require generating and publishing a container with the fatter executable.

I was hoping that there would be some way of manually saying “these here symbols relate to this here binary”, but I understand why that isn’t the case.

It doesn’t look like Istio publishes a version of the proxy container with an Envoy binary that includes symbols, so it would require generating and publishing a container with the fatter executable.

Elaborating just in case: symbols don’t have to be part of container images (we never expect them to be present there). For example, we can symbolize profiles for slim/stripped container images that don’t contain any symbols. The most common examples are ubuntu/debian or alpine images available in Docker Hub, which are used as base images for many many things. The associated debug symbols are made available in package mirrors (which we consume), and there are mechanisms by which we can relate them back to the actual binaries that people run (that don’t have symbols).

This isn’t isolated to linux distros - postgresql or clickhouse, for example, do publish the symbols for their docker images, and we can ingest them. These symbols are never part of the container images.

We could possibly ask the Istio folks to publish debug symbols for the docker images they publish. If that was the case, we would be able to relate them back to your profiles. I took the liberty to ask there: Debug symbols for istio/proxyv2 - Discuss Istio

Side note: alternatively, we’re implementing another mechanism which could potentially help provide some symbols in the envoy case (but with less guarantees of accuracy). We’ll let come back to you then so you can try things out, a few weeks from now

I am Istio folks, so I can ask around.

Just so I’m clear (this is much lower level than I’ve had to operate in some years, so I’m sorry if this doesn’t make sense!): if you build a binary, strip the symbols, and then run the stripped binary, you can relate the symbols if you upload them. But, if you build a binary without symbols, and build a second binary with symbols, you can’t use the symbols from the second, on the first.

Your hash is only hashing the executable part of the binary, not the symbol part?

It looks like there are separate builds for with and without: proxy/release-binary.sh at master · istio/proxy · GitHub

I am Istio folks, so I can ask around.

Ah, excellent :slight_smile: Well I must say we would be very eager to support this, especially as envoy is now a widespread component.

if you build a binary, strip the symbols, and then run the stripped binary, you can relate the symbols if you upload them.

That’s correct :slight_smile: Ideally a debug link can be added to the stripped binary to identify the file that has debug symbols, like this: TotalView for HPC - or see the --only-keep-debug section in man objcopy

(this is what most Linux distributions do)

But, if you build a binary without symbols, and build a second binary with symbols, you can’t use the symbols from the second, on the first.

Exactly. It could in theory be possible if all builds are 100% reproducible. In that case it would require reproducing the exact build environment that produced the original binary - which is a challenge (exact same libraries, exact same compiler, etc). And even then there are no guarantees that it would produce adequate symbols, because sometimes builds just aren’t reproducible depending on the build logic (and compilers can be capricious beasts). But most importantly, there are no good ways to test that symbols are in fact adequate, so it’s better to avoid this avenue altogether.

I forgot to reply to this - sorry:

Your hash is only hashing the executable part of the binary, not the symbol part?

We hash the binary that is backing the envoy process running alongside the prodfiler agent. The hash is completely agnostic to the presence of symbols. The hash will be different for a stripped binary and a non-stripped binary

But ultimately it might be better to ignore the hashing step, as the rules can be expressed without that:

  1. Always pass the deployed file (the binary that is running and being profiled) to the -d option of the upload script.
  2. If the file passed to -d does not have debug symbols, you have to pass a separate debug symbol file to -g (usually, a file generated by objcopy --only-keep-debug).
  3. When passing -g (separate debug symbols), the symbols must correspond to the binary passed to -d (in practice: they must come from the same build artifacts). If you pass the wrong file, you’ll probably get non-sensical symbols (or no symbols at all).

On 3, there are some safeguards we could add to detect that the files don’t match - in fact, making a note of that now :slight_smile:

Let’s see if we can get some symbols from the istio builds. If we can, we’ll definitely be interested in ingesting them automatically so that it doesn’t require user involvement.

Also, feel free to DM me and we can setup a live call :slight_smile: I’d be happy to help - and again, very interested to improve the experience.