Re: [lttng-dev] Tracing a docker containerized java application from the host - Jérémie Galarneau via lttng-dev

From: "Jérémie Galarneau via lttng-dev" <lttng-dev@lists.lttng.org>
To: Adel Belkhiri <adel.belkhiri@gmail.com>
Cc: "lttng-dev@lists.lttng.org" <lttng-dev@lists.lttng.org>
Subject: Re: [lttng-dev] Tracing a docker containerized java application from the host
Date: Fri, 17 Mar 2023 16:17:18 +0000	[thread overview]
Message-ID: <YT2PR01MB5984E4B3F7760594C6574611BCBD9@YT2PR01MB5984.CANPRD01.PROD.OUTLOOK.COM> (raw)
In-Reply-To: <CAMxxxdOiWf734KK3kOKOBJ56uj97Cc3rgfM2ppeMqJVuoW6-CQ@mail.gmail.com>

[-- Attachment #1.1: Type: text/plain, Size: 4644 bytes --]

I'm re-adding the list so every one can benefit from the exchange.

The agent and session daemon both attempt to communicate on "0.0.0.0" (the local host). I would assume you can expose/publish (in docker terms) the session daemon's port on the local interface too. I have not tried that though.

As for the contents of the /var/run/lttng folder, you ultimately have to make those files accessible to the applications. You can use the LTTNG_HOME environment variable to control where the session daemon creates those files and where the applications look for them. Then, adjust the container's mount points accordingly.

To record the events, the shared memory must be accessible to both ends (host and container). You will need to mount /dev/shm inside the container.

I think Michael's configuration for Kubernetes can help you understand how the various parts fit together:
https://github.com/mjeanson/gcloud-lttng/blob/master/pods/demo1.yaml.in

Jérémie

--
Jérémie Galarneau
EfficiOS Inc.
https://www.efficios.com
________________________________
From: Adel Belkhiri <adel.belkhiri@gmail.com>
Sent: March 16, 2023 19:23
To: Jérémie Galarneau <jgalar@efficios.com>
Subject: Re: [lttng-dev] Tracing a docker containerized java application from the host

Hello again,

I solved the issue. The problem was that the host address from inside the container was 172.0.0.1 but the agent was looking for 127.0.0.1. So, one last question, is it possible to instruct the agent to look for/communicate with the sessiond using a specific address (different from 127.0.0.1)? Also, I'm currently sharing the folder "/var/run/lttng" with the container, which in my opinion isn't a very elegant way. Then, is it possible to do otherwise? If I only copy the file "agent.port" inside the container, I can list the registered events, but I cannot record them. (I guess it is related to the location of the app socket file).

Adel

On Thu, Mar 16, 2023 at 3:57 PM Adel Belkhiri <adel.belkhiri@gmail.com<mailto:adel.belkhiri@gmail.com>> wrote:
Hi Jérémie,

Thank you for your reply. Yes, the container can communicate with the LTTtng session daemon that is running on the host machine. For instance, if I execute "lttng list -u" on the host I can see that there is an application registered and there are userspace events (e.g., lttng_jul:event,  statedumps, etc.). However, when I execute "lttng --jul -a", no java events are displayed.

Adel

On Thu, Mar 16, 2023 at 12:14 PM Jérémie Galarneau <jgalar@efficios.com<mailto:jgalar@efficios.com>> wrote:
Hi Adel,

The java tracing facilities make use of an agent that communicates with the session deamon through a TCP socket. You must to ensure your java application can connect to the session daemon's 'agent-tcp-port'.

https://lttng.org/man/8/lttng-sessiond/v2.13/#doc-opt--agent-tcp-port

Is that port properly exposed by your container configuration?

Jérémie

--
Jérémie Galarneau
EfficiOS Inc.
https://www.efficios.com
________________________________
From: lttng-dev <lttng-dev-bounces@lists.lttng.org<mailto:lttng-dev-bounces@lists.lttng.org>> on behalf of Adel Belkhiri via lttng-dev <lttng-dev@lists.lttng.org<mailto:lttng-dev@lists.lttng.org>>
Sent: March 15, 2023 20:03
To: lttng-dev@lists.lttng.org<mailto:lttng-dev@lists.lttng.org> <lttng-dev@lists.lttng.org<mailto:lttng-dev@lists.lttng.org>>
Subject: [lttng-dev] Tracing a docker containerized java application from the host

Hello everyone,

I am reaching out to seek assistance in tracing an instrumented Java application running within a Docker container from the host side.

In my setup, the LTTng session daemon, running on the host side, indicates that application registration has been completed (please refer to the logs). However, when running the command "lttng list --jul," no registered applications are displayed. Although the application successfully creates the "LttngLogHandler" object and attaches it to the logger, no traces are generated.

To allow the application to access the lttng socket file, the host's /var/run/lttng directory is shared with the container at launch (using the command "sudo docker run -p 8080:8080 -v /var/run/lttng:/var/run/lttng sample-image"). I am only interested in collecting userspace Java (--jul) traces.

I would greatly appreciate any assistance or guidance that you can provide. Thank you for your time.

Additional details include:

Ubuntu 22.04
Kernel: 5.15.0-60-generic
LTTng session daemon version: 2.13.2
Docker version 20.10.21, build 20.10.21-0ubuntu1~22.04.2
Regards,
Adel Belkhiri

[-- Attachment #1.2: Type: text/html, Size: 12000 bytes --]

[-- Attachment #2: Type: text/plain, Size: 156 bytes --]

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev