OTEL_SERVICE_NAME set in telemetry setup is not respected.

[msarahan]

OTEL_SERVICE_NAME is not included in the recorded base environment variables (https://github.com/rapidsai/shared-actions/blob/c20a4ad8af627e8c75dc2693d75596323838550a/telemetry-impls/stash-base-env-vars/action.yml)

I went back and forth on this, and I guess I left a mess. Some repos have this, like cudf, but it isn't doing anything. It only happens to be set to the same value by the summarize action:

shared-actions/telemetry-impls/summarize/action.yml

Line 38 in c20a4ad

OTEL_SERVICE_NAME=${OTEL_SERVICE_NAME:-${GITHUB_WORKFLOW}-${GITHUB_REPOSITORY#*/}}

We should either:

• fix it so that OTEL_SERVICE_NAME in telemetry setup applies
• change docs to specify that OTEL_SERVICE_NAME can be set in the summarize step instead
• not support custom names
• Some other way to support custom naming of top-level jobs

Chat with Jason Pascucci, who raised this issue (Jason in quote blocks, me not):

Although I can't tell what effect it has due to the grafana being offline, I notice another thing I wasn't expecting: telemetry-dispatch-setup sets the OTEL_SERVICE_NAME to the name of the current job. I expected that it would keep the OTEL_SERVICE_NAME that was configured in the telemetry-dispatch-stash-base-env-vars call.

OTEL_SERVICE_NAME is for the outer-most job. We use it to distinguish projects from each other. It really only matters in the Grafana/Tempo views. The same info is available in the attributes. which job in particular are you wanting to change?

I set OTEL_SERViCE_NAME to ci-cupynumeric in the outtermost setup telemetry call, but telemetry-dispatch-setup appears to overwrite it here, which I think might be wrong.

are you setting it like this? https://github.com/rapidsai/cudf/blob/main/.github/workflows/pr.yaml#L50-L58

pr.yaml

  telemetry-setup:
    continue-on-error: true
    runs-on: ubuntu-latest
    env:
      OTEL_SERVICE_NAME: 'pr-cudf'

the important difference is this different action, which happens only for the top-level job: https://github.com/rapidsai/cudf/blob/main/.github/workflows/pr.yaml#L58

pr.yaml

        uses: rapidsai/shared-actions/telemetry-dispatch-stash-base-env-vars@main

though I'm not seeing OTEL_SERVICE_NAME in those files. Let me see if I can track down where this comes in.

this is where it ultimately has to be set: https://github.com/rapidsai/shared-actions/blob/main/telemetry-impls/summarize/action.yml#L35-L46
OTEL_SERVICE_NAME is not explicitly set when calling that action: https://github.com/rapidsai/cudf/blob/main/.github/workflows/pr.yaml#L430-L440
ah, found it. It's not respecting the value in the setup env var. It is only coincidentally coming out to be the same value:

shared-actions/telemetry-impls/summarize/action.yml

Lines 38 to 46 in c20a4ad

	OTEL_SERVICE_NAME=${OTEL_SERVICE_NAME:-${GITHUB_WORKFLOW}-${GITHUB_REPOSITORY#*/}}
	export OTEL_SERVICE_NAME
	echo "OTEL_SERVICE_NAME=${OTEL_SERVICE_NAME}" >> $GITHUB_ENV
	TRACEPARENT=$(./shared-actions/telemetry-impls/traceparent.sh "${OTEL_SERVICE_NAME}")
	export TRACEPARENT
	echo "TRACEPARENT=${TRACEPARENT}" >> $GITHUB_ENV
	ls -lR telemetry-artifacts
	timeout 5m python3 ./shared-actions/telemetry-impls/summarize/send_trace.py

You can probably set the env: entry on the summarize job, like here: https://github.com/rapidsai/cudf/blob/main/.github/workflows/pr.yaml#L430-L440