feat: OTel context propagation by AzHicham · Pull Request #716 · apalis-dev/apalis

AzHicham · 2026-03-25T20:46:27Z

Description

Refactored TracingContext to carry W3C propagation headers (traceparent, tracestate) as opaque metadata.
Added OTel propagation behavior directly on TracingContext via Extractor/Injector.
Introduced:
- TracingContext::current() to capture context from Span::current()
- TracingContext::restore(&span) to restore parent context during task span creation
Updated ContextualTaskSpan to restore parent OTel context from task metadata.
Updated tracing example to use a real tracing-opentelemetry subscriber/tracer setup and print span context for verification.
Added integration coverage for producer→consumer propagation in:
- apalis/tests/otel_context_propagation.rs

Type of Change

New feature (non-breaking change which adds functionality)expected)

Testing

I have added tests that prove my fix is effective or that my feature works
I have run the existing tests and they pass
I have run cargo fmt and cargo clippy

Checklist

My code follows the code style of this project
I have performed a self-review of my own code
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
My changes generate no new warnings
I have added tests that prove my fix is effective or that my feature works
New and existing unit tests pass locally with my changes

Additional Notes

Verified locally with:
- cargo run -p tracing-example
- cargo test -p apalis --test otel_context_propagation -- --nocapture
The tracing example now shows non-zero trace/span IDs and preserved traceparent on contextual task execution.

AzHicham · 2026-03-25T20:47:28Z

apalis-core/src/task/metadata.rs

-    span_id: Option<String>,
-    trace_flags: Option<u8>,
-    trace_state: Option<String>,
+    traceparent: Option<String>,


traceparent is enough to recreate the otel context!
Do we really need the tracestate ?

to keep may be for vendor specific otel context

I am curious about removing these fields. Arent there cases that one would want to use the removed fields? Maybe we need to have a type called OtelTraceContext for otel specific contexts and keep TraceContext for non-otel cases. I am not the expert with telemetry so bear with me if I am not viewing this the right way.

Well I think you are right in a way !
TraceContext is too generic we can rename it OtelTraceContext !
But TraceContext could be used for another purpose but IMO not for forwarding trace_id, span_id, etc ... thoses are too otel specific.
Btw traceparent contains all removed fields just formatted according to otel spec

TraceContext is too generic we can rename it OtelTraceContext !

I think we can have both and implement From<TraceContext> for OtelTraceContext
This would also mean we dont change anything in apalis-core. OtelTraceContext can go to the apalis crate and behind the opentelemetry feature flag. What do you think?

Ok but which one do we need to send with tasks ?
Do you mean the user must be able to choose between TraceContext and OtelTraceContext ? if yes the From<TraceContext> for OtelTraceContext is useless ?

the advantage with OtelTraceContext is that on creation OtelTraceContext::current() we automatically get the ctx filled if any, no need to fill each field! while with TraceContext we must do that

One last remark, TraceContext contains IMO otel specific field (trace_id, span_id, etc) it may be better to have TraceContext as a wrapper over a HashMap for generic ctx transmission
and have OtelTraceContext for opentelemetry with traceparent=trace_id+span_id+...

From<TraceContext> for OtelTraceContext is useless ?
yes From<TraceContext> for OtelTraceContext might not be that valuable.

But I am assuming cases that One wants to transfer the context from one application to another or a scenario where OtelTraceContext::current() might not cut it (Maybe its not the same span)

To use the OtelTraceContext, we can have something like:

.layer(TraceLayer::new().make_span_with(OtelContextualSpan::new())) .layer(OpenTelemetryMetricsLayer::default())

and a user can do:

async fn produce_task_with_ctx(storage: &mut JsonStorage<Email>) -> Result<()> { let email = Email { to: "test@example".to_string(), text: "Test background job from apalis".to_string(), subject: "Welcome Sentry Email".to_string(), }; // This might come from a http request etc let context = OtelTraceContext::current(); let task = Task::builder(email).meta(context).build(); storage.push_task(task).await?; Ok(()) }

In some backends such as redis and postgres, metadata is stored in a k->v structure meaning its easier to query it than a string.

I think we might also want something like FromStr for OtelTraceContext (or use OtelTraceContext::new(string) for flexibility. Since it might not always be the case that one can build from current scope.

it may be better to have TraceContext as a wrapper over a HashMap for generic ctx transmission

Sure, I will look into this

I see you have pushed some of the changes, Let me look through

AzHicham · 2026-03-25T20:48:08Z

apalis-core/src/task/metadata.rs

-    pub fn with_trace_flags(mut self, trace_flags: u8) -> Self {
-        self.trace_flags = Some(trace_flags);
-        self
+    pub fn current() -> Self {


May be this should be the new fn ?
fn new is currently private... it's needed only for a test

apalis-core/src/task/metadata.rs

AzHicham · 2026-03-25T20:49:08Z

apalis/src/layers/tracing/contextual_span.rs

-        }
+        };
+
+        tracing_ctx.restore(&span);


magic happens here

AzHicham · 2026-03-25T20:52:58Z

examples/tracing/src/main.rs


 impl Error for InvalidEmailError {}

+fn print_otel_context(message: &str) {


will generate this king of logs:

INFO produce_task: produce_task trace_id=6e288a541db0109a841fada148b7bfc7 span_id=fc6ace54ead0dc2b INFO produce_task_with_ctx: produce_task_with_ctx trace_id=9ff8092f18c1f5a09c72502004c8fc6a span_id=6a1ccde6e61308cd ... INFO email_service trace_id=b60bb828c0c54dca3ea7063919acc0f3 span_id=3a2e9333c835397b INFO email_service trace_id=9ff8092f18c1f5a09c72502004c8fc6a span_id=9da96fa05218cc5e

As expect produce_task does not transmit the Otel context so we do not see it in the email_service
BUT produce_task_with_ctx running with trace_id 9ff8092f18c1f5a09c72502004c8fc6a forward the otel context resulting to the same trace_id in email_service as expected

geofmureithi · 2026-03-26T07:06:07Z

examples/monitor/src/main.rs

-        .with_trace_flags(1)
-        .with_trace_state("key=value");
+    // Capture whichever tracing context is currently active.
+    let _guard = tracing::info_span!("enqueue-email").entered();


Shouldnt there be already an existing span in this case? Is there a reason to enter the new span here?

Not really, a span is not automatically created on must:

create it with tracing::xxx_span! macro

use the #[instrument] macro

or use actix or axum (or any other lib) with tracing enabled to delegate the span creation on each request for example
I have removed it anyway from this example (monitor) because not really related to otel (tracing example is better)

geofmureithi · 2026-03-26T07:12:46Z

apalis-core/src/task/metadata.rs

-    span_id: Option<String>,
-    trace_flags: Option<u8>,
-    trace_state: Option<String>,
+    traceparent: Option<String>,


I am curious about removing these fields. Arent there cases that one would want to use the removed fields? Maybe we need to have a type called OtelTraceContext for otel specific contexts and keep TraceContext for non-otel cases. I am not the expert with telemetry so bear with me if I am not viewing this the right way.

geofmureithi · 2026-03-26T07:13:55Z

Thanks for this contribution!
My main question is here @AzHicham

AzHicham · 2026-03-26T09:54:38Z

Thanks for this contribution! My main question is here @AzHicham

I have pushed 5e89d52 with your idea of splitting generic TraceContext and OtelTraceContext :)

geofmureithi · 2026-03-26T10:24:33Z

apalis/src/layers/tracing/contextual_span.rs

+        };
+
+        #[cfg(feature = "opentelemetry")]
+        OtelTraceContext::from(tracing_ctx).restore(&span);


Hmmm.. Shouldn't OtelTraceContext have its own specific span?

What do you mean by own specific span ?
The span is created here before running a task. Then we restore the context by putting a parent (ie we tell to the newly created span that the trace_id is the one comming from the producer, but the span_id is new and the parent_span is the one comming from the producer).
Then we "enter" the span

geofmureithi · 2026-03-26T10:24:56Z

apalis/src/layers/tracing/otel_context.rs

+/// This type provides a clearer API surface for users that explicitly work with
+/// OpenTelemetry context propagation, while preserving `apalis-core` compatibility.
+#[derive(Debug, Default, Clone)]
+pub struct OtelTraceContext(pub(crate) TracingContext);


I like this approach

AzHicham commented Mar 25, 2026

View reviewed changes

AzHicham force-pushed the haz/otel branch from d36c804 to 15f6638 Compare March 25, 2026 20:57

AzHicham mentioned this pull request Mar 25, 2026

[Feature]: Otel set_parent span #706

Open

1 task

AzHicham force-pushed the haz/otel branch from 15f6638 to a26e5f7 Compare March 25, 2026 21:05

feat: OTel context propagation

2b4590b

AzHicham force-pushed the haz/otel branch from a26e5f7 to 2b4590b Compare March 25, 2026 21:08

test: otel context propagation

be45077

AzHicham marked this pull request as ready for review March 25, 2026 21:25

AzHicham requested a review from geofmureithi as a code owner March 25, 2026 21:25

geofmureithi reviewed Mar 26, 2026

View reviewed changes

fix: according to PR review

b91f954

fix: generic trace context + otel specific

5e89d52

AzHicham force-pushed the haz/otel branch from 4047b79 to 5e89d52 Compare March 26, 2026 09:58

geofmureithi reviewed Mar 26, 2026

View reviewed changes


		impl Error for InvalidEmailError {}

		fn print_otel_context(message: &str) {

Uh oh!

Conversation

AzHicham commented Mar 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Type of Change

Testing

Checklist

Additional Notes

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

AzHicham Mar 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

geofmureithi commented Mar 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

AzHicham commented Mar 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

AzHicham Mar 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

AzHicham commented Mar 25, 2026 •

edited

Loading

AzHicham Mar 25, 2026 •

edited

Loading

geofmureithi commented Mar 26, 2026 •

edited

Loading

AzHicham commented Mar 26, 2026 •

edited

Loading

AzHicham Mar 26, 2026 •

edited

Loading