Issues with OTLP exporter with None values

[aitor0307]

Describe your environment

OS: Ubuntu
Python version: 3.12
SDK version: opentelemetry-sdk==1.36.0
API version: opentelemetry-api==1.36.0
Celery: opentelemetry-instrumentation-celery==0.57b0

What happened?

I am using CeleryInstrumentor() to get logs of celery into Grafana/Loki

First of the issues I get is that I see an incompatibility with the latest opentelemetry-sdk (1.41) version with opentelemetry-api (it does not allow me to install).

Second issue is with None values in the celery results exported to Grafana. I am seeing a warning like:
Invalid type <class 'dict'> of value {'id': 'b2f12a15-b57d-4e9a-a439-345a323e7c1b', 'name': 'app.tasks.tasks.say_hello', 'args': '()', 'kwargs': '{}', 'eta': None}

I noticed that in opentelemetry 1.41 you already updated the encoder: https://github.com/open-telemetry/opentelemetry-python/blob/main/exporter/opentelemetry-exporter-otlp-proto-common/src/opentelemetry/exporter/otlp/proto/common/_internal/__init__.py

But I am uncapable of setting allow_null to True for CeleryInstrumentor().

Steps to Reproduce

Setup CeleryInstrumentor() and have a task that retuns None values somewhere.
In my case:

# API route to trigger the task
@bp.route('/long_task', methods=['GET'])
def trigger_task():
    task = say_hello.delay()
    return jsonify({"task_id": task.id, "status": "Task has been submitted"}), 202

@celery.task
def say_hello():
    sleep(10)
    logger.info(f"[HEALTHCHECK] say_hello config_class={type(Config).__name__} service_name={Config.SERVICE_NAME}")
    return "Hello, World!"

I also have a custom logger setup, which might be the one causing None to get inserted over the values.

Expected Result

No errors or warning escalated to Loki/Grafana

Actual Result

I monkey patched _encode_key_value:

def _encode_key_value(key: str, value: _Any, allow_null: bool = False) -> _PB2KeyValue:
if value is None:
return _PB2KeyValue(key=key, value=_encode_value(value, allow_null=True))
return _PB2KeyValue(key=key, value=_encode_value(value, allow_null=allow_null))

Additional context

I am unsure if this is a proper solution, I guess not and that it should be configured and CeleryInstrumentor some setting that allows to pass the allow_none=True, but I do not know how to do it.

Would you like to implement a fix?

None

Tip

_{React with 👍 to help prioritize this issue. Please use comments to provide useful context, avoiding +1 or me too, to help us triage it. Learn more here.}

[xrmx]

First of the issues I get is that I see an incompatibility with the latest opentelemetry-sdk (1.41) version with opentelemetry-api (it does not allow me to install).

You should update both at the same time. I think we got already some improvements for celery task serialization merged already in main so maybe it's already fixed. Anyway the failure is in serializing the spans or the logs?

[aitor0307]

thanks @xrmx

Yes, that's what I tried but when running pip-compile requirements.in I get an incompatibility error.
currently I have all these packages:

azure-core-tracing-opentelemetry==1.0.0b11
azure-monitor-opentelemetry==1.6.13
azure-monitor-opentelemetry-exporter==1.0.0b40
opentelemetry-api==1.36.0
opentelemetry-exporter-otlp==1.15.0
opentelemetry-exporter-otlp-proto-common==1.40.0
opentelemetry-exporter-otlp-proto-grpc==1.15.0
opentelemetry-exporter-otlp-proto-http==1.15.0
opentelemetry-instrumentation==0.57b0
opentelemetry-instrumentation-asgi==0.57b0
opentelemetry-instrumentation-celery==0.57b0
opentelemetry-instrumentation-dbapi==0.57b0
opentelemetry-instrumentation-django==0.57b0
opentelemetry-instrumentation-fastapi==0.57b0
opentelemetry-instrumentation-flask==0.57b0
opentelemetry-instrumentation-psycopg2==0.57b0
opentelemetry-instrumentation-redis==0.57b0
opentelemetry-instrumentation-requests==0.57b0
opentelemetry-instrumentation-urllib==0.57b0
opentelemetry-instrumentation-urllib3==0.57b0
opentelemetry-instrumentation-wsgi==0.57b0
opentelemetry-proto==1.15.0
opentelemetry-resource-detector-azure==0.1.5
opentelemetry-sdk==1.36.0
opentelemetry-semantic-co

If I try to force opentelemetry-sdk>1.40 says:

WARNING:pip._internal.metadata.base:Ignoring invalid distribution ~rllib3 (/home/aitor/lilihoenv/lib/python3.11/site-packages)
  ERROR: Cannot install -r requirements.in (line 19) and opentelemetry-sdk>=1.40 because these package versions have conflicting dependencies.
Traceback (most recent call last):
  File "/home/aitor/lilihoenv/lib/python3.11/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 96, in resolve
    result = self._result = resolver.resolve(
                            ^^^^^^^^^^^^^^^^^
  File "/home/aitor/lilihoenv/lib/python3.11/site-packages/pip/_vendor/resolvelib/resolvers/resolution.py", line 515, in resolve
    state = resolution.resolve(requirements, max_rounds=max_rounds)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/aitor/lilihoenv/lib/python3.11/site-packages/pip/_vendor/resolvelib/resolvers/resolution.py", line 456, in resolve
    raise ResolutionImpossible(self.state.backtrack_causes)
pip._vendor.resolvelib.resolvers.exceptions.ResolutionImpossible: [RequirementInformation(requirement=SpecifierRequirement('opentelemetry-sdk>=1.40'), parent=None), RequirementInformation(requirement=SpecifierRequirement('opentelemetry-sdk~=1.36.0'), parent=LinkCandidate('https://files.pythonhosted.org/packages/70/0d/16522e6cf83337e30b197d207ca8b436aa03372f0438251f3f43c3b5b1eb/azure_monitor_opentelemetry-1.6.13-py3-none-any.whl (from https://pypi.org/simple/azure-monitor-opentelemetry/) (requires-python:>=3.8)'))]

I tried removing azure_monitor_opentelemetry but then I got other incompatibilities.

---

Regarding the serialization of logs or spans, doesn't look like, I didn't see this error when using Azure logs (now I changed to Loki/Grafana) and after the monkey patch everything runs smooth.