Skip to content

Weaver tests are flaky on PyPy #5176

@xrmx

Description

@xrmx

They are skipped for now but would be nice to have them work reliably if possible. I guess we also have slow installing times due to missing wheels since failing runs took more than 10 minutes.

-------------------------------- live log call ---------------------------------
WARNING  urllib3.connectionpool:connectionpool.py:868 Retrying (Retry(total=9, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError("HTTPConnection(host='localhost', port=53135): Failed to establish a new connection: [Errno 111] Connection refused")': /health
WARNING  urllib3.connectionpool:connectionpool.py:868 Retrying (Retry(total=8, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError("HTTPConnection(host='localhost', port=53135): Failed to establish a new connection: [Errno 111] Connection refused")': /health
PASSED                                                                   [ 63%]
tests/opentelemetry-test-utils/tests/test_weaver_live_check.py::TestSDKInitLiveCheck::test_end_and_check_raises_on_violations 
-------------------------------- live log call ---------------------------------
WARNING  urllib3.connectionpool:connectionpool.py:868 Retrying (Retry(total=9, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError("HTTPConnection(host='localhost', port=38821): Failed to establish a new connection: [Errno 111] Connection refused")': /health
WARNING  urllib3.connectionpool:connectionpool.py:868 Retrying (Retry(total=8, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError("HTTPConnection(host='localhost', port=38821): Failed to establish a new connection: [Errno 111] Connection refused")': /health
WARNING  urllib3.connectionpool:connectionpool.py:868 Retrying (Retry(total=7, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError("HTTPConnection(host='localhost', port=38821): Failed to establish a new connection: [Errno 111] Connection refused")': /health
WARNING  urllib3.connectionpool:connectionpool.py:868 Retrying (Retry(total=6, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError("HTTPConnection(host='localhost', port=38821): Failed to establish a new connection: [Errno 111] Connection refused")': /health
WARNING  urllib3.connectionpool:connectionpool.py:868 Retrying (Retry(total=5, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError("HTTPConnection(host='localhost', port=38821): Failed to establish a new connection: [Errno 111] Connection refused")': /health
WARNING  urllib3.connectionpool:connectionpool.py:868 Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError("HTTPConnection(host='localhost', port=38821): Failed to establish a new connection: [Errno 111] Connection refused")': /health
WARNING  urllib3.connectionpool:connectionpool.py:868 Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError("HTTPConnection(host='localhost', port=38821): Failed to establish a new connection: [Errno 111] Connection refused")': /health
WARNING  urllib3.connectionpool:connectionpool.py:868 Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError("HTTPConnection(host='localhost', port=38821): Failed to establish a new connection: [Errno 111] Connection refused")': /health
WARNING  urllib3.connectionpool:connectionpool.py:868 Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError("HTTPConnection(host='localhost', port=38821): Failed to establish a new connection: [Errno 111] Connection refused")': /health
WARNING  urllib3.connectionpool:connectionpool.py:868 Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError("HTTPConnection(host='localhost', port=38821): Failed to establish a new connection: [Errno 111] Connection refused")': /health
ERROR    opentelemetry.test.weaver_live_check:weaver_live_check.py:318 WeaverLiveCheck did not start: WeaverLiveCheck process exited unexpectedly (code -6), logs: 
E0424 15:12:55.913446    7686 ev_epoll1_linux.cc:371] (event_engine) Epoll1Poller:0x1d0db120 encountered epoll_wait error: Bad file descriptor

FAILED                                                                   [ 72%]
tests/opentelemetry-test-utils/tests/test_weaver_live_check.py::TestSDKInitLiveCheck::test_end_no_violations 
-------------------------------- live log call ---------------------------------
WARNING  urllib3.connectionpool:connectionpool.py:868 Retrying (Retry(total=9, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError("HTTPConnection(host='localhost', port=54733): Failed to establish a new connection: [Errno 111] Connection refused")': /health
WARNING  urllib3.connectionpool:connectionpool.py:868 Retrying (Retry(total=8, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError("HTTPConnection(host='localhost', port=54733): Failed to establish a new connection: [Errno 111] Connection refused")': /health
PASSED                                                                   [ 81%]
tests/opentelemetry-test-utils/tests/test_weaver_live_check.py::TestSDKInitLiveCheck::test_end_with_violations 
-------------------------------- live log call ---------------------------------
WARNING  urllib3.connectionpool:connectionpool.py:868 Retrying (Retry(total=9, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError("HTTPConnection(host='localhost', port=39173): Failed to establish a new connection: [Errno 111] Connection refused")': /health
WARNING  urllib3.connectionpool:connectionpool.py:868 Retrying (Retry(total=8, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError("HTTPConnection(host='localhost', port=39173): Failed to establish a new connection: [Errno 111] Connection refused")': /health
PASSED                                                                   [ 90%]
tests/opentelemetry-test-utils/tests/test_weaver_live_check.py::TestSDKInitLiveCheck::test_report_span_statistics 
-------------------------------- live log call ---------------------------------
WARNING  urllib3.connectionpool:connectionpool.py:868 Retrying (Retry(total=9, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError("HTTPConnection(host='localhost', port=34619): Failed to establish a new connection: [Errno 111] Connection refused")': /health
WARNING  urllib3.connectionpool:connectionpool.py:868 Retrying (Retry(total=8, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError("HTTPConnection(host='localhost', port=34619): Failed to establish a new connection: [Errno 111] Connection refused")': /health
PASSED                                                                   [100%]

=================================== FAILURES ===================================
_________ TestSDKInitLiveCheck.test_end_and_check_raises_on_violations _________

self = <HTTPConnection(host='localhost', port=38821) at 0x1c237280>

    def _new_conn(self) -> socket.socket:
        """Establish a socket connection and set nodelay settings on it.
    
        :return: New socket connection.
        """
        try:
>           sock = connection.create_connection(
                (self._dns_host, self.port),
                self.timeout,
                source_address=self.source_address,
                socket_options=self.socket_options,
            )

.tox/pypy3-test-opentelemetry-test-utils/lib/pypy3.10/site-packages/urllib3/connection.py:204: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
.tox/pypy3-test-opentelemetry-test-utils/lib/pypy3.10/site-packages/urllib3/util/connection.py:85: in create_connection
    raise err
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

address = ('localhost', 38821), timeout = 5, source_address = None
socket_options = [(6, 1, 1)]

.tox/pypy3-test-opentelemetry-test-utils/lib/pypy3.10/site-packages/requests/adapters.py:678: ConnectionError

The above exception was the direct cause of the following exception:

self = <tests.test_weaver_live_check.TestSDKInitLiveCheck testMethod=test_end_and_check_raises_on_violations>

    def test_end_and_check_raises_on_violations(self):
        """end_and_check() raises LiveCheckError with the report attached."""
>       with WeaverLiveCheck(
            registry=_REGISTRY_DIR, policies_dir=_TESTDATA_DIR
        ) as weaver:

tests/opentelemetry-test-utils/tests/test_weaver_live_check.py:80: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
tests/opentelemetry-test-utils/src/opentelemetry/test/weaver_live_check.py:299: in __enter__
    return self.start()
tests/opentelemetry-test-utils/src/opentelemetry/test/weaver_live_check.py:314: in start
    self._wait_for_ready()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <opentelemetry.test.weaver_live_check.WeaverLiveCheck object at 0x000000001cb5dc90>

    def _wait_for_ready(self) -> None:
        retry = Retry(
            total=10,
            backoff_factor=1,
            backoff_max=1,
            # Any non-2xx response from /health means weaver isn't ready yet.
            status_forcelist=list(range(300, 600)),
            raise_on_status=True,
            allowed_methods=["GET"],
        )
        session = Session()
        session.mount("http://", HTTPAdapter(max_retries=retry))
        try:
            session.get(
                f"http://localhost:{self._admin_port}/health", timeout=5
            )
        except Exception as exc:  # pylint: disable=broad-except
            if self._process is not None and self._process.poll() is not None:
>               raise RuntimeError(
                    f"WeaverLiveCheck process exited unexpectedly (code {self._process.returncode})"
                ) from exc
E               RuntimeError: WeaverLiveCheck process exited unexpectedly (code -6)

tests/opentelemetry-test-utils/src/opentelemetry/test/weaver_live_check.py:342: RuntimeError
------------------------------ Captured log call -------------------------------
WARNING  urllib3.connectionpool:connectionpool.py:868 Retrying (Retry(total=9, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError("HTTPConnection(host='localhost', port=38821): Failed to establish a new connection: [Errno 111] Connection refused")': /health
WARNING  urllib3.connectionpool:connectionpool.py:868 Retrying (Retry(total=8, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError("HTTPConnection(host='localhost', port=38821): Failed to establish a new connection: [Errno 111] Connection refused")': /health
WARNING  urllib3.connectionpool:connectionpool.py:868 Retrying (Retry(total=7, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError("HTTPConnection(host='localhost', port=38821): Failed to establish a new connection: [Errno 111] Connection refused")': /health
WARNING  urllib3.connectionpool:connectionpool.py:868 Retrying (Retry(total=6, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError("HTTPConnection(host='localhost', port=38821): Failed to establish a new connection: [Errno 111] Connection refused")': /health
WARNING  urllib3.connectionpool:connectionpool.py:868 Retrying (Retry(total=5, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError("HTTPConnection(host='localhost', port=38821): Failed to establish a new connection: [Errno 111] Connection refused")': /health
WARNING  urllib3.connectionpool:connectionpool.py:868 Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError("HTTPConnection(host='localhost', port=38821): Failed to establish a new connection: [Errno 111] Connection refused")': /health
WARNING  urllib3.connectionpool:connectionpool.py:868 Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError("HTTPConnection(host='localhost', port=38821): Failed to establish a new connection: [Errno 111] Connection refused")': /health
WARNING  urllib3.connectionpool:connectionpool.py:868 Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError("HTTPConnection(host='localhost', port=38821): Failed to establish a new connection: [Errno 111] Connection refused")': /health
WARNING  urllib3.connectionpool:connectionpool.py:868 Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError("HTTPConnection(host='localhost', port=38821): Failed to establish a new connection: [Errno 111] Connection refused")': /health
WARNING  urllib3.connectionpool:connectionpool.py:868 Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError("HTTPConnection(host='localhost', port=38821): Failed to establish a new connection: [Errno 111] Connection refused")': /health
ERROR    opentelemetry.test.weaver_live_check:weaver_live_check.py:318 WeaverLiveCheck did not start: WeaverLiveCheck process exited unexpectedly (code -6), logs: 
E0424 15:12:55.913446    7686 ev_epoll1_linux.cc:371] (event_engine) Epoll1Poller:0x1d0db120 encountered epoll_wait error: Bad file descriptor
=========================== short test summary info ============================
FAILED tests/opentelemetry-test-utils/tests/test_weaver_live_check.py::TestSDKInitLiveCheck::test_end_and_check_raises_on_violations
======================== 1 failed, 10 passed in 16.23s =========================
pypy3-test-opentelemetry-test-utils: exit 1 (17.22 seconds) /home/runner/work/opentelemetry-python/opentelemetry-python> pytest /home/runner/work/opentelemetry-python/opentelemetry-python/tests/opentelemetry-test-utils/tests -ra pid=7639
  pypy3-test-opentelemetry-test-utils: FAIL code 1 (779.65=setup[762.43]+cmd[17.22] seconds)
  evaluation failed :( (781.37 seconds)

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions