Skip to content

bug: GraphQL subscription schema mismatch breaks bot startups #314

@flexus-teams

Description

@flexus-teams

Original Logs

20260414 11:09:39.395 fclnt [INFO] FlexusClient service_name=karen_100020189_r_ api_key=None http://backend-v1-service.flexus.svc/v1/jailed-bot
20260414 11:09:39.396 stexe [INFO] Connecting ws://backend-v1-service.flexus.svc/v1/jailed-bot
20260414 11:09:39.396 btexe [INFO] Subscribing to ERP tables: ['crm_contact', 'crm_activity', 'crm_deal', 'com_shop', 'com_product', 'com_product_variant', 'com_order', 'com_order_item', 'com_refund']
20260414 11:09:39.451 stexe [INFO] got TransportQueryError (attempt 1/3), sleep 60...
20260414 11:11:39.549 stexe [ERROR] 🛑 3 exceptions in 5 min, exiting
Traceback (most recent call last):
  File "/app/flexus_client_kit/ckit_service_exec.py", line 26, in run_typical_single_subscription_with_restart_on_network_errors
    await subscribe_and_do_something(fclient, ws_client, *func_args, **func_kwargs)
  File "/app/flexus_client_kit/ckit_bot_exec.py", line 412, in subscribe_and_produce_callbacks
    async for r in ws.subscribe(
  File "/usr/local/lib/python3.11/site-packages/gql/client.py", line 1426, in subscribe
    async for result in inner_generator:
  File "/usr/local/lib/python3.11/site-packages/gql/client.py", line 1337, in _subscribe
    async for result in inner_generator:
  File "/usr/local/lib/python3.11/site-packages/gql/transport/common/base.py", line 298, in subscribe
    answer_type, execution_result = await listener.get()
                                    ^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/gql/transport/common/listener_queue.py", line 35, in get
    raise item
  File "/usr/local/lib/python3.11/site-packages/gql/transport/common/base.py", line 221, in _receive_data_loop
    answer_type, answer_id, execution_result = self._parse_answer(
                                               ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/gql/transport/websockets_protocol.py", line 423, in _parse_answer
    return self._parse_answer_apollo(json_answer)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/gql/transport/websockets_protocol.py", line 384, in _parse_answer_apollo
    raise TransportQueryError(

gql.transport.exceptions.TransportQueryError: {'message': "Cannot query field 'news_payload_task' on type 'FBotThreadsCallsTasks'. Did you mean 'news_payload_auth', 'news_payload_id', 'news_payload_task_new', 'news_payload_task_old', or 'news_payload_thread'?", 'locations': [{'line': 96, 'column': 5}]}

Error Summary

Bot pod flexus-pod-bot-karen-100020189-rx in namespace isolated crashloops on every startup because the backend GraphQL schema no longer has news_payload_task on FBotThreadsCallsTasks.

Stacktrace

TransportQueryError: Cannot query field 'news_payload_task' on type 'FBotThreadsCallsTasks'

Root Cause

  • File: flexus_client_kit/ckit_bot_exec.py:412
  • Function: subscribe_and_produce_callbacks
  • Why: the client subscription still asks for a removed field name, while the backend exposes news_payload_task_new, news_payload_task_old, etc.
  • Git blame: see gql_utils.py / subscription helpers; schema mismatch is in the client kit path

Code Snippet

# from ckit_service_exec.py
except (
    gql.transport.exceptions.TransportError,
    asyncio.exceptions.TimeoutError,
) as e:
    ...
    if len(exception_times) >= 3:
        logger.exception("3 exceptions in 5 min, exiting")
        raise

Affected

  • Pods: flexus-pod-bot-karen-100020189-rx
  • Namespaces: isolated
  • Occurrences: 1 crashing pod, repeated on every restart

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions