Skip to content

Ensure error conditions always roll back failed transactions #280

@MoralCode

Description

@MoralCode

this was observed in augur.tasks.github.contributors.process_contributors

Stack Trace
Traceback (most recent call last):
  File "/augur/.venv/lib/python3.11/site-packages/celery/app/trace.py", line 453, in trace_task
    R = retval = fun(*args, **kwargs)
                 ^^^^^^^^^^^^^^^^^^^^
  File "/augur/.venv/lib/python3.11/site-packages/celery/app/trace.py", line 736, in __protected_call__
    return self.run(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/augur/augur/tasks/github/contributors.py", line 28, in process_contributors
    key_auth = GithubRandomKeyAuth(logger)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/augur/augur/tasks/github/util/github_random_key_auth.py", line 17, in __init__
    github_api_keys = GithubApiKeyHandler(logger).keys
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/augur/augur/tasks/github/util/github_api_key_handler.py", line 38, in __init__
    self.config_key = self.get_config_key()
                      ^^^^^^^^^^^^^^^^^^^^^
  File "/augur/augur/tasks/github/util/github_api_key_handler.py", line 60, in get_config_key
    return get_value("Keys", "github_api_key")
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/augur/augur/application/db/lib.py", line 42, in get_value
    config_setting = execute_session_query(query, 'one')
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/augur/augur/application/db/util.py", line 43, in execute_session_query
    return catch_operational_error(func)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/augur/augur/application/db/util.py", line 21, in catch_operational_error
    return func()
           ^^^^^^
  File "/augur/.venv/lib/python3.11/site-packages/sqlalchemy/orm/query.py", line 2798, in one
    return self._iter().one()  # type: ignore
           ^^^^^^^^^^^^
  File "/augur/.venv/lib/python3.11/site-packages/sqlalchemy/orm/query.py", line 2847, in _iter
    result: Union[ScalarResult[_T], Result[_T]] = self.session.execute(
                                                  ^^^^^^^^^^^^^^^^^^^^^
  File "/augur/.venv/lib/python3.11/site-packages/sqlalchemy/orm/session.py", line 2306, in execute
    return self._execute_internal(
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/augur/.venv/lib/python3.11/site-packages/sqlalchemy/orm/session.py", line 2188, in _execute_internal
    result: Result[Any] = compile_state_cls.orm_execute_statement(
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/augur/.venv/lib/python3.11/site-packages/sqlalchemy/orm/context.py", line 293, in orm_execute_statement
    result = conn.execute(
             ^^^^^^^^^^^^^
  File "/augur/.venv/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1416, in execute
    return meth(
           ^^^^^
  File "/augur/.venv/lib/python3.11/site-packages/sqlalchemy/sql/elements.py", line 516, in _execute_on_connection
    return connection._execute_clauseelement(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/augur/.venv/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1639, in _execute_clauseelement
    ret = self._execute_context(
          ^^^^^^^^^^^^^^^^^^^^^^
  File "/augur/.venv/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1812, in _execute_context
    conn = self._revalidate_connection()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/augur/.venv/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 669, in _revalidate_connection
    self._invalid_transaction()
  File "/augur/.venv/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 659, in _invalid_transaction
    raise exc.PendingRollbackError(
sqlalchemy.exc.PendingRollbackError: Can't reconnect until invalid transaction is rolled back.  Please rollback() fully before proceeding (Background on this error at: https://sqlalche.me/e/20/8s2b)

we need to be more consistent about explicitly rolling things back on failure, so the un-rollled-back transaction doesnt just stay open in the db session for an error to happen when that session is released to the connection pool for some other part of the app to use

Metadata

Metadata

Assignees

No one assigned

    Labels

    deployed versionLive problems with deployed versions

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions