Skip to content

[Bug]: 500 when requesting /api/files/upload_archive concurrently #3732

@un-def

Description

@un-def

Steps to reproduce

# .dstack.yml
type: task
files:
  - blob
commands:
  - sleep 10
dd if=/dev/urandom of=blob bs=1M count=1
for i in {1..4}; do dstack apply --name task-$i --yes --detach & done

Actual behaviour

Only one run submitted:

dstack ps --last 4
 NAME             BACKEND       GPU  PRICE  STATUS      SUBMITTED
 task-4           ssh (remote)  -    $0     exited (0)  3 mins ago
 dev-environment  ssh (remote)  -    $0     stopped     1 hour ago
 dev-environment  ssh (remote)  -    $0     stopped     2 days ago
 dev-environment  ssh (remote)  -    $0     stopped     4 days ago

All other submissions failed with 500

Expected behaviour

No response

dstack version

0.20.15

Server logs

ERROR:    Exception in ASGI application
Traceback (most recent call last):
  File "/home/def/dev/dstack/.venv/lib/python3.13/site-packages/sqlalchemy/dialects/postgresql/asyncpg.py", line 545, in _prepare_and_execute
    self._rows = deque(await prepared_stmt.fetch(*parameters))
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/def/dev/dstack/.venv/lib/python3.13/site-packages/asyncpg/prepared_stmt.py", line 176, in fetch
    data = await self.__bind_execute(args, 0, timeout)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/def/dev/dstack/.venv/lib/python3.13/site-packages/asyncpg/prepared_stmt.py", line 267, in __bind_execute
    data, status, _ = await self.__do_execute(
                      ^^^^^^^^^^^^^^^^^^^^^^^^
        lambda protocol: protocol.bind_execute(
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
            self._state, args, '', limit, True, timeout))
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/def/dev/dstack/.venv/lib/python3.13/site-packages/asyncpg/prepared_stmt.py", line 256, in __do_execute
    return await executor(protocol)
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "asyncpg/protocol/protocol.pyx", line 206, in bind_execute
asyncpg.exceptions.UniqueViolationError: duplicate key value violates unique constraint "uq_file_archives_user_id_blob_hash"
DETAIL:  Key (user_id, blob_hash)=(efa6c378-e86e-4738-b76a-57d1c9999e4b, c0265b5a6111efd917048b1cc50a3c6d0e23ff54756df49217d839915f0175a8) already exists.

Additional information

No response

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions