Task priority being ignored

[kyleabbott]

I am seeing an issue with task priority in the following setup:

1. hq v0.25.0 (built from source)
2. multiple running hq workers
3. multiple jobs with tasks that have no explicit priority set (default to zero - validated via hq task info to have priority==0)
4. multiple jobs where all tasks have priority -5
5. a new job is submitted where all tasks have priority -1 (validated via hq task info)

(all jobs submitted via JDF)

The issue is that the job in (5) ends up being allocated almost all of the worker resources while thousands of runnable tasks in (3), that hq sees as having higher priority, are in the WAITING state. Somehow the job being submitted last with all of its tasks having priority -1 has jumped the queue of priority 0 tasks.

I have not used the priority feature much, but I have used it occasionally prior to upgrading to v0.25.0 and did not notice this issue before.

[spirali]

This seems like a bug. Can you share with us the resource requests of these tasks? Or ideally the part of job definition file?

[kyleabbott]

Here is hq task info for a couple of randomly selected tasks:

From new job:
+--------------+----------------------------------------------------------------------------------+
| Task ID      | 24039                                                                            |
| State        | RUNNING                                                                          |
| Resources    | cpus: 1 compact                                                                  |
| Priority     | -1                                                                               |

From earlier submitted job:
+--------------+-------------------------------------------------------------------------------+
| Task ID      | 42                                                                            |
| State        | WAITING                                                                       |
| Resources    | cpus: 1 compact                                                               |
| Priority     | 0                                                                             |

the JDFs are very simple:

[[task]]
id = 24039
command = ["bin/foo", "--bar", ...]
cwd = "/path/to/stuff"
priority = -1

[task.env]
ENV_VAR_A = "1"
ENV_VAR_B = "1"

And the priority == 0 job above looks the same, except the "priority" line is omitted from the JDF.