`bst build` command fails intermittently

[mir-faisal-ahmed]

Running bst build, is inconsistent; Buildstream, seems to be attempting to resolve the link elements first, giving a circular dependency error; This error seems occurs intermittently, which might be pointing to a race condition; as running bst build <$target_element.bst> does not generate the same circular dependency issue.

[harrysarson]

Hi, I had some free time recently so did some digging into this.

Buildstream, seems to be attempting to resolve the link elements first, giving a circular dependency error;

I have managed to reproduce this issue.

It is a bug in the loader, essentially the same
issue as #1455 but the fix
(#1701) does not fix these issues
because we are doing bst build link.bst rather than bst build junction.bst:element.bst. Doing the load via the link element confuses the
logic for ensuring fully loaded because the provinance_node is not None
in Loader.get_loader.

I think I have a fix I can propose for this (along similar lines to #1701). Will try and put an PR up at some point.

This error seems occurs intermittently, which might be pointing to a race condition

This error is not caused by a race condition (there is no concurrency in the
loader). The intermittent failures come from the order in which elements are
loaded when running bst build or bst show with no arguments.

The implementation of argument-less bst [build|show] generates a list of
elements to load via a call to Project.get_default_targets()[^1] which does an
os.walk on the elements directory. os.walk does not iterate in a
deterministic order and so the order in which elements are loaded varies.

If the problematic element are loaded first then buildstream crashes with the
circular deps issue and if other elements are loaded first then buildstream runs
successfully. This explains the intermittent failure.