nat eval --skip_workflow can delete existing trajectories

[maxjeblick]

Version

1.4

Which installation method(s) does this occur on?

PyPi

Describe the bug.

When running nat eval --config_file config.yml --skip_workflow without
--dataset workflow_output.json, the already existing dataset workflow_output.json will be overwritten and all existing trajectories will be deleted.

• IMO one could improve this behavior by some more restrictive rules, preventing accidental overwrite of existing data, e.g. adding an --overwrite_existing flag. This feature might be useful in general, as nat seems to be very generous with deleting existing experiment data.
• workflow_output.json is already defined in config.yml, so I'd expect --skip_workflow to check if workflow_output.json already contains trajectories.

Minimum reproducible example

See commands above, should work with any nat agent that has some evaluator.

Relevant log output

No response

Other/Misc.

No response

Code of Conduct

• I agree to follow the NeMo Agent Toolkit Code of Conduct
• I have searched the open bugs and have found no duplicates for this bug report