I paste here a feedback I received by email.
Feedback from running a competition on Codabench
I organized a small competition on Codabench on a classification task where participants submitted pre-trained model weights. The competition was structured in two phases: a first phase with evaluation on the validation dataset so that students could verify their submission was working, after which the goal was to automatically evaluate their last submission in the test phase.
The competition itself is here: https://codabench.org/competitions/14130/
Issues encountered
Auto-migration does not work correctly
The first problem I encountered is that auto-migration was not working: the same task was being re-executed instead of the task of the phase the submissions were migrated to. This same behavior also occurred when triggering migration manually via the button in the web interface.
Some submissions never finish. Certain submissions hang indefinitely, and the only workaround is to re-run them and hope for the best.
Workaround and additional observation
After the competition, I waited for the auto-migration bug to be fixed. I then tried to re-trigger the migration, without success. The solution I eventually found was to create a new phase and re-run the submissions from the initial phase (during which I was able to measure that at least 5% of submissions do not execute correctly on the first try), and then perform the migration.
In the process, I noticed that students who had chosen to "Submit as an organization" (for instance to display each group member's name) initially had the correct name on the leaderboard, but after re-execution the name was reset to the username of the person who had submitted.
I paste here a feedback I received by email.
Feedback from running a competition on Codabench
I organized a small competition on Codabench on a classification task where participants submitted pre-trained model weights. The competition was structured in two phases: a first phase with evaluation on the validation dataset so that students could verify their submission was working, after which the goal was to automatically evaluate their last submission in the test phase.
The competition itself is here: https://codabench.org/competitions/14130/
Issues encountered
Auto-migration does not work correctly
The first problem I encountered is that auto-migration was not working: the same task was being re-executed instead of the task of the phase the submissions were migrated to. This same behavior also occurred when triggering migration manually via the button in the web interface.
Some submissions never finish. Certain submissions hang indefinitely, and the only workaround is to re-run them and hope for the best.
Workaround and additional observation
After the competition, I waited for the auto-migration bug to be fixed. I then tried to re-trigger the migration, without success. The solution I eventually found was to create a new phase and re-run the submissions from the initial phase (during which I was able to measure that at least 5% of submissions do not execute correctly on the first try), and then perform the migration.
In the process, I noticed that students who had chosen to "Submit as an organization" (for instance to display each group member's name) initially had the correct name on the leaderboard, but after re-execution the name was reset to the username of the person who had submitted.