Display warning when grader compilation fails when testing submissions

[Binaryvector]

When running problemtools with the flag -p submissions, no warning or any other message is displayed, when the grader program fails to compile. Instead, problemtools silently switches to the default grader.

Currently an error is only displayed when also adding graders to the problem part flag, but I think it makes sense to add a warning even if that flag is not used, because:

1. When testing only submissions, the grader program has to be compiled anyway, so one might as well display a warning if the compilation fails.
2. Runtime errors in the grader cause a JE, i.e. they are considered errors when testing submissions. So maybe a compilation error should be considered an error too?
3. Since the verdict of a submission depends on the used grader, I think that information is important to be displayed when testing submissions. Currently problemtools just fails silently and uses the default grader instead, which might give the wrong verdict.

[ghamerly]

@Binaryvector can you provide a minimal problem package that demonstrates these issues (and how you test them)?

[Binaryvector]

https://www.dropbox.com/s/f4jmutiucduj4km/hello.zip?dl=0
This is based on your hello world example. I changed the problem specification to be a scoring problem and added a custom grader that has an error.

Without the error, the grader would give every submission a TLE. Problemtools does not use this grader because of the error, so it scores everything as AC instead.

verifyproblem hello -p submissions
Here problemtools does not display anything regarding the grader and just silently uses the default grader.

Loading problem hello
Checking submissions
   AC submission hello.cc (C++) OK: AC (0) [CPU: 0.00s @ test case secret/hello]
   AC submission hello.py (Python 2) OK: AC (0) [CPU: 0.01s @ test case secret/hello]
   AC submission hello_alarm.c (C) OK: AC (0) [CPU: 1.00s @ test case secret/hello]
   Slowest AC runtime: 1.003, setting timelim to 5 secs, safety margin to 10 secs
hello tested: 0 errors, 0 warnings

verifyproblem hello -p submissions graders
Now problemtools throws an error regarding the grader.

Loading problem hello
Checking submissions
   AC submission hello.cc (C++) OK: AC (0) [CPU: 0.00s @ test case secret/hello]
   AC submission hello.py (Python 2) OK: AC (0) [CPU: 0.01s @ test case secret/hello]
   AC submission hello_alarm.c (C) OK: AC (0) [CPU: 1.00s @ test case secret/hello]
   Slowest AC runtime: 0.997, setting timelim to 5 secs, safety margin to 10 secs
Checking graders
ERROR in graders: Compile error for grader.py (Python 2):
          File "/tmp/verify-hello-Y0KCs2/grader.py/grader.py", line 7
            print "TLE 0"
                ^
        SyntaxError: invalid syntax
hello tested: 1 error, 0 warnings

When I fix the grader by commenting the line 'someError(', then
verifyproblem hello -p submissions
returns the expected TLE verdicts

Loading problem hello
Checking submissions
ERROR in submissions: AC submission hello.cc (C++) got TLE [test case: test case secret/hello, CPU: 0.00s @ test case secret/hello]
ERROR in submissions: AC submission hello.py (Python 2) got TLE [test case: test case secret/hello, CPU: 0.01s @ test case secret/hello]
ERROR in submissions: AC submission hello_alarm.c (C) got TLE [test case: test case secret/hello, CPU: 1.01s @ test case secret/hello]
   Slowest AC runtime: 1.012, setting timelim to 5 secs, safety margin to 10 secs
hello tested: 3 errors, 0 warnings

[Matistjati]

This will be (unintentionally) fixed by #376. Output when running on master:

matistjati@DESKTOP-BJV27M6:~/temp$ vt hello/ -p submissions
Loading problem hello
Checking submissions
   AC submission hello.cc (C++) OK: AC (0) [testcase: testcase secret/hello, CPU: 0.00s @ testcase secret/hello]
ERROR Compile error for AC submission hello.py (Python 3 w/Pypy):
          File "/tmp/verify-hello-wj6999so/hello.py/hello.py", line 1
            print 'Hello World!'
                  ^
        SyntaxError: Missing parentheses in call to 'print'. Did you mean print('Hello World!')?
   AC submission hello_alarm.c (C) OK: AC (0) [testcase: testcase secret/hello, CPU: 1.00s @ testcase secret/hello]
   Slowest AC runtime: 1.002, setting timelim to 6 secs, safety margin to 12 secs
hello tested: 1 error, 0 warnings

Output when running on #376:

matistjati@DESKTOP-BJV27M6:~/temp$ vt hello/ -p submissions
Loading problem hello
Checking submissions
CRITICAL Failed to compile grader grader.py (Python 3 w/Pypy)

[gkreitz]

Closed by #376