No Open Blockers

Programming is very much an iterative process, especially when you’re not entirely clear what you’re trying to do when you first start. It’s a good practice to look back and question certain aspects of a design. I’m at that point with CodeTurtle, and I’m not entirely surprised to be here. I knew that once I started to use it I was going to get a feeling for the two main aspects: the UI and the API for grading projects. I’m (somewhat surprisingly) pretty happy with the UI as it stands today, but the project grading needs some love. With spring break this week, I’m in a prime position to make some solid infrastructure changes in time to get the third project graded before we return from break.

Originally, I started out with a pretty simplistic idea. I planned on running various tests against the submission and indicate if points should be awarded for each verification (a grade item). It was loosely based on the idea of being a unit test framework with points. There was a very vague concept of relating tests to each other by way of a string in each grade item called the test suite.

This has a few primary issues:

• There is no real sense of aggregation of the grade items. I had a way to assign a “test suite” name to them, but that wasn’t sufficient.
• There is no way of saying “this is the total amount of points for the project”. I didn’t want to put grader developers in a position of declaring that there was 150 possible points and then making sure the grade items summed to that total. Instead, I calculate the total as the grade items are added to the project’s grade. I stand by this approach, but it has a bitch of a flaw: if there is an unexpected error from the student’s submission (as students, it’s understandable that this is a pretty common occurrence) and the grade items are not added, that project has a lower total grade than the rest.
• The grader developer has to implement one method (grade()) as the hook into CodeTurtle. That means any protection against the above scenario must be provided in the grader itself. This led to a lot of try/catch blocks as the grader’s grade() method called out to smaller grade methods.

After some time at my whiteboard, I came up with the following changes.

• The model now goes: a project grader has one or more grade test methods, each of which has one or more grade assertions. The idea of an aggregation of grade assertions is now a first-class concept.
• Each grade test method is run by CodeTurtle independently of each other, preventing an unexpected error in one from stopping the execution of the others.
• The domain model has been changed to reflect this, allowing an unexpected error in a grade test to be captured and indicated at the test level (as compared to simply a failed assertion).
• Grade tests are indicated through the @GradeTest annotation. It takes a name parameter that is the display name used on the report.
• Additionally, @Setup and @Teardown annotations indicate methods that should be run at the start and end of running all of the tests (yes, this looks very much like JUnit, it’s intentional).
• All project graders must extend the ProjectGrader base class, which provides (among other things) the all important register() method (more on that later).

The most important change is how grade assertions are created. At the start of each grade test, the register() method should be run for each assertion that will take place over the course of that test. The register method returns a GradeAssertion object that can be used later in the method to indicate the success/failure of each assertion (through the new correct() and incorrect(String result) methods respectively).

This was a big change as it allows me to dynamically calculate the total number of points instead of requiring the grader developer to manually add up the total points, but with a much greater degree of safety than previously offered. Making these calls to register at the outset of a test method guarantees that any unexpected exception that occurs will still result in a correct total. Capturing the unexpected exception and associating it with the test itself provides an explanation of why some assertions may not have been run.

To add some concreteness to this, here’s a quick example of what a grader might look like:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

public class CalcGrader extends ProjectGrader {
 
  @GraderTest(name = "Add Test")
  public void add() {
    GradeAssertion twoNums = register(5,
      "Adds two numbers correctly");
    GradeAssertion nullNums = register(5,
      "Handles null parameter");
 
    // Setup to get to student's add method
    int result = add(2, 3);
    if (result == 5) {
      twoNums.correct();
    } else {
      twoNums.incorrect("Adding [2, 3] failed. Expected [5]," + 
      "Found [" + result + "]");
    }
 
    try {
      add(null, null);
      nullNums.incorrect(
        "Null parameters did not cause exception");
    } catch (IllegalArgumentException e) {
      nullNums.correct();
    }
}

It’s a bitch changing a domain model (in this case, adding a layer for GradeTest in between Grade and GradeAssertion). I think I got everything hashed out, short of a few variables that may still carry the old naming (GradeItem became GradeAssertion). So far, so good, as grading project 3 was significantly easier and cleaner than it was when I started it on the old model. There are still a few more projects in the semester, so I’ll have a few more chances to see how this stands up before actually declaring CodeTurtle “1.0″.