This page is part of the WebCatDevelopmentBlog.

Web-CAT Development Activities: October 2005

• [2005-10-17] Code coverage measurement for C++

  While at OOPSLA, I found some time to integrate BullseyeCoverage (see http://www.bullseye.com) into our C++ grading script. Everything seems to be working on my local development machine, and I will upload it to our production server once I return from the conference.

• [2005-10-17] Improved messages in student feedback for Java

  A long while back, ManuelPerez made a number of suggestions for improving the readability and effectiveness of the feedback summary that Web-CAT generates for students. It took a while for me to get around to it, but I've completely worked over the feedback messages that students see on the summary page for Java assignments. Now there is a more direct explanation of exactly how their work is assessed, exactly how their score is calculated, and exactly what they can do to improve their testing results.

• [2005-10-17] Requiring all student tests to pass If you are truly following TDD principles, then no code gets released until all its unit tests pass 100%. For students, this means that no student code submissions are acceptable unless the student's tests pass 100%. The C++ grading script, as well as the generic scripts we use for a few other languages, enforce this choice. Historically, the Java grading script hasn't--instead, it simply multiplies the student's score by the number of his or her tests that pass, so failing tests reduce the score. I added another option to the Java script so instructors can choose which scoring model they want to use, so now both are available.

• [2005-10-17] Supporting different coverage metrics for Java

  GrantBraught had requested support to use statement-level coverage in calculating student scores on Java assignments. I had designed in basic support to allow instructors to choose different coverage metrics for use in scoring since Clover already collects this data, but didn't have it completed implemented yet. It took a little work, but I finished the missing pieces and now instructors can choose one of five different coverage criteria for scoring student assignments, ranging from simple method coverage on the low end, to a combination of method, statement, and condition coverage on the high end. I'm not positive whether clover implements full condition/decision measurement or simply branch measurement, but I should find out. Also, one must be careful with using some of the more rigorous metrics when students write exception handlers, since they will need to fully test their exception handling code like everything else. We might work out a way to optionally "screen out" exception handlers from coverage measures, but we haven't figured out how to do that effectively yet.

• [2005-10-17] A few Java grading tweaks I am sitting in the eTX workshop listening to the educationally oriented paper presentations and just uploaded some minor fixes in the Java grading script. Last night, I fixed a bug where the "20 point" max deduction in the markup.properties file (which no one ever customizes) was always in use, irrespective of the number of points set for static analysis tools by the instructor. I fixed the markup.properties file and edited the main Java grading script so that the point balance the instructor sets is respected. Further, I added a scaling factor so that instructors can decide how the generic deductions in the default markup.properties scoring scheme get translated into real point deductions for students. Finally, I disabled the PMD check for duplicate string literals, since such duplication often appears in test cases in innocuous ways.

• [2005-10-16] Miscellaneous C++ fixes During our most recent C++ assignment, I noticed a couple of small bugs in the C++ grader. I fixed them this weekend while I was at OOPSLA. Compiler error messages were being incorrectly labeled as warnings, causing erroneous messages about "no tests included in submission" to be generated when the student's code failed to compile. Also, path information referring to reference test files was leaking out under some circumstances. These are both fixed.

• [2005-10-16] Highlighting uncovered code in Java The Java grader uses Clover to determine which parts of a student's submission are executed during the student's testing. Code that is tested insufficiently is highlighted. It turns out that the "whole line" highlighting disappeared in Web-CAT, due to a change in the css markup generated by the latest Clover release. I fixed our css file to bring the whole-line highlighting back for untested code.

• [2005-10-13] Incorrect indentation in C++ student programs The C++ grader uses an open-source beautifier to generate an HTML view of a student's source code. It turns out the beautifier was trying to "enforce" good indentation style, and was, in fact, messing up some student code. Further, the beautifier didn't properly respect varying tab width settings used by different IDEs. I had to revise the beautifier code to suppress its attempts to "fix up" indentation so that it renders the student's true indentation. I also added supports for preferred tab widths by language, and set the C++ tab width to be 4. Our Eclipse workspace settings, as well as those we use in BlueJ, use an indentation level of 4 spaces, and use spaces-only rather than tabs for indentation. However, some students use environments that include tabs for indentation with a width of 4, so the beautifier uses a width of 4 (instead of the default of 8) for tabs in C++ code.

• [2005-10-13] Preventing heap corruption in C++ student programs

  The C++ grader uses the heap tracker in our EclipsePlugins/CxxTestPlugin. One feature of the heap tracker is that it detects (and prevents) heap corruption by intercepting faulty delete calls (double deletes, or delete calls on non-heap-allocated memory). However, I was still seeing student programs where such behavior was producing a "crash" during grading. It turns out that the heap tracker had a small cut-and-paste error that was causing this interception code to be "turned off" in some cases, allowing the erroneous deletes to proceed, resulting in a corrupted heap. I fixed this bug in the grading script we use on Web-CAT. I'll have to push the fix into a new version of our EclipsePlugins/CxxTestPlugin later, when I have time.

• [2005-10-13] Sometimes raw XML instead of rendered output shows up when entering comments

  GrantBraught e-mailed me today to report some behavior others may have also noticed. Every once in a while, when grading a student assignment to enter comments, the edit area displays raw XML instead of a rendered view of the student code. This appears to be related to a problem we thought we had fixed. Last year, we found that occasionally, one's browser would render a page with some images missing or with the style sheet omitted. The HTML source was identical, it is just that some of the links on the page (to a css file, a gif file, etc.) were not being loaded by the browser for some reason. Although this affected IE users more often, it did show up in other browsers, too. We never did find the error, but I always suspected it was a problem with the WebObjects adapter that sits between WO applications and the web server--after all, it is the web server that is responsible for serving up these other files. We had used relative URLs that started with "/" for these kinds of external resources. Back in the spring, I changed all these URLs to use absolute addresses, and that seemed to fix the problem. After Grant's question, I checked, and sure enough, there were a few relative paths to the HTML editing widget we use for entering comments. I replaced those with absolute URLs and now have my fingers crossed that this might fix the problem ...

• [2005-10-12] Beautifier ignores backslashes in student code The Perl implementation of the open-source beautifier we are using was accidentally ignoring backslash escapes in student source code, which was causing it to produce incorrect markup for some student programs. The problem was actually a minor bug in the Beautifier's language definition files. The Beautifier handles escape characters fine, but the C++ and Java definition files did not define backslash as an escape character. I made the repair on our server. Note that this bug only affected languages other than Java, since our Java grading script actually uses Clover to generate the HTML view of student code; all other languages use the Beautifier instead.

• [2005-10-12] Segmentation fault hints for C++ programs While looking over student results on a C++ assignment, I realized that hints about segmentation faults and other program failures were not always making it to students--even though the C++ grading script specifically detects them. I took a look at the grading script and fixed a small logic error that was causing these hints to be treated improperly. Now, any and all hints caused by segmentation faults or assertion failures are shown to students, and the limit on the number of hints shown to students only applies to regular test failures, not signal-based failures like these.

• [2005-10-05] Style/Coding was still appearing when unwanted

  SimonGray reported a simple bug today. He is using Web-CAT for C++ assignments, where no static analysis tools have yet been plugged into the tool chain. As a result, he was leaving blank the "tool points" amount in his submission rules, which is supposed to completely suppress the Style/Coding line of feedback in score summaries that students see. Unfortunately, the line was still appearing :-(. Fortunately, it was an easy bug to fix, and now the line is properly omitted when it isn't being used.

• [2005-10-04] Unescaped <'s in compiler messages

  ManuelPerez also noticed a bug a few days ago. When the Java compiler generates error messages, html entities were not being properly escaped. As a result, compiler errors containing characters like < and > were not displaying properly in browsers. He found this out when he received a compiler error that quoted a source line containing a generic class instantiation. I fixed the Java grading script to escape the necessary html entities (the C++ grading script already did this, but it was missing on the Java script).

• [2005-10-04] Generating PDF source code printouts

  ManuelPerez asked for the ability to generate downloadable PDF printouts of student submissions. We want to use these in peer review research with one of our graduate students, ScottTurner. It took a bit of playing around, but now the java grading script can call enscript and ps2pdf to create a single PDF document containing all of the source files in a Java submission. The instructor can configure this as a script option when setting up the processing of an assignment. When it is enabled, the PDF file is generated as part of the grading process, and appears as an additional entry in the list of items the student can download while their viewing feedback on Web-CAT. It should be easy enough to add this feature for other languages as well if (and when) it is needed. Incidentally, the hardest parts were: printing all Java files, no matter the package or directory structure, fixing the page size for enscript (which uses A4 instead of US letter by default), and fixing the fancy headers so they do not leak server information (using file names rather than full server-side path names in printout headers).

• [2005-10-03] External header and linker libraries for c++ grading I added a few extra features to the TDD-based g++ grading script today. Our latest programming assignment uses a custom-written framework, so I added the ability for instructors to upload a collection of header files to add to the include path. I also added the ability for instructors to add a binary library archive that will be included in the linkage step for student submissions. Both of these items can be set for a group of assignments, and then additional values can be added for specific assignments.