Simon Gray
Email: <sgray AT SPAMFREE wooster DOT edu>
I am an Associate Professor in the Department of Mathematical Sciences at The College of Wooster.
Background
Our CS 2 course is taught in C++. There are three hours of lecture per week and one three hour lab per week. The lab projects are fairly small and can often be completed within the lab time. Students also complete several larger programming projects during the semester.
I introduced unit test into our CS2 course last fall using Web-CAT to do the automated grading. I used Web-CAT again in the course this spring. We used eclipse with some plugins developed at Virginia Tech to support unit testing with CxxTest and electronic submissions using eclipse’s submitter tool.
Overall it has been a positive experience for me as well as for the students. Here I share some thoughts on what worked well, the challenges of taking this approach, and what I will do differently the next time I use this approach in a course. If you are thinking about using Web-CAT for a course, I would be happy to talk with you about my experiences.
How It Works
Within Web-CAT you create an assignment, specifying the due date (day and hour), whether late assignments will be accepted (how late and daily penalty), how many points are assigned by Web-CAT via automated grading (the rest is determined by you after inspection of the source code) and provide your test suite for the assignment. There is a pretty intuitive interface for doing all this.
Students get to submit as many times as they wish up to the due date. With each submission they get feedback on how many points they earned out of the total possible. Web-CAT provides some feedback when there are errors. All of a student’s test cases must pass before Web-CAT will attempt to grade the assignment using your test suite. A student’s grade depends on the number of test cases that pass. Once the deadline has passed, you can go to the assignment to view all the submissions and the grades assigned by Web-CAT. You can insert comments on the quality of the code directly into an online copy of the code. Once you have marked the assignment as “finished” the student receives an email saying that the assignment has been graded and they can login to see the results. Students can see their final score and look through the online code to see your comments. There are other functions within Web-CAT, but these are the ones I used the most.
The Good
* Test driven development is a sounder way for students to develop solutions than their traditional approach of trying to solve the entire thing in one sitting and turning it in when it compiles and produces plausible output.
* TDD forces students to be more thoughtful about their solutions. To develop the tests they have to think carefully about the relationship between an operation and the state of an object. The notion of object state and how it can change is an area where students (mine anyway) often stuggle, so additional work via preparing test cases is useful.
* Providing tests to demonstrate that their code provides the expected behavior and does so correctly gives students more responsibility for the quality of their solutions.
* As a result of my experience over the last two semesters, I am now putting more effort into testing in my other classes. For example, in our Programming Languages and Compilers class this semester, students add features to the language Triangle, then modify the compiler to implement those additions. After the first assignment, I discovered that students were verifying their implementation in a largely haphazard way. I now have them prepare trial programs to test the new language features as part of the process of describing the syntax and semantics of the new features. That is, before they do the implementation in the compiler. I have been surprised by how hard they find this. Clearly there is a need to develop materials to help students with this process.
Challenges
* You need to create assignments to that produce classes that are easily testable. My original assignments were not all geared this way. I had time to change some, but not all of them to make them testable.
* You need to have coded a solution and produced your own test suite before you hand out the assignment, because you need to provide your own test suite to Web-CAT.
* Make sure the interface is clear or there will be problems when Web-CAT compiles your test suite with student code. If you aren’t careful, “undeclared identifier” errors abound! I like to produce ADT specifications and have students develop classes from them. I needed to work very hard to make sure the operation signatures were clear in the ADT so the students would all develop the same interface. Even typos can be a problem. A typo in the name of one operation was spotted and corrected by some students and taken as is by others, producing some programs that would compile with my test code and others that would not. This is frustrating for me and drives students nuts.
* This leads to another challenge. Sometimes the output from Web-CAT is not very enlightening and occasionally students would complain that they did not have enough guidance to make the necessary fixes when Web-CAT told them there was a problem. The typo problem is one example. Web-CAT said something to the effect that the problem may be related to the student code not implementing all the required methods. Well, they had developed all the right methods, they just used the method name with the wrong spelling. This was my fault (and was, of course, eventually corrected), but there was a lot of frustration before the fix was in place. Other times the problem lay with the student not reading the specification carefully so they produced an incorrect signature or the method didn’t do what it was supposed to do. That is their bad, but Web-CAT did not give them much guidance in what to do to fix the problem.
* Some of my students did not adjust quickly to the hard deadline Web-CAT enforces. I would occasionally get frustrated/angry emails from students saying there was “something wrong with Web-CAT” because it wouldn’t let them submit their solution. The problem was almost always that they were submitting past the deadline. This is more a culture issue – prior to now, my students just expected that they could turn in the assignment late. Web-CAT does support late submissions with a penalty. I just didn’t use it, largely because I really wanted to get students away from the idea that it is okay to start late because you can always submit late. Starting late usually means that the solution is going to be put together quickly and without the careful thought I am trying to encourage them to apply. Another facet of this is encouraging students to take more responsibility for what they do. What I have said to them is: "I have given you access to a tool that will let you know whether your solution is correct or not, so there is no excuse for not getting a perfect score on every assignment. If you leave it to the last minute so that you don't have the time to make multiple submissions, learn from the feedback and correct mistakes, that is your fault and you have to suffer the consequences." To their credit, they don't always like this, but they accept that it is fair.
* Teaching students how to develop test cases. This was much more challenging than I thought it would be. Students caught on to CxxTest quickly and could see how to express test cases using it, but they found it difficult to imagine a wide range of scenarios to test. This improved gradually over the semester, but not as fast as I expected.
To do next time
* item Much more can be done with assignment construction. I simply did not have the time to convert all my labs and programming assignments into problems that were easily testable. It would be valuable to begin a discussion in this area.
* Develop better materials to teach students about unit testing and how to develop effective tests. Again, discussion and sharing of ideas and materials is welcome.
* Do a better job of explaining to students how they can use Web-CAT as a tool and how to interpret the results Web-CAT provides. In the fall, I was learning along with the students. I did a better job in the spring, but student feedback this semester tells me I need to do more. This is an area where I need to work with Steve.