The priorities for the Web-CAT development team are proposed and debated on this page.

To express your opinion about the need for (or lack of urgency on) any particular item, edit this page and add a "+1" or "-1" comment like this: 

  * +1 I really need this feature because ....
   @ SIG @

  * -1 This feature is not very important because ....
   @ SIG @

The @SIG@ symbol will be replaced by your personal wiki signature when you save the page (don't put in the spaces around SIG in the example above--they are in the example so that they won't be replaced and you can see what the original will look like). Periodically, items in this list will be moved around based on votes cast in this way.

Major Changes

The items in this section involve significant effort to complete, which limits the number of them that can be undertaken at any given time. They are arranged roughly in current priority order. Feel free to cast your own votes for things you want to see have higher priority.

  1. {1} Add auto-update support for subsystems and grading plug-ins within Web-CAT. The idea is that the application should be "self-patching", being able to download and install updates to various subsystems or plug-ins over the web. This by itself isn't a critical capability, but turns out to be tightly tied to correctly implementing the next item. Since it is on the critical path for what is below, it has been elevated in priority.

  2. {1} Make the WAR distribution completely self-contained. Reconfigure the distributed Web-CAT WAR file so that the major grading plug-ins, together with all their external dependences, can be included within the WAR. That means including ANT, Checkstyle, PMD, all necessary supporting Perl modules, and so on. Basically, all external dependencies except for MySQL and a servlet container should be included in the self-configuring, self-installing distributable. Doing this requires major changes to the way grading plug-ins work, and how they interact with their environment.

  3. Complete the "instructor's guide" recipes in the cookbook.

    • +1 More docs would be great, but they are rarely "complete". I suggest you split this into more specific topics, and focus on what people want most. The wiki logs might also provide data on most-read topics. -- ClifKussmaul 2008-03-07 12:01:25

  4. Add full peer review support for running peer review assignments on Web-CAT.

    • +1 this would be wonderful and can easily connect to our minimUML and Moodle projects. -- ManuelPerez 2006-06-20 15:53:19

    • 0 Consider improving Moodle integration, and then improving rubrics, peer review, etc, in Moodle. -- ClifKussmaul 2008-03-07 12:01:25

  5. Build in a flexible batch-processing capability that allows instructors to perform actions on all submissions across an entire course section (or even multiple sections). Many educators that are interested in automated grading are also interested in preventing or detecting plagiarism and other forms of cheating. A great deal of work by other researchers has gone into tools for automatically detecting plagiarism. Two commonly used tools include MOSS and JPlag, both of which provide plagiarism detection services to the CS education community over the internet. While Web-CAT provides a sophisticated plug-in API for managing the processing that takes place for each individual submission, it does not provide a convenient way for instructors to perform “batch” post-processing across all submissions in one or more courses. We will extend Web-CAT’s plug-in support to include instructor-definable post-processing steps to carry out across an entire set of submissions, and implement plug-ins that provide for sending a set of submissions to MOSS or JPlag for cheating analysis.

  6. Implement flexible support for rubric-based hand-grading. Web-CAT already provides a rich interface for course staff to manually “mark up” student code with comments, suggestions, and point deductions, and already lets instructors choose their own balance between the amount of credit allocated to hand-grading and the amount devoted to the automated analyses provided by the assessment plug-ins they have chosen to use. However, many instructors prefer to use more structured strategies for hand-grading, which are usually framed as grading rubrics. Indeed, one can imagine grading rubrics as the basis for performing even the automated portions of an evaluation. We will expand Web-CAT’s support for manual grading so that instructors can define their own grading rubrics that course staff can use for manual scoring. We will also explore how instructor-defined rubrics can be combined with and used as part of scoring during automated analysis.

    • +1 This will support TA grading by having a rubric available. Make sure it is possible to publish the rubric as an option so the students can see how they will be graded. This also opens up the possibility for using Web-CAT for submission and grading of non-programming assignments (UML diagrams, for example). -- ManuelPerez 2006-06-20 15:53:19

    • 0 Consider improving Moodle integration, and then improving rubrics, peer review, etc, in Moodle. -- ClifKussmaul 2008-03-07 12:01:25

  7. Implement a generalized data reporting back end that allows instructors to access the full range of data collected by Web-CAT. Web-CAT has full information on every compilation error, every student test case failure, every instructor test case failure, every error message, every exception, every late submission, every change to a student’s program between successive submissions, and so on. While it is entirely impractical to collect and manage such information for traditional hand-graded programming assignments, this body of information is automatically collected as part of normal operation when automated assessment tools are used. We will develop an open-ended, generalized data reporting capability that allows instructors to design custom reports that extract the data they wish to use. We will use existing open-source tools for the reporting engine and report design language, and implement the necessary access functions for exposing Web-CAT’s internal data model. The result will be a powerful contribution for exploring new and innovative instructional tracking questions that arise with CS educators in practice.

    • +1 Good to have to support not only data collection but also browsing of class performance. -- ManuelPerez 2006-06-20 16:01:08

  8. Complete the Web-CAT-to-Moodle gateway implementation, so that Web-CAT assignments can be offered through Moodle, with submissions and feedback available in a fully-transparent manner.

    • 0 Not so high in priorities in my list. The current version of Web-CAT has an easy way to create assignments and the plugins we use are appropriate. This would be nice, but not high in priority list. -- ManuelPerez 2006-06-20 16:01:08

    • +1 Very useful, and preferable to adding general course-management functions to Web-CAT. -- ClifKussmaul 2008-03-07 12:01:25

  9. Add support for individual extensions to assignment due dates, so that instructors can grant permission for some students to have different due dates than the rest of the class. Also, explore the possibility of using this feature to support contract-based individualized assignment scheduling, in the vein of personalized system of instruction (PSI).

    • +1 This is a much needed feature. -- ManuelPerez 2006-06-20 16:01:08

  10. Look into building a Web-CAT-specific JUnit wrapper for instructor reference tests that provides more sophisticated tracking and generation of hints and feedback messages to students.

    • +1 Yes, I have some simple methods that I use that encapsulate reflection-based calls to check that classes have appropriate constructors or names, etc. I can make these available. Another set of classes/methods needed is ways to query Web-CAT settings. For example, the number of hints provided or the level of hints provided can be determined dynamically by a set of JUnit classes by querying Web-CAT at runtime. We can also obtain student PID and deadlines, etc. -- ManuelPerez 2006-06-20 16:01:08

  11. Add support for generating difficulty and discrimination scores for individual test cases written by instructors.

    • 0 Would be nice to have, but lower priority at this stage. -- ManuelPerez 2006-06-20 16:01:08

  12. Add WebDAV support for instructors to use when managing the uploaded files they use to configure grading plug-ins. WebDAV support could also be used in managing the files within uploaded plug-ins, or to allow read-only access to the generated files produced for each assignment (browsable by instructors and TAs).

    • -1 Don't think it is as a needed as it appears at least now. -- ManuelPerez 2006-06-20 16:01:08

  13. Look into implementing Manuel Perez's date-based hint-limiting logic, or Marmoset's "release testing tokens" scheme.

    • +1 I would love this :) -- ManuelPerez 2006-06-20 16:01:08

  14. Write an appropriate student style guide for coding/style, tied into feedback messages generated by Checkstyle and PMD.

    • -1 Would be nice to have but it is low priority. -- ManuelPerez 2006-06-20 16:01:08

  15. Implement a generalized LDAP authentication plug-in for the authenticator subsystem.

    • -1 -- ManuelPerez 2006-06-20 16:01:08

    • 0 Consider improving Moodle integration, and then utilizing Moodle authentication options. -- ClifKussmaul 2008-03-07 12:01:25

  16. Complete the on-line help pages that are available through Web-CAT.

    • -1 -- ManuelPerez 2006-06-20 16:01:08

    • +1 More docs would be great, but they are rarely "complete". I suggest you split this into more specific topics, and focus on what people want most. The wiki logs might also provide data on most-read topics. -- ClifKussmaul 2008-03-07 12:01:25

  17. Re-engineer the grading queue so that there can be multiple grading queues on different processors on different machines, so that grading can be run in parallel (perhaps even on different OSes in a heterogeneous cluster).

Minor Changes

The items in this section involve much less effort to complete than the major changes above. As such, many of the items in this list are pursued opportunistically, when other enhancements are being made in the same areas of the code. As such, the ordering here is not very strong, since opportunity drives what gets done here more than the ordering. However, feel free to cast your own votes to express your opinions about which items you want to see get done. Votes matter a lot in this list.

  1. Incorporate Grant Braught's PMD and Checkstyle customizations (including more readable messages targeted at CS1 students) into the Java TDD grading plug-in.
  2. Modify the "alert box" that shows errors and other feedback messages on Web-CAT's dynamically generated web pages. Currently, this box only supports errors, but it should be extended to support a wider variety of messages so that it can be used for general feedback purposes.
  3. Add appropriate feedback to the My Profile page so that it indicates when one's password is changed successfully (requires a better "alert box").
  4. Track down and fix the source of the assignment summary graphing bugs. It appears that incorrect/inconsistent data accumulates in the graph summary statistics somehow. Recalculating from scratch always fixes this, but that requires manual intervention.
  5. Fix the bugs in rendering of stacked area graphs. When the top line of a stacked area graph goes from convex to concave, for some reason JFreeChart incorrectly paints the filll on top of the line instead of underneath it, leading to some weird-looking graphs. Figure out this problem and fix it.
  6. Fix the x-axis scaling problems where some student graphs are "all crunched up" instead of being plotted with a reasonable x-axis scale. This is probably due to bugs in the auto-scaling code written for this purpose.
  7. Write an installation walk-through recipe for the cookbook.
  8. Clean up and round out the features of the file manager component, which instructors use to edit the file space for their uploaded files.
  9. Fix the bugs in the most-recently-used comment dropdown list that graders use when marking up code.
  10. Add help links with appropriate URLs to the Checkstyle and PMD messages that are inserted into in student code as feedback. Ideally, these URLs should point to some sort of style guide. Support is already in place to generate these links, but the necessary data for which links go with which messages is missing.
  11. Fix the bug where graphical summaries in assignment offerings cause page crashes for administrators who are trying to display or edit information about assignment offering objects.

Other Suggested Changes

This section provides a place where readers can contribute other items that they would like to see added to the priority list for developers. If you're new to Web-CAT or you're not a developer, it is safest to put new items here. Items from this list that receive useful feedback will be moved into the major or minor changes section by developers.

Comments

Add a comment

CategoryDeveloperInfo CategoryCookbook