Introduction

Web-CAT will run through a list of grading scripts in an order you specify. Your grading scripts will manipulate a properties file that Web-CAT will read and extract grading information.

As an instructor, you will usually tailor an already existing grading script to your needs rather than write a completely new one.

• If you are using an already existing grading script, modify it to suit your needs according to the instructions supplied with the script. Then follow the instructions in To configure automated grading below.

• If you are writing a completely new grading script, read the grading script technical information in To create your own grading scripts below.

To configure automated grading

The Automated Grading Scripts table. On this table, you specify the scripts that will grade each student's submission.

1. On the Create/Edit an Assignment screen, click the Add button next to Add another step to add a grading script to the assignment's grading sequence.

2. Choose or upload a grading script.

   • To use a pre-existing grading script, click the circle next to the grading script you would like to use, and then click Next.

   • To upload a new grading script, click the circle next to Upload a new grading script. Then click Next. If prompted, click the circle next to the file in the grading script that Web-CAT should execute, and then click Next.

3. Enter a Time Limit for the grading script, if desired.'

4. You can add more grading scripts by repeating steps 1 through 3. Grading scripts will be run sequentially in the order shown.

   • To change the order of the grading scripts, click the up or down arrows in the Move column of the grading script you would like to move.

5. You're now ready for step 15 in Creating and Editing Assignments.

To create your own grading scripts

• == About grading scripts ==

  A grading script can be any executable program. It could be a Perl script, a batch file, or an executable compiled from C++, for example. It can be a single file or a zip file containing multiple files.

  Communication to and from each grading script is done via a single properties file. Web-CAT writes a properties file, and then passes the name of that properties file to the grading script as the only command line parameter. The script can rewrite or modify this properties file as needed. Web-CAT then re-reads the properties file and extracts grading results and feedback information from it. If you have a sequence of multiple grading scripts, Web-CAT will pass the same, newly modified properties file tot he next grading script in the sequence. Web-CAT will re-read the properties file, and this cycle repeats until all grading scripts in the sequence finish. == About properties files == These files are supported by many languages. They're similar to .INI files.

  • They are plain text.

  • They contain one key=value pair per line.

  • Lines that start with # are comments.

  • === A sample properties file passed to a grading script === Here's a sample of what an initial properties file that Web-CAT could automatically generate and pass to a grading script:

      -------------------------------------------
      # Automatically generated configuration file
      #
      # Wed Sep  8 13:38:06 2004
      
      max.score.correctness=40.0
      max.score.tools=20.0
      numReports=0
      pid=lrendin
      resultDir=H:/Web-CAT.data/Grader/VTEdAuth/Fall2004/10/Project1/lrendin/4/Results
      scriptHome=H:/Web-CAT.data/Grader/UserScripts/DBAuth/edwards/1705f04L01_zip
      timeout=90
      workingDir=F:/Web-CAT.scratch/WorkArea/VTEdAuth/lrendin
      ------------------------------------------- 
      

    • max.score.correctness is the highest amount of points to award if the student's submitted program produces correct behavior and output.

      • This value is determined from information you entered when you were ../Setting Submission Rules. In brief, Points available - Pts from TA - Pts from tools = max.score.correctness.

    • max.score.tools is the highest amount of points that static analysis tools may assign. These tools usually look at the form and layout of the code, examine how the code is commented, and the like.)

      • This value is from the Pts from tools field on the ../Setting Submission Rules page.

    • pid is the Personal ID string that uniquely identifies the student whom this submission is from.

    • resultDir gives the location in which the student's program will produce output.

    • scriptHome gives the location of the grading scripts that will grade this submission.

    • timeout is the maximum number of seconds that Web-CAT will wait for the grading script and student submission to terminate.

    • workingDir is the current working directory in which the grading scripts will run.

    === A sample properties file returned from a grading script === Your grading script creates feedback for the student by modifying the properties file during the grading process. Web-CAT reads the newly-modified properties file and displays your feedback to the student. Here's a sample of what Web-CAT might see when a grading script finishes editing the properties file:

      # Automatically generated configuration file
      #
      # Wed Sep  8 13:38:06 2004
      
      codeMarkup1.className=CollectBeepersTest
      codeMarkup1.conditionals=0
      codeMarkup1.conditionalsCovered=0
      codeMarkup1.deductions=-6
      codeMarkup1.elements=6
      codeMarkup1.elementsCovered=6
      codeMarkup1.loc=80
      codeMarkup1.methods=6
      codeMarkup1.methodsCovered=6
      codeMarkup1.ncloc=32
      codeMarkup1.remarks=6
      codeMarkup1.statements=7
      codeMarkup1.statementsCovered=7
      codeMarkup2.className=CollectBeepers
      codeMarkup2.conditionals=0
      codeMarkup2.conditionalsCovered=0
      codeMarkup2.deductions=-1
      codeMarkup2.elements=2
      codeMarkup2.elementsCovered=2
      codeMarkup2.loc=71
      codeMarkup2.methods=2
      codeMarkup2.methodsCovered=2
      codeMarkup2.ncloc=45
      codeMarkup2.remarks=2
      codeMarkup2.statements=33
      codeMarkup2.statementsCovered=33
      max.score.correctness=40.0
      max.score.tools=20.0
      numCodeMarkups=2
      numReports=2
      pid=lrendin
      report1.file=test.log
      report1.mimeType=text/html
      report2.file=instr.log
      report2.mimeType=text/html
      resultDir=H:/Web-CAT.data/Grader/VTEdAuth/Fall2004/10/Project1/lrendin/4/Results
      score.correctness=17.7777777777778
      score.tools=13
      scriptHome=H:/Web-CAT.data/Grader/UserScripts/DBAuth/edwards/1705f04L01_zip
      statElementsLabel=Methods Executed
      timeout=90
      workingDir=F:/Web-CAT.scratch/WorkArea/VTEdAuth/lrendin
      ------------------------------------------- 
      

    • ==== Analyzing student submissions with the codeMarkup property ====

      The grading script generates codeMarkup# properties for each graded file in the student submissions. For example, if a student's submission has three files, then your grading script should create a listing containing the properties codeMarkup1, codeMarkup2, and codeMarkup3. The grading script analyzes each codeMarkup--each file in the student's submission--for three types of code:

      • conditional paths, such as  if  statements or boolean tests in  while  loops;

      • methods, which are functions in the student's code; and

      • statements, which are any other line of executable code.

      But not all of these types have to be used when calculating a student's grade; they merely provide "raw" data about what the student's testing actually accomplished. Web-CAT provides the element and elementsCovered properties to allow you to use a formula that combines the data about conditionals, methods, and statements to calculate a grade. For example, in freshman classes when students are just learning how to program, requiring that each method be tested might be enough; your grading script could simply ignore the data about conditional paths and statements. In fact, this is what the sample properties file above is doing, since the number of elements equals the number of methods. For high-level courses, you could require that each statement be tested.

      In the end, you make the decision about the grading formula that your grading script uses. Your grading script will use the data about conditionals, methods, and statements to write the element and elementsCovered properties.

      The listing below provides a reference for all codeMarkup properties:

      codeMarkup#.className

      the name of the file represented by the codeMarkup.

      codeMarkup#.conditionals

      the number of conditional paths that the grading script found in the file associated with the codeMarkup#. (A conditional path is controlled by a boolean test, such as  if ( x == 5 )  or  while ( !fooVector.atEnd() ) .)

      • Note:  if  statements and  while  loops typically contain two paths, not one. Since the statement can be evaluated with true or false conditions--each of which causes different results--there are two execution paths that a student will need to test.

      codeMarkup#.conditionalsCovered

      the number of conditional paths counted by codeMarkup#.conditionals that the student's tests executed.

      codeMarkup#.methods

      the number of methods (function calls) that the grading script found in the file associated with the codeMarkup#.

      codeMarkup#.methodsCovered

      the number of methods counted by codeMarkup#.methods that the student's tests executed.

      codeMarkup#.statements

      the number of statements (lines of executable code) that the grading script found in the file associated with the codeMarkup#.

      codeMarkup#.statementsCovered

      the number of statements counted by codeMarkup#.statements that the student's tests executed.

      codeMarkup#.elements

      the number of required elements (some combination of conditonals, methods, and statements) that the student's tests must execute for a perfect test score. This property is written and determined by a formula in your grading script that calculates your desired combination of conditionals, methods, and statements.

      codeMarkup#.elementsCovered

      the number of elements counted by codeMarkup#.elements that the student's tests executed.

      codeMarkup#.deductions

      the number of points (given as a negative value) that the student lost on the file associated with the codeMarkup#.

      codeMarkup#.loc

      the number of "lines of code" in the file associated with the codeMarkup#.

      codeMarkup#.ncloc

      the number of "non-commented lines of code" in the file associated with the codeMarkup#.

      codeMarkup#.remarks

      the number of comments generated by static analysis tools that have been directly embedded in the source code view for the student.

      score

      represents the cumulative grade for the entire submission.

      correctness

      is the number of points that the student was awarded for correct program output and behavior.

      tools

      is the number of points that the student was awarded by the grading tools. These usually examine the student's source code for proper formatting and documentation.

      numCodeMarkups

      the number of codeMarkups in the properties file. Note that this is equal to the number of files in the student's submission.

      ==== Generating reports to provide student feedback ==== The feedback that Web-CAT provides the student doesn't have to be a raw listing of the individual codeMarkups.

      Your grading script can provide zero, one, or multiple feedback files. Each feedback file is called a report, and they are described by sequentially-numbered report# properties. Each report may have any of the following properties:

      report#.file

      the report's file name. It does not include the path or the directory.

      report#.mimeType

      a MIME type describing the contents of the file. Usually, this either "text/plain" or "text/html". See Common MIME Types on the Internet for more help on MIME types.

      report#.to

      indicates the intended recipient of the feedback associated with this report#. This value can be  student, admin, administrator, or both. Default value is student if omitted.

      • Reports addressed to student will appear in the student's web feedback report in Web-CAT.

      • Reports addressed to admin or administrator will be e-mailed to both the course instructor and the Web-CAT administrator.

      report#.inline

      indicates whether the file associated with this report# should be included as inline text within the main feedback Web page shown to the student or is presented as a link to a downloadable file in a drop-down list. This value can be true,false,on,off,yes, or no. Default value is true if omitted.

      report#.label

      text of the link in the drop-down list used to display non-inlined reports. Required if report#.inline is false.

      report#.border

      indicates whether or not a horizontal seprator should be placed before and after an inlined report. Default value is false if omitted.

Go back to Creating and Editing Assignments.

Go to the table of contents for this InstructorsUserGuide.