VPL: Basic Custom Evaluation Example

Sometimes you need to go beyond the default settings in VPL. That means creating a custom “run” and “evaluate” setup. Here, I’ll simplify a great example by Dominique Tiebaut, Emeritus prof at Smith College, written for an assignment in C. (archive) We’ll check a “hello world” example in C, but won’t use the default evaluate.cases tool built into VPL. Instead, we’ll develop three main files:

1. the student’s submission template (“requested files”)
2. the run script (vpl_run.sh)
3. the evaluate script (vpl_evaluate.sh)

Let’s begin with helloWorld.c, the student submission template:

/* helloWorld.c
 * This is the student submission template.
 * Normally, the student would modify this template.  But not in this example.
 * Place this file in "Gear" -> Requested Files */

#include <stdio.h>

int main(void){
   printf("hello world.\n");
}

You can test this file out manually, from a terminal on any computer with the gcc compiler tools like this:

gcc helloWorld.c -o helloWorld

and then executing it like this:

./helloWorld

and it will print

hello world.

to your screen. Nothing complicated. The original example included multiple tests that required injecting input data into the student submission. We’re simply going to look at the output without providing any variations. There are a few things to note about the two files, vpl_run.sh and vpl_evaluate, that we’ll add to our assignment:

1. VPL will execute a bash script called vpl_execution. This script is based on both your vpl_run.sh and vpl_evaluate.sh scripts.
2. Two instances of “#!/bin/bash” in each of the vpl_run.sh and vpl_evaluate.sh files. The first one allows the script to run directly. The second will feed subsequent commands into the vpl_execution script. VPL will then run vpl_execution to test the student’s submission and submit the grade.
3. Extra backslashes. The “run a bash script to create a bash script” process that we’re following requires that some elements in your script be preceded by a ‘\’ (backslash).
   • For example:
     • Normally: gcc ${prog1}.c -o ${prog1}
     • Here: gcc \${prog1}.c -o \${prog1}
4. The vpl_run.sh file is nearly identical to vpl_evaluate.sh.
   • Start by writing vpl_evaluate.sh
   • Include the Grade and Comment print statements in vpl_evaluate.sh (e.g. echo "Grade :=>> \$grade")
   • Copy vpl_evaluate.sh into vpl_run.sh
   • Modify your print statements (e.g. remove the “:=>>” to become echo "Possible grade : \$grade")

The shell scripts will look like this skeleton:

#! /bin/bash
# skeleton vpl_evaluate.sh script

#variables
maxGrade=100
prog1=helloWorld

# everything that follows will be put into vpl_execution
cat > vpl_execution <<EEOOFF
#! /bin/bash
 
# --- program tested  ---
gcc \${prog1}.c -o \${prog1} 

# --- create solution. Put in a file called solution.txt ---
cat > solution.txt <<EOF
hello world.
EOF

# --- calculate a grade ----
grade=\$( expr \$maxGrade )

# --- compare the student's solution to the real solution ----


# --- Provide grade back to Moodle ---
echo "Grade :=>> \$grade"
echo "Comment :=>> and a comment..."

EEOOFF
# create of vpl_execution is done.

# allow vpl_execution to be executed. 
chmod +x vpl_execution

Our goal is to provide students with feedback and a grade, either if the student submission is incorrect:

and to provide different feedback and a better grade when the student submission is correct:

So, here is the complete vpl_evaluate.sh. It will compare the model solution you provide with the student’s result. Note the use of the =>> in the print statements. These will direct output related to the actual grade and any comments to Moodle. Dominique Thiebaut added some filtering in the student solution to get rid of white space differences, which is a nice touch. Here is vpl_evaluate.sh:

#!/bin/bash
#
# vpl_evaluate.sh script 
# The output of the student program is tested against the expected output file.
# 
# James Andrew Smith; drsmith@yorku.ca July 2021
# Based on example by D. Thiebaut at Smith College
# http://www.science.smith.edu/dftwiki/index.php/Tutorial:_Moodle_VPL_--_Testing_a_C_Program
# -----------------------------------------------------------------


cat > vpl_execution <<EEOOFF
#!/bin/bash

# Variables (don't add whitespace)
studentProgram=helloWorld
maxGrade=100
minGrade=0
consolationGrade=10
 
# --- program tested (no extension) ---
gcc -o \${studentProgram} \${studentProgram}.c



# --- create test input files ---
cat > solution.txt <<EOF
hello world.
EOF


grade= \$minGrade

for i in 1 ; do
   echo "Comment :=>>-TEST \$i"

   # ==============================================
   # TEST i
   # ==============================================
   #--- run program, capture output, display to student ---

   ./\${studentProgram}  &> user.out
   cp user.out user.out.org

   #--- remove non numbers and non minus---
   #cat user.out | sed 's/[^0-9\ -]*//g' > dummy.out
   #mv dummy.out user.out
 
   #--- remove multiple spaces --- 
   cat user.out | sed 's/  */ /g' > dummy.out
   mv dummy.out user.out

   #--- remove blank lines ---
   cat user.out | sed '/^\s*$/d' > dummy.out
   mv dummy.out user.out

   #--- compute difference --- 
   diff -y -w --ignore-all-space user.out solution.txt > diff.out
   #echo "----- diff.out ------"
   #cat diff.out
   #echo "---------------------"
   diff -y -w --ignore-all-space user.out solution.txt > diff.out

   
   #--- reject if different ---
   if ((\$? > 0)); then
      echo "Comment :=>>- Your output is incorrect."

      #--- display test file ---
      echo "Comment :=>>- Your program tested with:"
      echo "<|--" 
      cat solution.txt
      echo "--|>"

      echo "Comment :=>> ---------------"
      echo "Comment :=>>- Your output:"
      echo "Comment :=>> ---------------"
      echo "<|--"
      cat user.out
      echo "--|>"
      echo ""
      echo "Comment :=>> ---------------"
      echo "Comment :=>>- Expected output (only the numbers): "
      echo "Comment :=>> ---------------"
      echo "<|--"
      cat solution.txt
      echo "--|>"
   
      grade=\$( expr \$minGrade )
   
      #--- consolation grade ("thanks for coming out")---   
      #grade=\$( expr \$consolationGrade )

      # --------------------- REWARD IF CORRECT OUTPUT -----------------
   else
      #--- good output ---
      echo "Comment :=>>- Congrats, your output is correct."
      echo "Comment :=>> --------------------------------."
      echo "<|--"
      cat user.out.org
      echo "--|>"
      grade=\$( expr \$maxGrade )
   fi
done

echo "Grade :=>> \$grade"

EEOOFF

 
chmod +x vpl_execution

and here is the run script, vpl_run.sh:

#!/bin/bash
#
# vpl_run.sh script 
# The output of the student program is tested against the expected output file.
# 
# James Andrew Smith; drsmith@yorku.ca July 2021
# Based on example by D. Thiebaut at Smith College
# http://www.science.smith.edu/dftwiki/index.php/Tutorial:_Moodle_VPL_--_Testing_a_C_Program
# -----------------------------------------------------------------


cat > vpl_execution <<EEOOFF
#!/bin/bash

# Variables (don't add whitespace)
studentProgram=helloWorld
maxGrade=100
minGrade=0
consolationGrade=10
 
# --- program tested (no extension) ---
gcc -o \${studentProgram} \${studentProgram}.c



# --- create test input files ---
cat > solution.txt <<EOF
hello world.
EOF


grade= \$minGrade

for i in 1 ; do
   echo "Comment :-TEST \$i"

   # ==============================================
   # TEST i
   # ==============================================
   #--- run program, capture output, display to student ---

   ./\${studentProgram}  &> user.out
   cp user.out user.out.org

   #--- remove non numbers and non minus---
   #cat user.out | sed 's/[^0-9\ -]*//g' > dummy.out
   #mv dummy.out user.out
 
   #--- remove multiple spaces --- 
   cat user.out | sed 's/  */ /g' > dummy.out
   mv dummy.out user.out

   #--- remove blank lines ---
   cat user.out | sed '/^\s*$/d' > dummy.out
   mv dummy.out user.out

   #--- compute difference --- 
   diff -y -w --ignore-all-space user.out solution.txt > diff.out
   #echo "----- diff.out ------"
   #cat diff.out
   #echo "---------------------"
   diff -y -w --ignore-all-space user.out solution.txt > diff.out

   
   #--- reject if different ---
   if ((\$? > 0)); then
      echo "Comment :- Your output is incorrect."

      #--- display test file ---
      echo "Comment :- Your program tested with:"
      echo "<|--" 
      cat solution.txt
      echo "--|>"

      echo "Comment : ---------------"
      echo "Comment :- Your output:"
      echo "Comment : ---------------"
      echo "<|--"
      cat user.out
      echo "--|>"
      echo ""
      echo "Comment : ---------------"
      echo "Comment :- Expected output (only the numbers): "
      echo "Comment : ---------------"
      echo "<|--"
      cat solution.txt
      echo "--|>"
   
      grade=\$( expr \$minGrade )
   
      #--- consolation grade ("thanks for coming out")---   
      #grade=\$( expr \$consolationGrade )

      # --------------------- REWARD IF CORRECT OUTPUT -----------------
   else
      #--- good output ---
      echo "Comment :- Congrats, your output is correct."
      echo "Comment : --------------------------------."
      echo "<|--"
      cat user.out.org
      echo "--|>"
      grade=\$( expr \$maxGrade )
   fi
done

echo "Possible Grade : \$grade (you need to evaluate for official grade)"

EEOOFF

 
chmod +x vpl_execution

Conclusion

This is a summary of how to set up a custom evaluation scheme in VPL. While it was designed for a simple C assignment it could be modified for other programming languages.

Don’t forget!

When setting up the activity, don’t forget to include the local VPL server location

and don’t forget to turn on “run”, “evaluate” and “autograde” execution options:

---

James Andrew Smith is an associate professor in Electrical Engineering and Computer Science Department in York University’s Lassonde School.  He lived in Strasbourg, France and taught at the INSA Strasbourg and Hochschule Karlsruhe while on sabbatical in 2018-19 with his wife and kids. Some of his other blog posts discuss the family’s sabbatical year, from both personal and professional perspectives.