Automated Student Evaluations in C (part 4)

In this fourth post, I'm going to modify the way the unit tests in C get picked up by the VPL "run" and "evaluate" scripts so that rather than look for a particular phrase returned by the unit test (something like "The unit test expected to see 5 but the student's function returned 1"), it's just looking for a specific number of times that the word "PASS" is returned. And, just to make sure that the student doesn't print that word a bunch of times, we also look to ensure that the word "FAIL" is not returned by the unit tests.

A key thing to point out here is that the reference (teacher) solution that the students are supposed to aim for is a function found in the main.c file. As well, all of the unit tests are defined there, as are the values that are fed into the unit tests. You can have as many unit tests as you want, but you need to inform both vpl_run and vpl_evaluate of that number so that both of them can search for the correct number of times that "PASS" is returned.

Here you can see a succesful "run" and "evaluate":

and here are the same tests, but for an incorrect student submission:

And here are the files:

main.c is the key file:

// main.c
#include <stdlib.h>
#include <stdio.h>
#include "studentTemplate.h"
#include "teacherSolution.h"
#include "unity.h"
#include "unity_internals.h"

// Constants & defined fixed values
#define INPUT1_TEST1 2      // Test 1 values.
#define INPUT2_TEST1 3
#define INPUT1_TEST2 -300   // Test 2 values.
#define INPUT2_TEST2 -12
#define INPUT1_TEST3 106    // Test 3 values
#define INPUT2_TEST3 30
#define INPUT1_TEST4 -2     // Test 4 values
#define INPUT2_TEST4 -3

// don't forget to update the vpl_run and vpl_evaluate scripts 
// to reflect the number of unit tests.



// Unit Testing "unity" setup.
void setUp(void){
		// Unit Testing "unity" setup.
		// don't actually need anything in here.
		// but we need this, otherwise it won't compile.
}

// Unit Testing "unity" wrap up..
void tearDown(void){
		// Unit Testing "unity" wrapping up.
		// don't actually need anything in here.
		// but we need this, otherwise it won't compile.
}

// A set of Unit tests
void unitTest1(){ // First unit test.
	TEST_ASSERT_EQUAL_INT_MESSAGE(teacherMethod(INPUT1_TEST1, INPUT2_TEST1),
							studentMethod(INPUT1_TEST1, INPUT2_TEST1),
							"Fix your function.  It doesn't match the teacher reference.");	
	}
void unitTest2(){ // Second unit test.
	TEST_ASSERT_EQUAL_INT_MESSAGE(teacherMethod(INPUT1_TEST2, INPUT2_TEST2),
							studentMethod(INPUT1_TEST2, INPUT2_TEST2),
							"Fix your function.  It doesn't match the teacher reference.");	
	}
void unitTest3(){ // Third unit test.
	TEST_ASSERT_EQUAL_INT_MESSAGE(teacherMethod(INPUT1_TEST3, INPUT2_TEST3),
							studentMethod(INPUT1_TEST3, INPUT2_TEST3),
							"Fix your function.  It doesn't match the teacher reference.");	
	}
void unitTest4(){ // 4th unit test.
	TEST_ASSERT_EQUAL_INT_MESSAGE(teacherMethod(INPUT1_TEST4, INPUT2_TEST4),
							studentMethod(INPUT1_TEST4, INPUT2_TEST4),
							"Fix your function.  It doesn't match the teacher reference.");	
	}
	
// Teacher reference solution.
int teacherMethod(int a, int b)
{
	return a+b; 	// right answer
}
	
int main(void)
{

	
	UNITY_BEGIN();
	// Run all the tests.
	RUN_TEST(unitTest1);
	RUN_TEST(unitTest2);
	RUN_TEST(unitTest3);
	RUN_TEST(unitTest4);

	return -1; 
}

And here is the student's function under test, studentTemplate.c (a.k.a the "requested" file)

#include <stdlib.h>
#include <stdio.h>

int studentMethod(int a, int b)
{
//	return a-b;  	// wrong answer	
	return a+b; 	// right answer
}

next up we have the two headers for teacher and student functions, teacherSolution.h and studentTemplate.h:

// teacherSolution.h
// Teacher reference solution.
int teacherMethod(int, int);


// studentTemplate.h
int studentMethod(int, int);

And now for the VPL scripts, starting with the new vpl_run.sh script:

#!/bin/bash
#
# vpl_run.sh script 
# Here, a student function is called by a main function.
# The output of the student function is tested against an expected output string.
# 
# James Andrew Smith; drsmith@yorku.ca July 2021 (updated August 2023)
# Based on example by D. Thiebaut at Smith College
# http://www.science.smith.edu/dftwiki/index.php/Tutorial:_Moodle_VPL_--_Testing_a_C_Program
# now at
# https://web.archive.org/web/20190208191644/http://www.science.smith.edu/dftwiki/index.php/Tutorial:_Moodle_VPL_--_Testing_a_C_Program
#
# August 2023 update: removed the SED whitespace processing as (1) it messes with
# the ASCII art from the flowchart and (2) it's not necessary with the intoduction
# of the Unity unit tester.
# -----------------------------------------------------------------


cat > vpl_execution <<EEOOFF
#!/bin/bash

# the number of times we need to see a unit test pass.
NUM_PASSES=4    # TO-DO : update this (in vpl_run and _evalute) to correspond to main.c unit test number.

# output file from execution of the main program
OUTFILE="user.out"

# Variables (don't add whitespace)
studentProgram=studentTemplate
teacherProgram=main
maxGrade=1
minGrade=0
consolationGrade=0
 
# Set the minimum grade.  (slashes due to this script writing vpl_executable)
grade=\$( expr \$minGrade )

# --- program tested (no extension) ---
gcc -Wall *.c -o a.out  

# Run the executable and output to a file.
./a.out  &> \$OUTFILE

# If the file contains the string "PASS" a certain number of times (set above)
# but also check to see that "FAIL" is not returned.
if [ \$(grep -o "PASS" \$OUTFILE | wc -l) -eq \$NUM_PASSES ]; then
    # Check if the file does not contain the string "FAIL"
    if ! grep -q "FAIL" \$OUTFILE; then
        echo "The test on your code succeeded."
    else
        echo "The test on your code did _not_ succeed."
    fi
else
    echo "The test on your code did _not_ succeed."
fi

echo "***********************"
echo "The test on the student's code returned the following message:"
cat \$OUTFILE
echo "***********************"

echo "***********************"
echo "No grade has been saved."
echo "Use the evaluation button for grading."
echo "***********************"

EEOOFF

 
chmod +x vpl_execution

And the vpl_evaluate.sh file:

#!/bin/bash
#
# vpl_evaluate.sh script 
# Here, a student function is called by a main function.
# The output of the student function is tested against an expected output string.
# 
# James Andrew Smith; drsmith@yorku.ca July 2021 (updated August 2023)
# Based on example by D. Thiebaut at Smith College
# http://www.science.smith.edu/dftwiki/index.php/Tutorial:_Moodle_VPL_--_Testing_a_C_Program
# now at
# https://web.archive.org/web/20190208191644/http://www.science.smith.edu/dftwiki/index.php/Tutorial:_Moodle_VPL_--_Testing_a_C_Program
# Updated Aug 11, 2023 to do a grep search for "PASS"
# -----------------------------------------------------------------


cat > vpl_execution <<EEOOFF
#!/bin/bash

# the number of times we need to see a unit test pass.
NUM_PASSES=4    # TO-DO : update this (in vpl_run and _evalute) to correspond to main.c unit test number.

# output file from execution of the main program
OUTFILE="user.out"

# Variables (don't add whitespace)
studentProgram=studentTemplate
teacherProgram=main
maxGrade=1							# Maximum grade is set here. (uploaded to eClass)
minGrade=0
consolationGrade=0
 
# Set the minimum grade.  (slashes due to this script writing vpl_executable)
grade=\$( expr \$minGrade )

# --- program tested (no extension) ---
gcc -Wall *.c -o a.out  

# Run the executable and output to a file.
./a.out  &> \$OUTFILE


# If the file contains the string "PASS" a certain number of times (set above)
# but also check to see that "FAIL" is not returned.
if [ \$(grep -o "PASS" \$OUTFILE | wc -l) -eq \$NUM_PASSES ]; then
    # Check if the file does not contain the string "FAIL"
    if ! grep -q "FAIL" \$OUTFILE; then
      echo "Comment :=>>- Congratulations: your function is CORRECT."
      echo "Comment :=>> -----------------"
      echo "Comment :=>>- The result of the testing program, with the student function included:"
      echo "<|--"
      cat user.out
      echo ""
      echo "--|>"
      grade=\$( expr \$maxGrade )
    else
      echo "Comment :=>>- Your program is NOT correct. Fix it."
      echo "Comment :=>> ---------------"
      echo "Comment :=>>- The result of the testing program, with the student function included:"
      echo "Comment :=>> ---------------"
      echo "<|--"
      cat user.out
      grade=\$( expr \$minGrade )
    fi
else
      echo "Comment :=>>- Your program is NOT correct. Fix it."
      echo "Comment :=>> ---------------"
      echo "Comment :=>>- The result of the testing program, with the student function included:"
      echo "Comment :=>> ---------------"
      echo "<|--"
      cat user.out
      grade=\$( expr \$minGrade )
fi
   

# Send the grade to eClass via a special-formatted comment.
echo "The following grade is being sent to eClass: \$grade"
echo "Grade :=>> \$grade"

EEOOFF

 
chmod +x vpl_execution

And then the unity.c, unity.h and unity_internals.h files need to be added. I won't repeat them here.

---

James Andrew Smith is a Professional Engineer and Associate Professor in the Electrical Engineering and Computer Science Department of York University's Lassonde School, with degrees in Electrical and Mechanical Engineering from the University of Alberta and McGill University.  Previously a program director in biomedical engineering, his research background spans robotics, locomotion, human birth and engineering education. While on sabbatical in 2018-19 with his wife and kids he lived in Strasbourg, France and he taught at the INSA Strasbourg and Hochschule Karlsruhe and wrote about his personal and professional perspectives.  James is a proponent of using social media to advocate for justice, equity, diversity and inclusion as well as evidence-based applications of research in the public sphere. You can find him on Twitter. Originally from Québec City, he now lives in Toronto, Canada.