I’m working on question sets for introductory programming classes in languages like Java, C and Matlab, with a possibility of extending into other commonly-used languages (in our department) like Python and Verilog. The idea is that these questions could be deployed into a protected lab test or an open in-class “flipped homework” environment. Key to this is the ability to assign similar, but different, questions to students, either simultaneously or sequentially. In a class with 400 to 600 students, that means developing quite a number of questions. It also means coming up with verified solutions. And it cries out for a systematic approach to verifying.
Enter unit testing.
Unit tests aren’t common in the robotics and embedded systems worlds, but are really important here. While I’ll apply these question banks to my Matlab and Java courses, I’m baselining my question banks in C and leveraging the fine work by the ThrowTheSwitch folks to use their Unity unit testing framework. C doesn’t have a commonly-used unit testing framework and the embedded space, where C is really the main language, is particularly difficult to apply a unit testing framework to. That’s due to a number of factors, including small memory footprint, a wide range of architectures (1990s and early-2000s 8-bit chips like the PIC16F84 and ATMEGA328 are still super popular!), and the lack of standard display on most devices.
In my series of posts on automated grading in C and VPL I wrote how I am using the Diagon app to generate ASCII-art flowcharts as the basis for the student questions. Each flowchart is generated using a C-like pseudo-code syntax, making it straight-forward to convert the pseudo-code into proper C. I take each of these, create a separate function for each in my TeacherReferenceSolutions.c file and then place those functions into an array with function pointers.
From there, I select which of those teacher reference functions I want to aim for and then manually write up the student solution in StudentSolution.c. This is similar to the process followed for assignments in Virtual Programming Lab, but it’s all done within the CLion IDE (or whatever C compatible IDE or command line process you want to follow).
Here is the main.c file:
// main.c
#include <stdlib.h>
#include <stdio.h>
#include <locale.h> // for UTF8
#include <math.h>
#include "unity.h"
#include "unity_internals.h"
#include "TeacherReferenceSolutions.h"
#include "StudentSolution.h"
#include <stdlib.h>
//#include <wchar.h> // for printing characters to console...
#include <string.h> // for searching for strings in Unix / C
//#include <windows.h> // for printing characters to console when in CLion / Windows. Don't need in VPL / Linux.
#include <stdint.h>
#include <time.h> // for randomization.
//#include "ChooseSolutionTimeOfDay.h" // to be able to time slice the day to choose a problem linked to time.
/* references:
* file location: https://stackoverflow.com/questions/31495311/clion-c-cant-read-open-txt-file-in-project-directory
* add these two b/c https://github.com/ThrowTheSwitch/Unity/blob/master/docs/UnityGettingStartedGuide.md
* and https://stackoverflow.com/questions/61114392/linker-error-with-unity-c-unit-testing-framework
* eclass example: https://eclass.yorku.ca/mod/vpl/forms/executionfiles.php?id=876284
*
* */
#define INDEX_REFERENCE_TO_TEST 10; /* to-do: make sure to select the teacher reference solution */
// Define the flowchart file name.
#define FLOWCHARTFILE "diagon_flowcharts.txt" // needs to be in the directory where compiling happens.
// define the six pairs of inputs for the six unit tests.
#define TEST1_INPUT1 (-809) // Unit test 1 inputs
#define TEST1_INPUT2 (-1092)
#define TEST2_INPUT1 (-51) // Unit test 2 inputs
#define TEST2_INPUT2 9
#define TEST3_INPUT1 (-10) // Unit test 3 inputs
#define TEST3_INPUT2 52
#define TEST4_INPUT1 (-2) // Unit test 4 inputs
#define TEST4_INPUT2 2
#define TEST5_INPUT1 (-1) // Unit test 5 inputs
#define TEST5_INPUT2 23
#define TEST6_INPUT1 0 // Unit test 6 inputs
#define TEST6_INPUT2 (-5)
#define TEST7_INPUT1 1 // Unit test 7 inputs
#define TEST7_INPUT2 (-1)
#define TEST8_INPUT1 2 // Unit test 8 inputs
#define TEST8_INPUT2 1
#define TEST9_INPUT1 3 // Unit test 9 inputs
#define TEST9_INPUT2 23
#define TEST10_INPUT1 5 // Unit test 10 inputs
#define TEST10_INPUT2 102367
#define TEST11_INPUT1 7 // Unit test 11 inputs
#define TEST11_INPUT2 5
#define TEST12_INPUT1 9 // Unit test 12 inputs
#define TEST12_INPUT2 23
#define TEST13_INPUT1 10 // Unit test 13 inputs
#define TEST13_INPUT2 7
#define TEST14_INPUT1 13 // Unit test 14 inputs
#define TEST14_INPUT2 17
#define TEST15_INPUT1 24 // Unit test 15 inputs
#define TEST15_INPUT2 18
#define TEST16_INPUT1 53 // Unit test 16 inputs
#define TEST16_INPUT2 78
#define TEST17_INPUT1 105 // Unit test 17 inputs
#define TEST17_INPUT2 800
#define TEST18_INPUT1 1026 // Unit test 18 inputs
#define TEST18_INPUT2 1012
#define TEST19_INPUT1 129 // Unit test 19 inputs
#define TEST19_INPUT2 (-222)
#define TEST20_INPUT1 892 // Unit test 20 inputs
#define TEST20_INPUT2 (-1999)
/* In a global context,
* Define a function pointer type for functions (currently one output and one input @ 32bits int)
* */
typedef int32_t (*referenceSolutions)(int32_t);
/* - - - - - - - - - - - - - - - - - - - - - - -
* Teacher solution array. (to-do: customize this)
*
* Add more functions in the array if there are more... and align with TeacherReferenceSoloutions.c and diagon_flowcharts.txt
*
* - - - - - - - - - - - - - - - - - - - - - - */
referenceSolutions teacherFunctions_g[] = {function1, function2, function3, function4, function5, function6, function7, function8, function9, function10}; // To-do: add more here.
int32_t teacherFuncIndex_g = 0;
int sizeTeacherFunctionsArray_g = 10; // To-Do: update to reflect size of teacherFunctions_g
void setUp(void){
// set stuff up here
}
void tearDown(void){
// clean stuff up here.
}
/* ------------------------------------------------------------------
* Start of definitions of unit tests. (six of them, currently. )
* vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv */
/* Define Test 1: test the student function against the teacher function. */
void test1(){
// The specific test types are found in unity.h
// Student method first, then reference.
TEST_ASSERT_EQUAL_INT32(studentMethod(TEST1_INPUT1), teacherFunctions_g[teacherFuncIndex_g](TEST1_INPUT1)); // expected out vs. input
}
/* Define Test 2: test the student function against the teacher function */
void test2(){
// The specific test types are found in unity.h
TEST_ASSERT_EQUAL_INT32(studentMethod(TEST2_INPUT1), teacherFunctions_g[teacherFuncIndex_g](TEST2_INPUT1)); // expected out vs. input
}
/* Define Test 3: test the student function against the teacher function */
void test3(){
// The specific test types are found in unity.h
TEST_ASSERT_EQUAL_INT32(studentMethod(TEST3_INPUT1), teacherFunctions_g[teacherFuncIndex_g](TEST3_INPUT1)); // expected out vs. input
}
/* Define Test 4: test the student function against the teacher function */
void test4(){
// The specific test types are found in unity.h
TEST_ASSERT_EQUAL_INT32(studentMethod(TEST4_INPUT1), teacherFunctions_g[teacherFuncIndex_g](TEST4_INPUT1)); // expected out vs. input
}
/* Define Test 5: test the student function against the teacher function */
void test5(){
// The specific test types are found in unity.h
TEST_ASSERT_EQUAL_INT32(studentMethod(TEST5_INPUT1), teacherFunctions_g[teacherFuncIndex_g](TEST5_INPUT1)); // expected out vs. input
}
/* Define Test 6: test the student function against the teacher function */
void test6(){
// The specific test types are found in unity.h
TEST_ASSERT_EQUAL_INT32(studentMethod(TEST6_INPUT1), teacherFunctions_g[teacherFuncIndex_g](TEST6_INPUT1)); // expected out vs. input
}
void test7(){
// The specific test types are found in unity.h
TEST_ASSERT_EQUAL_INT32(studentMethod(TEST7_INPUT1), teacherFunctions_g[teacherFuncIndex_g](TEST7_INPUT1)); // expected out vs. input
}
void test8(){
// The specific test types are found in unity.h
TEST_ASSERT_EQUAL_INT32(studentMethod(TEST8_INPUT1), teacherFunctions_g[teacherFuncIndex_g](TEST8_INPUT1)); // expected out vs. input
}
void test9(){
// The specific test types are found in unity.h
TEST_ASSERT_EQUAL_INT32(studentMethod(TEST9_INPUT1), teacherFunctions_g[teacherFuncIndex_g](TEST9_INPUT1)); // expected out vs. input
}
void test10(){
// The specific test types are found in unity.h
TEST_ASSERT_EQUAL_INT32(studentMethod(TEST10_INPUT1), teacherFunctions_g[teacherFuncIndex_g](TEST10_INPUT1)); // expected out vs. input
}
void test11(){
// The specific test types are found in unity.h
TEST_ASSERT_EQUAL_INT32(studentMethod(TEST11_INPUT1), teacherFunctions_g[teacherFuncIndex_g](TEST11_INPUT1)); // expected out vs. input
}
void test12(){
// The specific test types are found in unity.h
TEST_ASSERT_EQUAL_INT32(studentMethod(TEST12_INPUT1), teacherFunctions_g[teacherFuncIndex_g](TEST12_INPUT1)); // expected out vs. input
}
void test13(){
// The specific test types are found in unity.h
TEST_ASSERT_EQUAL_INT32(studentMethod(TEST13_INPUT1), teacherFunctions_g[teacherFuncIndex_g](TEST13_INPUT1)); // expected out vs. input
}
void test14(){
// The specific test types are found in unity.h
TEST_ASSERT_EQUAL_INT32(studentMethod(TEST14_INPUT1), teacherFunctions_g[teacherFuncIndex_g](TEST14_INPUT1)); // expected out vs. input
}
void test15(){
// The specific test types are found in unity.h
TEST_ASSERT_EQUAL_INT32(studentMethod(TEST15_INPUT1), teacherFunctions_g[teacherFuncIndex_g](TEST15_INPUT1)); // expected out vs. input
}
void test16(){
// The specific test types are found in unity.h
TEST_ASSERT_EQUAL_INT32(studentMethod(TEST16_INPUT1), teacherFunctions_g[teacherFuncIndex_g](TEST16_INPUT1)); // expected out vs. input
}
void test17(){
// The specific test types are found in unity.h
TEST_ASSERT_EQUAL_INT32(studentMethod(TEST17_INPUT1), teacherFunctions_g[teacherFuncIndex_g](TEST17_INPUT1)); // expected out vs. input
}
void test18(){
// The specific test types are found in unity.h
TEST_ASSERT_EQUAL_INT32(studentMethod(TEST18_INPUT1), teacherFunctions_g[teacherFuncIndex_g](TEST18_INPUT1)); // expected out vs. input
}
void test19(){
// The specific test types are found in unity.h
TEST_ASSERT_EQUAL_INT32(studentMethod(TEST19_INPUT1), teacherFunctions_g[teacherFuncIndex_g](TEST19_INPUT1)); // expected out vs. input
}
void test20(){
// The specific test types are found in unity.h
TEST_ASSERT_EQUAL_INT32(studentMethod(TEST20_INPUT1), teacherFunctions_g[teacherFuncIndex_g](TEST20_INPUT1)); // expected out vs. input
}
/* ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
* End of definitions of unit tests.
* ------------------------------------------------------------------- */
// Run the tests...
int main(void)
{
int32_t indexSpecificFlowchart = 0;
int32_t indexFlowchartTemp = 0;
int32_t specificFlowchartStart = 0;
int32_t specificFlowchartEnd = 0;
// specify which function to run the test on.
indexSpecificFlowchart = INDEX_REFERENCE_TO_TEST;
teacherFuncIndex_g = indexSpecificFlowchart - 1;
/* run the unit tests */
/* ---------------------------------------------
* Start: Run all of the unit tests (six right now)
* vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv */
printf("Multiple tests underway...\n");
UNITY_BEGIN();
RUN_TEST(test1);
RUN_TEST(test2);
RUN_TEST(test3);
RUN_TEST(test4);
RUN_TEST(test5);
RUN_TEST(test6);
RUN_TEST(test7);
RUN_TEST(test8);
RUN_TEST(test9);
RUN_TEST(test10);
RUN_TEST(test11);
RUN_TEST(test12);
RUN_TEST(test13);
RUN_TEST(test14);
RUN_TEST(test15);
RUN_TEST(test16);
RUN_TEST(test17);
RUN_TEST(test18);
RUN_TEST(test19);
RUN_TEST(test20);
/* add more RUN_TEST()s, as needed */
/* ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
* End of running the unit tests. (six of them, currently. )
* ------------------------------------------------------------------- */
// wrap up the unity testing...
return UNITY_END();
}