Automated student testing of Matlab code, Part 1 | Prof. James Andrew Smith's Blog

It's important to have dynamic 24/7 interactive activities for programming classes. There are plenty of solutions out there, including Matlab Grader and Möbius. The folks at Jetbrains have their Academy system, too. What I need, at this stage, is a no bells-and-whistles, auto-grader for Matlab code that can run a unit test on a student's text file and securely record whether the student got it right or wrong.

Here at York University's EECS department we commonly use a number of different systems:

EECS "Submit" (command line and web)
Virtual Programming Lab
PrairieLearn
Matlab Grader
Other Moodle tools like H5P

We also have a system called "Lab Test Mode" which works to selectively shutdown internet access for desktop computers in our computer labs and allow us to deploy secured tests to our students. While it would be great to have VPL, PrairieLearn and Matlab Grader available to for testing Matlab exercises within Lab Test Mode, complete with access to the student grade book on Moodle/eClass, that doesn't appear to be feasible at this stage in 2023.

One workable approach, however, could be to have Matlab unit tests work on student text files submitted through Web Submit from a desktop computer locked down in Lab Test Mode. Behind the Web Submit system we would need to have Matlab run a unit test on student's work.

Here's a proof of concept using three files. The objective is to have the student write a Matlab function called myFunction(a) in a file called myFunction.m. The function is to accept one input parameter, a, and return a value. Inside the function the student is to implement an equation that would be provided by the system. This target equation would be pre-determined from a text file embedded in the test system. The third file would be the unit test function that Matlab would apply to the student's submission to determine if the student's submission was correct. Here are the three files:

1st file: the model equation in equation.txt.

5*a + 2

2nd file: the student's submission, myFunction.m, (assume they started with a template)

function output = myFunction(a)
    y = 5*a + 2; % student wrote this.
    output = y;
end

3rd file: the unit test function, testMyFunction.m:

function testMyFunction
    if exist('equation.txt', 'file') == 2
        % File exists
        % read the file to obtain the equation
        % default to another equation if no file is present.
        fileID = fopen('equation.txt','r');
        firstLine = fgets(fileID);
        fclose(fileID);
    else
        % File does not exist
        firstLine = "2*a + 0";
    end
    % convert the string into a function with an input parameter, a.
    fileFunction = str2func(['@(a)',firstLine]);
    % Test the myFunction with sample input
    input1 = randi([0,100]);
    expectedOutput = fileFunction(input1);      % teacher's solution (reference)
    testedOutput = myFunction(input1);          % student's solution. 
    try
        assert( testedOutput == expectedOutput);
        result = true;
    catch
        result = false;
    end
    if (result == true)
        disp([sprintf("Test succeeded. The tested function worked as tested.  Input %d yielded an answer of %d for an expected answer of %d", input1, testedOutput,  expectedOutput)])
    else
        disp([sprintf("Test failed. The tested function did not work as tested.  Input %d yielded an answer of %d for an expected answer of %d", input1, testedOutput,  expectedOutput)])
    end
end

The two displayed lines at the end of the unit test would need to be modified to be able to record a grade within Web Submit. Plus, need to get creative about what kinds of equations and problems can be inserted into the equation.txt file. We also need a way to customize the problem text that is displayed to the student based on the content of the equation.txt file. That could include randomization or customization based on student ID.

Regardless, the above three files can be tested and worked on within desktop Matlab. Think of them as a proof of concept to refine this idea further.

I intend to revisit this after consulting with other instructors and amazing technical staff so that we can come up with something that works within the toolkit that we have in our department.

James Andrew Smith is a Professional Engineer and Associate Professor in the Electrical Engineering and Computer Science Department of York University's Lassonde School, with degrees in Electrical and Mechanical Engineering from the University of Alberta and McGill University. Previously a program director in biomedical engineering, his research background spans robotics, locomotion, human birth and engineering education. While on sabbatical in 2018-19 with his wife and kids he lived in Strasbourg, France and he taught at the INSA Strasbourg and Hochschule Karlsruhe and wrote about his personal and professional perspectives. James is a proponent of using social media to advocate for justice, equity, diversity and inclusion as well as evidence-based applications of research in the public sphere. You can find him on Twitter. Originally from Québec City, he now lives in Toronto, Canada.