Grading student programs using ASSYST
SIGCSE '97 Proceedings of the twenty-eighth SIGCSE technical symposium on Computer science education
An online programming assessment tool
ACE '03 Proceedings of the fifth Australasian conference on Computing education - Volume 20
Rethinking computer science education from a test-first perspective
OOPSLA '03 Companion of the 18th annual ACM SIGPLAN conference on Object-oriented programming, systems, languages, and applications
WIPE: a programming environment for novices
ITiCSE '05 Proceedings of the 10th annual SIGCSE conference on Innovation and technology in computer science education
CUTE: a concolic unit testing engine for C
Proceedings of the 10th European software engineering conference held jointly with 13th ACM SIGSOFT international symposium on Foundations of software engineering
MSR '05 Proceedings of the 2005 international workshop on Mining software repositories
Proceedings of the 35th annual ACM SIGPLAN-SIGACT symposium on Principles of programming languages
Looper: Lightweight Detection of Infinite Loops at Runtime
ASE '09 Proceedings of the 2009 IEEE/ACM International Conference on Automated Software Engineering
Non-termination checking for imperative programs
TAP'08 Proceedings of the 2nd international conference on Tests and proofs
Using run time traces in automated programming tutoring
Proceedings of the 16th annual joint conference on Innovation and technology in computer science education
| Hi-index | 0.00 |
Infinite looping problems that keep student programs from termi-nating may occur in many kinds of programming assignments. While non-terminating code is easier to diagnose interactively, it poses different concerns when software tests are being run auto-matically in batch. The common strategy of using a timeout to preemptively kill test runs that execute for too long has limita-tions, however. When one test case gets stuck in an infinite loop, forcible termination prevents any later test cases from running. Worse, when test results are buffered inside a test execution framework, forcible termination may prevent any information from being produced. Further, overly generous timeouts can de-lay the availability of results, and when tests are executed on a shared server, one non-terminating program can delay results for many people. This paper describes an alternative strategy that uses a fine-grained timeout on the execution of each individual test case in a test suite, and that adaptively adjusts this timeout dynamically based on the termination behavior of test cases com-pleted so far. By avoiding forcible termination of test runs, this approach allows all test cases an opportunity to run and produce results, even when infinite looping behaviors occur. Such fine-grained timeouts also result in faster completion of entire test runs when non-terminating code is present. Experimental results from applying this strategy to 4,214 student-written programs are dis-cussed, along with experiences from live deployment in the class-room where 8,926 non-termination events were detected over the course of one academic year.