[DynInst_API:] A new report fuzz testing

[Barton Miller <bart@xxxxxxxxxxx>]

Hi all. As many of you have noted, fuzz testing has become a very hot area
of research, with all sorts of new and impressive innovations. While I've
followed this, recent experiences with some of these new fuzz tools had
me wondering if the traditional simple fuzz testing that we created 30
years ago was still useful and applicable.  So, last semester, I again
offered fuzz testing as one of the semester project options in my graduate
OS class (CS736).

This paper is the result of that study.

Title: The Relevance of Classic Fuzz Testing: Have We Solved This One?

Authors: Barton P. Miller, Mengxiao Zhang and Elisa R. Heymann
					
Abstract:
   As fuzz testing has passed its 30th anniversary, and in the face of the
   incredible progress in fuzz testing techniques and tools, the question
   arises if the classic, basic fuzz technique is still useful and
   applicable? In that tradition, we have updated the basic fuzz tools and
   testing scripts and applied them to a large collection of Unix utilities
   on Linux, FreeBSD, and MacOS. As before, our failure criteria was whether
   the program crashed or hung. We found that 9 crash or hang out of 74
   utilities on Linux, 15 out of 78 utilities on FreeBSD, and 12 out of 76
   utilities on MacOS. A total of 24 different utilities failed across the
   three platforms. We note that these failure rates are somewhat higher
   than our in previous 1995, 2000, and 2006 studies of the reliability of
   command line utilities.
   
   In the basic fuzz tradition, we debugged each failed utility and
   categorized the causes the failures. Classic categories of failures, such
   as pointer and array errors and not checking return codes, were still
   broadly present in the current results. In addition, we found a couple of
   new categories of failures appearing. We present examples of these
   failures to illustrate the programming practices that allowed them to
   happen.

   As a side note, we tested the limited number of utilities available in a
   modern programming language (Rust) and found them to be of no better
   reliability than the standard ones.

You can fetch the paper at:

    ftp://ftp.cs.wisc.edu/paradyn/technical_papers/fuzz2020.pdf
    (A version will be posted on arxiv.org soon.)

Comments and feedback are definitely welcome!

regards,

--bart