Testing Your Tests?

This is an idea that I still consider in the “alpha” phase, and one that I have not yet even begun to code a proof of concept of, but it is an idea that I would be interested in some community feedback on.  Testing has always been an integral part of the culture of Perl and one of the great strengths of the Perl community.  At a NY Perl Mongers meeting in June I saw a preview of the YAPC talk by James Keenan on testing that I think provides a nice introduction to testing for any that are not familiar with Perl’s Test::* modules. Here is a link to the video of the YAPC talk if any are interested:  http://youtu.be/A31VhNeGaw0

What I have been thinking about a lot lately though, is “how do you test your tests?”  In other words, how you do you know if the tests that you put in place are truly adequate.  What I have been thinking of is building a Perl program that will create let’s say 10 variants of a program that each contain a random “mutation” of the original code.  For example, let’s say that it is a math program being developed and tested, at random points in the code a mathematical operator (e.g. +) can be changed to another random operator (e.g. -, *, %, etc). This process would be repeated 10X to create 10 variants of the code that each has one random operator changed. 

Each of these variants could then be run against the testing set.  If the variants result in testing failures in all cases, the tests probably provide sufficient coverage of the codes functionality.  If one or more of the variants pass all tests, it is an indication that the tests are not robust enough to adequately assess the proper functionality of the application. I’ll try to post some proof of concept code soon.  Thoughts?