User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:7.0) Gecko/20110922
            Thunderbird/7.0
MIME-Version: 1.0
Content-Type: text/plain; charset=ISO-8859-15; format=flowed
Content-Transfer-Encoding: 7bit
Message-ID:  <4E82369C.20007@latex-project.org>
Date:         Tue, 27 Sep 2011 22:48:28 +0200
Reply-To: Mailing list for the LaTeX3 project
              <LATEX-L@listserv.uni-heidelberg.de>
Sender: Mailing list for the LaTeX3 project <LATEX-L@listserv.uni-heidelberg.de>
From: Frank Mittelbach <frank.mittelbach@LATEX-PROJECT.ORG>
Subject: volunteers sought: trip test for expl3 modules
To: LATEX-L@listserv.uni-heidelberg.de
Precedence: list
Status: R

One of the recent bug reports reminded me that I wanted to ask for 
volunteers to help making our regression test suite better.

As you may know for 2e we implemented a fairly extensive test suite of 
files that can be run automatically and that would identify sudden 
changes in the interfaces that may happen due to dependencies between 
functions that may not be apparent on the surface.

Over time this suite has helped to keep the number of serious blunders 
down to a minimum (though nevertheless we managed to make them).

It also helped very much in the transition from 2.09 to 2e as we 
"certified" some results initially on 2.09 and could then see if 2e 
would still give the same result.

The basic idea is to write test files in a certain format and store away 
the log file (after some manipulation, i.e., removing stuff that would 
change on rerun, or is dependent on the distribution).

One important aspect is that the test should show "results" in the log, 
but it should not show a lot of implementation details if that can be 
helped, e.g. a \tracingall would not be a good output as any change in 
the inner implementation would result in a diff.

Once such a pair is inspected and the output is declared to be correct 
it is added to the suite to be automatically run from thereafter and the 
result compared with the frozen test log.

For expl3 we also started to produce such tests, but test coverage is 
far from perfect.

So what we are looking for are individuals that would interested to help 
here. It needs some devious mindset (writing good tests is certainly an 
art :-) you need to get in to a state of mind where you think about, 
what are the boundaries of the functionality, where the developer might 
have messed up, as well as seriously thinking of how to test the main 
functionality that you do not want to see changing.

For those interested to help, you would need a development version of 
expl3, either form svn or from the github, where you can find testfiles 
for example, e.g.:

https://github.com/latex3/svn-mirror/tree/master/l3kernel/testfiles

As I said help here would be really useful to stabilize the code 
further. For example l3fp is probably being reimplemented fairly soon. 
If so having a comprehensive test set would ensure that the 
reimplementation is not suddenly changing the interfaces or contains 
mistakes.

So if you are interested talk to us so that we can help you to get 
comfortable with it

thanks in advance
frank