References: <57029370.4090909@free.fr> <57029C43.8090605@latex-project.org>
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:38.0) Gecko/20100101
            Thunderbird/38.7.0
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
Message-ID:  <57037A5C.5040000@morningstar2.co.uk>
Date:         Tue, 5 Apr 2016 09:42:04 +0100
Reply-To: Mailing list for the LaTeX3 project
              <LATEX-L@LISTSERV.UNI-HEIDELBERG.DE>
Sender: Mailing list for the LaTeX3 project <LATEX-L@LISTSERV.UNI-HEIDELBERG.DE>
From: Joseph Wright <joseph.wright@MORNINGSTAR2.CO.UK>
Subject: Re: Bug in \NewDocumentEnvironment
To: LATEX-L@LISTSERV.UNI-HEIDELBERG.DE
In-Reply-To:  <57029C43.8090605@latex-project.org>
Precedence: list
Envelope-To: <rainer.schoepf@GMX.NET>
Status: R

On 04/04/2016 17:54, Frank Mittelbach wrote:
> guess this is a case of \protected being harmful
> 
> Am 04.04.2016 um 18:16 schrieb Julien RIVAUD (_FrnchFrgg_):
> 
>> \NewDocumentEnvironment{foo}{m}{%
>>      \begin{tabular}{#1}%
>> }{%
>>      \end{tabular}%
>> }
>>
>> \begin{document}
>>
>> \fbox{
>>      \begin{foo}{ccc}
>>          \hline
>>          Test & a & b \\
>>          \hline
>>      \end{foo}
>> }
>>
>> \fbox{
>>      \begin{tabular}{ccc}
>>          \hline
>>          Test & a & b \\
>>          \hline
>>      \end{tabular}
>> }
>>
>> \end{document}
>>
>> The first and second \fbox-es should produce the same output, but don't:
>> the foo environment starts a second line at closing time, before the
>> control is passed to the author-supplied closing code.
> 
> actually that's not equiv, equiv would be something like defining "baz"
> 
> \newenvironment{baz}[1]{%
>     \begin{tabular}{#1}%
> }{%
>     \end{tabular}%
> }
> 
> but yes, same difference
> 
> reason is the following difference in processing:
> 
> \show\endfoo
> \show\endbaz
> 
>> \endfoo=\protected macro:
> ->\environment foo end aux .
> l.10 \show\endfoo
> 
> ?
>> \endbaz=\long macro:
> ->\end {tabular}.
> l.17 \show\endbaz
> 
> the  \protected hides the \crcr inside \endtabular.
> 
> Now it may be that the protected is needed for other reasons (hope not)
> but perhaps it is just overprotection .. let Joseph say :-)

This is a tricky area and in part at least reflects the fact that xparse
(and expl3) is about both 'useful things now' and 'exploring ideas for
building a new format'. I hope that broadly xparse is useful 'now' so we
are probably doing something right.

As observed, the issue here is that as set up in LaTeX2e, the tabular
environment relies on the \halign primitive and the rather tricky
expansion behaviour that relies on. At least from e-TeX 2.0 the
\protected mechanism prevents expansion at the start of a cell so we
manage to 'hide' the \crcr. (I'll come back to this below.)

For xparse as a package 'on top' of LaTeX2e probably the easiest
solution is simply to drop the \protected status for end-of-environment
code. As Frank has (basically) observed, I'm a fan of having a clear
separate between protected and expandable code but here at least for the
present some pragmatism may be best. We could of course add some new
xparse function instead, but environments are not really expandable so
\DeclareExpandableDocumentEnvironment would be wrong and in any case the
internal are not readily accessible so it's likely not required/sensible.

Looked at in the context of 'for the future format' I'm not sure at
present what might be best. As \halign is rather tricky it's not at all
clear that a new tabular implementation should use it at all. However,
the bigger issue there would be not the internals of environments but
\end itself. As I've noted, environments aren't expandable: they form
groups and need to do assignments. Overall I think having these macros
\protected is a 'good thing' so would be unkeen on ending up with a 'not
really expandable' \end (or equivalent: nothing is actually decided in
terms of the interfaces in a new format). It would be possible to
arrange that for example \\ or \hline (or equivalent) do f-type
expansion of following material by ending them with \romannumeral`-0. We
already have a similar trick in l3galley to allow detection of 'ignored'
paragraphs. However, that would also expand other \protected macros
which might not be desirable (see for example siunitx S column escape).
There are possible solutions to this involving much the same stuff as in
l3galley (inserting a marker \relax of the correct name where expansion
is to stop, and skipping it where a \crcr/\omit/\noalign is to be
found). (I also suspect I would take a different tack with parts of the
siunitx table interface if/when writing for a new format, so that may
not be an issue, particular if the tabular implementation is richer in
the first place.)

I wonder about the fact that e-TeX v1 -> v2 included adding \halign
cells to places where \protected is 'active': presumably there was a
good reason for this though it may (since Peter B.'s death) be hard to
pin this down.

Thoughts?

Joseph