Content-Type: text/plain; format=flowed; delsp=yes; charset=iso-8859-1
MIME-Version: 1.0
References: <495160FA.8000004@morningstar2.co.uk>
            <9EECA5EE-12A7-4C01-8311-7658BD3E8E04@gmail.com>
            <op.umx28toejli536@macintosh.local>
            <21363E65-E3FB-4495-A94E-6789AC0619A0@gmail.com>
            <18778.35736.43421.950797@morse.mittelbach-online.de>
Content-Transfer-Encoding: 7bit
User-Agent: Opera Mail/9.63 (MacPPC)
Message-ID:  <op.umz0d0pnjli536@macintosh.local>
Date:         Wed, 31 Dec 2008 00:19:02 +0100
Reply-To: Mailing list for the LaTeX3 project
              <LATEX-L@LISTSERV.UNI-HEIDELBERG.DE>
Sender: Mailing list for the LaTeX3 project <LATEX-L@LISTSERV.UNI-HEIDELBERG.DE>
From: =?iso-8859-1?Q?Morten_H=F8gholm?= <morten.hoegholm@GMAIL.COM>
Subject: Re: Back to "token list" nomenclature; was Re: \tlist_if_eq:nn
To: LATEX-L@LISTSERV.UNI-HEIDELBERG.DE
In-Reply-To:  <18778.35736.43421.950797@morse.mittelbach-online.de>
Precedence: list
Status: R

On Tue, 30 Dec 2008 21:59:04 +0100, Frank Mittelbach wrote:

>  c) the other important difference is how you get something out of the  
> bin:
>    with tlps you can simply use the pointer and that is really the normal
>    usage (the \tlp_use:N is really syntactic sugar) while with toks you
>    absolutely need the accessor in every case.

Generally, I like to have accessor functions.

> So my proposal for tlps is that we document they support balanced token  
> lists
> not containing #. we can add a footnote saying that technically that is  
> only
> an approimation of the truth and that the reality is ... but that we  
> recommend
> that this feature is not to be used or only in places where the author  
> is in
> full control of the processing and the input.

This would be fine by me.

> A followup question then is
>
>  d) what exactly do we want to support in the input stream, ie on the  
> level of
>     the functions currently called \tlist...
>
> Morten said he wrote those with the idea in mind that they should accept
> arbitrary balanced token lists similar to toks. If we can do this  
> reliably
> (and want to) then, yes, we should probably call them \toks_...

It's been a while since tlist was introduced but I do recall a little  
about the motivation behind them and the naming consequence of clist.

It all started with a desire to do "proper" mapping functions. For clist  
one of the problems of \@for is that it expands its comma list once which  
is fine for some things but less so for others. Therefore, the clist  
module got both a N variant for those stored in bins and an n variant for  
those straight from the input stream.

A similar need arose for tlps. However, it didn't really make sense to me  
to have a token list pointer function have an n type argument - then it  
wouldn't be a pointer. Hence, tlist was conceived which was the best I  
could come up with at the time. These quickly proved to have a life of  
their own, testing input streams for being empty, blank, turning them into  
harmless character tokens[1], providing new names for the \uppercase and  
\lowercase primitives, getting head and tail of a list, etc.

> Personally I'm not sure we can do that reliably in the end (assuming more
> functions pop up where we want to build an inline version operating on  
> the
> input stream) and I don't really think it is necessary.

As long as there is no attempt to actually store a part of such a tlist  
argument in a macro, then there is no problem. Is there? Much like my  
argument for making the kernel functions almost exclusively \long, knowing  
what gets into the low-level functions is hard to predict. That's part of  
the reason why seq and prop were transformed into toks registers some time  
ago.


[1] Another type which we have not really touched much: the "str" defined  
as tokens with catcode 12 (and space catcode 10) as output by several  
primitives.

-- 
Morten