MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
References: <AFA791BD-820C-4762-AF99-680B5A3DFA07@gmail.com>
            <4936E30A.5080209@morningstar2.co.uk>
            <87ljuxlybp.fsf@fawkes.hogwarts>
            <49370743.7050004@morningstar2.co.uk>
Message-ID:  <18745.37877.845526.230207@morse.mittelbach-online.de>
Date:         Fri, 5 Dec 2008 21:49:57 +0100
Reply-To: Mailing list for the LaTeX3 project
              <LATEX-L@LISTSERV.UNI-HEIDELBERG.DE>
Sender: Mailing list for the LaTeX3 project <LATEX-L@LISTSERV.UNI-HEIDELBERG.DE>
From: Frank Mittelbach <frank@MITTELBACH-ONLINE.DE>
Subject: Re: expl3 "token list" terminology
To: LATEX-L@LISTSERV.UNI-HEIDELBERG.DE
In-Reply-To:  <49370743.7050004@morningstar2.co.uk>
Precedence: list
Status: R

Joseph Wright writes:
 > Andreas Matthias wrote:
 > 
 > > I disagree. I don't like it at all. Characters and tokens are
 > > distinct things in terms of TeX.
 > > 
 > > LaTeX3 is already far away from TeX. Maybe too far. Don't push
 > > it even farther away by reinventing terms and definitions.
 > > 
 > > "Token list pointer" might not be the most appropriate name but,
 > > please, don't call it characters.
 > 
 > I think the point here was that a toks is a list of tokens, and so is a
 > tlp, but they are not the same.  I think we are all happy with _toks for
 > what TeX calls a toks.  So the question is how to name/describe a macro
 > used to store tokens (which is currently a tlp).  I'd have called it a
 > string, but I can see that this causes the same problem as calling the
 > contents characters.  How about a "token macro" (= _tmac)? (I see that
 > l3toks describes a toks as a "token register".)  Perhaps this points up
 > the underlying structure better?

I'm coming somewhat late to this discussion, but ...

TeX (and as such expl3) deals with tokens and the replacement text of a macro
without arguments is a list of tokens and not a list of characters or a string
or...

the contents of a toks register too is a list of tokens

they are not the same in the way they can be accessed or manipulated but their
content is the same. So I strongly disagree with Will's initial statement

 >   The problem that I'm trying to solve is what to call the contents of  
 >   these two distinct beasts. 

(but i guess in a later mail he disagreed with himself too :-) 

instead we have:

 - tlists = lists of tokens

   and a bunch of functions to manipulate them directly in the input stream
   (they got added fearly recently and I think they need to be looked at once
   more)

 - tlp = a storage bin that holds tlists

   and a bunch of functions that operate on bin and or its content
 
 - toks = a register that also holds tlists

   but allows a number of special processing (provided by the underlying TeX
   engine)

-------------

leaving out the name question for a minute ...

a) is the concept of the tlp bins useful (separating them from
macros/functions)?

b) do we need both tlp and toks?

I think the answer to both questions is yes. We do need the additional
functions provided by toks registers only but we also need the power of simple
tlp bins all over the place and calling them a bin for tlists rather than
using macros / functions without argument for this purpose (even if internally
this is what they end up being) is useful.

-------------

Coming back to the names:

 - I personally don't like the idea of token macro (i would rather get away
   from using "macro" in this context). On the other hand it isn't really
   wrong either.

 - token list pointer may not be the best but it is "fairly accurate" and it
   is short :-) and I started to like them

 - of course one could go _tlreg and _tlmac or _treg and _tmag instead of
   _toks and _tlp but I don't think this would gain anything. In fact I think
   it is not going to help.

 - _toks are really only there to provide the functionality that TeX offers
   for toks registers so keeping a close link to that isn't A Bad Thing

 - with tlp's which are conceptually a concept only within expl3 one could
   choose a different name, but I don't really see the benefit

---------------

so in short I personally think the names are nice enough, what would be need
to work out better in documentation is that both _toks and _tlp hold tlists as
their content but offer different manipulation possibilities and since you
can't easily provide the "toks" ones within the tlp family the extra type is
kept.  Perhaps it should be pointed out that one should restrict the use of
_toks to internal situations where this processing is needed


frank