References: <4B727378.8060704@morningstar2.co.uk>
            <4B729944.5050308@residenset.net>
            <OFD3BDE8BF.D25CA425-ON802576C6.0043FFAD-802576C6.0043FFAE@mcs-notes1.open.ac.uk>
            <4B72B36E.6010401@morningstar2.co.uk>
            <OF7A9BB41F.2FC253BE-ON802576C6.005B33FC-802576C6.005B340D@mcs-notes1.open.ac.uk>
            <4B730157.5060605@morningstar2.co.uk>
User-Agent: Gnus/5.1006 (Gnus v5.10.6) Emacs/21.4 (usg-unix-v)
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Message-ID:  <i7zl3gu98t.fsf@hilbert.math.albany.edu>
Date:         Wed, 10 Feb 2010 15:27:30 -0500
Reply-To: Mailing list for the LaTeX3 project
              <LATEX-L@listserv.uni-heidelberg.de>
Sender: Mailing list for the LaTeX3 project <LATEX-L@listserv.uni-heidelberg.de>
From: William F Hammond <hammond@CSC.ALBANY.EDU>
Subject: Re: LaTeX3 8-bit only?
To: LATEX-L@listserv.uni-heidelberg.de
In-Reply-To:  <4B730157.5060605@morningstar2.co.uk> (Joseph Wright's message of
              "Wed, 10 Feb 2010 18:56:23 +0000")
Precedence: list
Status: R

Joseph Wright <joseph.wright@MORNINGSTAR2.CO.UK> writes:

> . . .
> I was thinking of input encodings, where my point was (supposed to) be
> that something like the inputenc "utf8" approach would be an approach
> I hope we can avoid as there are better solutions (in the form of
> engines which deal with the issue). (Of course, that leaves UTF-16
> issues, but I'd hope that engine developments can help out).
>
> (I'd point out that LaTeX3 code is intended for use in new documents,
> and the rest of the computer world is standardising on UTF-8 as far as
> I can see. So I'd hope very much that having an approach based on this
> concept is not too risky.)

At some point I expect that LaTeX will want to provide for arbitrary
unicode "word" characters in command names.  If that is the case, then
shouldn't standard handling of text-encodings for whole document
instances apply?  In particular, I think that text-encoding
pre-processing (by something like GNU "recode") to meet the needs of
the particular TeX engine would be the way to proceed.

Presumably the UTF-8 and UTF-16 text encodings for unicode are both
supported everywhere in the XML world.  While, UTF-8 is more efficient
for Western languages, UTF-16 is likely to be favored in regions where
the unicode sections devoted to local character sets take more than 4
bytes per character.

I also imagine that eventually many documents will have origin under
author-level XML document types, and so what is fed to a TeX engine
would then be output from a pipeline.  In that case I suppose UTF-8
would be a reasonable standard.  Let me also point out that, to the
extent that XML origination is realized, TeX engines might never need
to bite the bullet on non-ascii command names.  That is, LaTeX
provision of non-ascii command names could be handled via XML front
ends that are sponsored by the LaTeX Project.

                                    -- Bill