Mail-Followup-To: LATEX-L@LISTSERV.UNI-HEIDELBERG.DE
References: <CANQYN6xiwms0LdCgW-6AQ9Pok9qbtc6k0ZgN40z6RK9+a=RdWg@mail.gmail.com> <4E945D77.6090309@morningstar2.co.uk>
            <20111011153219.GA3677@oberdiek.my-fqdn.de>
            <CANQYN6zkcHWNBUfKK_+nDc5r4hcBa7pHyUE0_1mKu+sHQbWC8A@mail.gmail.com>
            <20111012093958.GA9734@oberdiek.my-fqdn.de>
            <CANQYN6xfTa34wrshLWHmTOVg5U2ATzzK1ML=_qHZw=mGRRdxrw@mail.gmail.com>
            <20111012180101.GA17461@oberdiek.my-fqdn.de>
            <CANQYN6wGugJKNBxKfiWBTuir7--GYEtWN1MKDe7PfwvJ5u6tug@mail.gmail.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Message-ID:  <20111013105404.GA32369@oberdiek.my-fqdn.de>
Date:         Thu, 13 Oct 2011 12:54:04 +0200
Reply-To: Mailing list for the LaTeX3 project
              <LATEX-L@listserv.uni-heidelberg.de>
Sender: Mailing list for the LaTeX3 project <LATEX-L@listserv.uni-heidelberg.de>
From: Heiko Oberdiek <heiko.oberdiek@GOOGLEMAIL.COM>
Subject: Re: Strings, and regular expressions
To: LATEX-L@listserv.uni-heidelberg.de
In-Reply-To:  <CANQYN6wGugJKNBxKfiWBTuir7--GYEtWN1MKDe7PfwvJ5u6tug@mail.gmail.com>
Precedence: list
Status: R

On Thu, Oct 13, 2011 at 05:56:14AM -0400, Bruno Le Floch wrote:

> > I wouldn't do it manually. There are mappings files for Unicode:
> >   http://unicode.org/Public/MAPPINGS/
> > In project I am using these mappings together with a perl script
> > to generate the .def files.
> 
> Thank you for the link. It seems that the simplest would be to
> directly use the tables provided there as the .def files. Simply
> \catcode`\#=14, and set a few other default catcodes, then input the
> file, looping over the lines. Are all of the lines of the form
> 
> 0xHH    0xHHHH    # comment

Not all, dec-mcs.txt is different:

  sprintf('=%02X     U+%04X  %s\n', <code>, <unicode>, <text>)
  no comments

> (or comment lines), with H = some hexadecimal digit? In other words,
> are all those encodings 8-bit only, and with only Unicode points
> <65536?

In the directory MAPPINGS there are encodings with > 8-bit.
And a quick look doesn't reveal Unicode points > U+FFFF.

> > It could be made even expandable in linear time
> > with a large lookup table (256).
> 
> Right. I was thinking in terms of UTF-8 for some reason, and the
> lookup table would be too big.

In practice the table would be larger than 256 (16x16) to support
lowercase and uppercase digits ([0-9a-fA-F]).
The size would be 484 = (10 + 2 x 6) x (10 + 2 x 6).

Yours sincerely
  Heiko Oberdiek