Mail-Followup-To: LATEX-L@LISTSERV.UNI-HEIDELBERG.DE
References: <CANQYN6xiwms0LdCgW-6AQ9Pok9qbtc6k0ZgN40z6RK9+a=RdWg@mail.gmail.com> <4E945D77.6090309@morningstar2.co.uk>
            <20111011153219.GA3677@oberdiek.my-fqdn.de>
            <CANQYN6zkcHWNBUfKK_+nDc5r4hcBa7pHyUE0_1mKu+sHQbWC8A@mail.gmail.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Message-ID:  <20111012093958.GA9734@oberdiek.my-fqdn.de>
Date:         Wed, 12 Oct 2011 11:39:58 +0200
Reply-To: Mailing list for the LaTeX3 project
              <LATEX-L@listserv.uni-heidelberg.de>
Sender: Mailing list for the LaTeX3 project <LATEX-L@listserv.uni-heidelberg.de>
From: Heiko Oberdiek <heiko.oberdiek@GOOGLEMAIL.COM>
Subject: Re: Strings, and regular expressions
To: LATEX-L@listserv.uni-heidelberg.de
In-Reply-To:  <CANQYN6zkcHWNBUfKK_+nDc5r4hcBa7pHyUE0_1mKu+sHQbWC8A@mail.gmail.com>
Precedence: list
Status: R

On Tue, Oct 11, 2011 at 11:07:13PM -0400, Bruno Le Floch wrote:

> > hyperref already reencodes bookmark strings with setting
> > pdfencoding=auto. The bookmark string is construced in
> > Unicode encoding. Then the reencoding to PDFDocEncoding is tried.
> > If successful the result string is used, otherwise the Unicode string.
> > For the reencoding stuff package stringenc is used and don't need
> > to be expandable for hyperref.
> 
> Thank you Heiko. The stringenc package provides _many_ different
> encodings. Can you point me to which are useful for pdf purposes? 

Most important for PDF strings:

* PDFDocEncoding
* UTF-16

(hyperref also uses "ascii-print" in case of XeTeX because of
encoding problems with \special.)

> I guess that most "iso-..." and "cp..." encodings are an overkill for
> a kernel.

They should be loadable as files similar to LaTeX's .def files
for inputenc or fontenc. Then the kernel can provide a base set
and others can be provided by other projects. But I don't see
the disadvantage if such a base set is not minimal.

Then, when strings are written to PS/PDF, they need further
escaping:
* String escaping, provided by \pdfescapestring.
* Name escaping, provided by \pdfescapename.
* Hex strings, provided by \pdfescapehex.
The latter is also useful for other contexts, e.g. for protecting
arbitrary string data in auxiliary files. As hex string special
characters like '{', '}', '\', '#', ... do not harm.
  These pdfTeX features are provided for LuaTeX in package `pdftexcmds'
and package `pdfescape' provides the features for other engines.

> Also, when you say "Unicode encoding", I presume that this means
> native strings for XeTeX and LuaTeX, but what about pdfTeX? Do you use
> "UTF-16" (if so, LE or BE?), or some other UTF?

In the context of bookmarks and other PDF strings "Unicode"
means UTF-16 (hyperref uses BE, but there is a byte order mark).
And the strings are a sequence of bytes. The big chars of XeTeX or
LuaTeX don't help, because they get written as UTF-8.

Yours sincerely
  Heiko Oberdiek