User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:38.0) Gecko/20100101
            Thunderbird/38.2.0
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
Message-ID:  <55FA92C9.7020604@morningstar2.co.uk>
Date:         Thu, 17 Sep 2015 11:15:37 +0100
Reply-To: Mailing list for the LaTeX3 project
              <LATEX-L@LISTSERV.UNI-HEIDELBERG.DE>
Sender: Mailing list for the LaTeX3 project <LATEX-L@LISTSERV.UNI-HEIDELBERG.DE>
From: Joseph Wright <joseph.wright@MORNINGSTAR2.CO.UK>
Subject: \tl_to_(lower|upper)case:n - Generating char tokens
To: LATEX-L@LISTSERV.UNI-HEIDELBERG.DE
Precedence: list
Envelope-To: <rainer.schoepf@GMX.NET>
Status: R

Hello all,

Some of you may have noticed that the team have not been happy for a
while with \tl_to_(lower|upper)case:n. There are a few issues:

 - The names are wrong as "_to_" is in all other cases used
   for expandable 1-to-1 type conversions
 - Using the \lowercase/\uppercase primitives means
   these functions fail for Unicode case mapping
 - Using a case changing function for generating
   odd catcodes is not ideal

We've been working on alternative interfaces that allow the required
outcomes, and now are just about there. Over the past few weeks, we've added

 - \char_(g)set_active_eq:NN
 - \char_(g)set_active_eq:nN

for active characters and

 - \char_generate:nn

for creating arbitrary character tokens. The latter covers (almost) all
sensible use cases: catcodes 1, 2, 3, 4, 6, 7, 8, 11 and 12. Currently
catcode 10 and 13 are not available, but we expect to address at least
the catcode 10 case shortly.

We have done a sweep over current uses in TeX Live of
\tl_to_(lower|upper)case:n and \tex_(lower|upper)case:D and believe we
now can deprecate use of \tl_to_(lower|upper)case:n in favour of the
above (for odd chars), using x-type expansion and \tl_to_str:n (for
catcode 12 chars) or the expandable case changing functions. I've
contacted various package authors to ask for adjustments in this regard
and will shortly contact the remaining people. We'll also adjust the
expl3 internals themselves to use the higher-level functions as far as
possible (internally we do need the primitives to set up the interfaces).

There is one proviso in the above. On XeTeX, \char_generate:nn uses
\Ucharcat if available, and is otherwise limited to the 8-bit range.
This means that at present there are a small number of use cases that
would break with older XeTeX's if modified to use \char_generate:nn. The
team will in time alter the minimum XeTeX version to reflect that but at
present the small number of people generating Unicode chars on XeTeX
will need to continue to use \tex_lowercase:D.

Assuming the above transition can be carried out smoothly, we will
formally deprecate \tl_to_(lower|upper)case:n shortly for removal after
the end of next year.

Regards,

Joseph