Message-ID: <559534D0.2060001@morningstar2.co.uk>
Date: Thu, 02 Jul 2015 13:55:44 +0100
From: Joseph Wright <joseph.wright@morningstar2.co.uk>
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64;
 rv:31.0) Gecko/20100101 Thunderbird/31.7.0
MIME-Version: 1.0
To: heiko.oberdiek@googlemail.com, mpg@elzevir.fr,
        Philipp Maximilian Stephani <p.stephani2@gmail.com>,
        elie.roux@telecom-bretagne.eu
Subject: LuaTeX support in the LaTeX kernel
Content-Type: multipart/mixed; boundary="------------020201020305050602090205"
Cc: "latex-team@latex-project.org" <latex-team@latex-project.org>,
        Lars Madsen <daleif@imf.au.dk>
Precedence: list
Errors-To: latex-team-bounces@latex-project.org
Sender: "Latex-team" <latex-team-bounces@latex-project.org>
Status: R

This is a multi-part message in MIME format.
--------------020201020305050602090205
Content-Type: text/plain; charset=windows-1252
Content-Transfer-Encoding: 7bit

[Resending without the .ins files: hopefully those who don't get them
can sort out their own versions!]

Hello all,

Long e-mail coming up. Executive summary: the LaTeX team are looking for
some input into aspects of support for LuaTeX in the LaTeX kernel.

As you'll be aware, the LaTeX team have recently taken steps to add core
support for modern TeX to the LaTeX2e kernel, replacing a rather awkward
arrangement where material was added by the format-building process.
Most importantly here, that means that the kernel now provides
allocators for \marks and \XeTeXintercharclass registers, and enables
the additional LuaTeX primitives 'out of the box'.

For LuaTeX there are other new register types that need allocation but
that has not to date been covered at all by any pre-built formats. As
such, support for this area has been left to package authors, with both
the "luatex" and "luatexbase" packages available. This situation is not
ideal as it means that something which can really only be done once
(setting up an allocation scheme) is potentially done multiple times
with conflicting implementations. Moreover, it means that any code
targeting LuaTeX has to pull in quite a bit of material, which has
implications for work within the team on for example proper inputenc
support for LuaTeX.

As the TL'15 changes to the kernel went (relatively) smoothly, we are
now hoping to address the LuaTeX-specific side of things in a similar
way. What we are hoping to do is to extend the kernel to cover the
*essentials* in this area while leaving more 'optional' stuff to
packages. The latter may be some modified version of what exists already
or may (perhaps more likely) be a new package provided by the team or
one or more members (David and me, most probably). (The latter might
include providing a version of the LaTeX kernel code for plain TeX such
that a new package could cover both cases.)

What we are looking for at this stage is feedback on our minimisation of
the features provided. Much of what we have is more-or-less direct
copies of the luatexbase material, but there are modifications. Below
I'll list what we have included and any notable changes or omissions.

Attached is a proposed addition to the kernel (ltluatex) along with
various tests/demos. The demos are potentially the basis for the package
level support I mention.

*Allocators in general*

As this code is kernel-level it can make assumptions packages cannot. In
particular, from TL'15 we've reserved a set of \count registers in the
extended pool for team use. The allocators here therefore use hard-coded
positions, as is done for the TeX90 allocation routine.

We have provided the basic allocators but not \set... commands: these
seem to be comfortably handled in package code.

*Attribute allocation*

This seems straight forward: note that we've done it from the TeX end.
David has written luaregisternames to demonstrate that Lua access to the
allocated number is straight-forward (and this could be covered by a
package).

*Catcode allocation*

We have just covered the core idea of allocation. In contrast to
luatex/luatexbase we have gone for a simple allocator as for other
registers. We think that a stack mechanism belongs at the package level
(catcode tables seem more useful from Lua than from TeX in any case).
David has sketched out a pool-based approach to a stack in the attached
catcodestack document.

We have not at this stage pre-defined any tables. We suspect a core set
(IniTeX/string/document) may well be useful: thoughts welcome.

*Lua functions*

These are not currently covered by package code but seem trivial to
handle and so are included.

*Callbacks*

We have significantly minimised the callback allocator from luatexbase
to cover just the idea of registering multiple callbacks but not
resetting/removing/... We are also not currently convinced by the
'priority' idea: it seems to be only partly solving a problem.

We have kept a public interface to the built-in callback.register
function with the idea being that a more sophisticated package interface
could be introduced and replace the kernel version.

*Module management*

We have not added this: arguments for inclusion at the kernel level are
welcome.

----

Whilst not linked to adding the above, it is also worth raising here
that we are considering the position on LuaTeX primitive naming. The
current approach adds \luatex... to all names, but this leads to some
rather awkward naming (\luatexluafunction, \luatexluaescapestring, ...)
and means that anyone reading the LuaTeX manual has to know that the
'base' names do not apply. The situation with \Umath... (used to be
\XeTeX...) and the the \pdf... primitives not related to PDF output also
suggests adding the engine name here is not ideal.

We are therefore considering switching the kernel approach to use the
'raw' names, retaining the prefixed ones at least at present when using
the latexrelease mechanism. Obviously this would have an impact on
package authors but the number directly using LuaTeX primitives is
small. Moreover, we will adjust expl3 in the next release to provide all
XeTeX and LuaTeX primitives in the \<engine>_<name>:D namespace. Whilst
that is normally meant for 'team use only' we recognise that the 'new'
primitives are not covered yet by any team code so this is a reasonable
compromise.  As most of the code for font support uses expl3 anyway,
this should ease a transition.

We know of one package that uses a name currently set up by LuaTeX
(breqn: \mathstyle) but this is fixable: team members look after that
code and the use sense is the same so change required is small.

This is as I say a separate area but does impact on the allocators at
least in as far as the naming of the new commands.

----

The key aim here to address long-term maintainablity whilst aiming to
keep things stable and usable for end-users. Thus it's important to get
agreement on what is and is not appropriate. Feedback/thoughts/... on
any or all of the above most welcome.

Joseph


--------------020201020305050602090205
Content-Type: text/x-tex;
 name="luaregisternames.dtx"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
 filename="luaregisternames.dtx"

% \iffalse
%% Source File: textcase.dtx
%% Copyright 2015 David Carlisle
%%
%% This file may be distributed under the terms of the LPPL.
%% See README for details.
%
%<*dtx>
          \ProvidesFile{luaregisternames.dtx}
%</dtx>
%<package>\NeedsTeXFormat{LaTeX2e}
%<package>\ProvidesPackage{luaregisternames}
%<driver> \ProvidesFile{luaregisternames.drv}
% \fi
%         \ProvidesFile{luaregisternames.dtx}
       [2015/06/24 v0.01 luaregisternames for luaLaTeX  (DPC)]
%
% \iffalse
%<*driver>
\documentclass{ltxdoc}
\ifx\directlua\undefined\else
\usepackage{luaregisternames}
\fi
\begin{document}
\DocInput{luaregisternames.dtx}
\end{document}
%</driver>
% \fi
%
% \GetFileInfo{luaregisternames.dtx}
%
% \title{The \textsf{luaregisternames} Package\thanks{This file
%        has version number \fileversion, last
%        revised \filedate.
% Please report any issues at https://github.com/davidcarlisle/dpctex/issues}}
% \author{David Carlisle}
% \maketitle
%
% \section{Introduction}
% Sometimes (notably in the case of lua attributes) it is necessary to
% access a register \emph{by number} that has been allocated by \TeX.
% This package provides a function to look up the relevant number
% using luatex's internal tables.
% 
% so after |\newluatexattribute\myattrib|
%
% If |\myattrib| has been defined by (say)
% |luatexattributedef\myattrib=15| then the lua function call
% |registernumber("myattrib")| will return 15. If the string passed as
% argument does not correspond to a token defined by
% |luatexattributedef|, |\countdef| or similar commands, the lua value
% |false| is returned.
%
%
% \section{Example}
%
%\begin{verbatim}
% \newcommand\test[1]{%
% \typeout{#1: \expandafter\meaning\csname#1\endcsname^^J
% \space\space\space\space
% \directlua{tex.write(registernumber("#1") or "bad input")}%
% }}
% 
% \test{undefinedrubbish}
% 
% \test{space}
% 
% \test{hbox}
% 
% \test{@MM}
% 
% \test{@tempdima}
% \test{@tempdimb}
% 
% \test{strutbox}
%
% \test{sixt@@n}
%
% \attrbutedef\myattr=12
% \myattr=200
% \test{myattr}
%
%\end{verbatim}
%
%\ifx\directlua\undefined\else
% \newcommand\test[1]{%
% \typeout{#1: \expandafter\meaning\csname#1\endcsname^^J
% \space\space\space\space
% \directlua{tex.write(registernumber("#1") or "bad input")}%
% }}
% 
% \test{undefinedrubbish}
% 
% \test{space}
% 
% \test{hbox}
% 
% \test{@MM}
% 
% \test{@tempdima}
% \test{@tempdimb}
% 
% \test{strutbox}
%
% \test{sixt@@n}
%
% \luatexattributedef\myattr=12
% \myattr=200
% \test{myattr}
%
%\fi
%
% If this document has been processed with lualatex then the above
% code would have produced the following in the log and terminal
% output.
%\begin{verbatim}
% undefinedrubbish: \relax
%      bad input 
% space: macro:-> 
%      bad input 
% hbox: \hbox
%      bad input 
% @MM: \mathchar"4E20
%      20000 
% @tempdima: \dimen14
%      14 
% @tempdimb: \dimen15
%      15 
% strutbox: \char"B
%      11 
% sixt@@n: \char"10
%      16 
% myattr: \attribute12
%      12 
%\end{verbatim}
%
% Notice how undefined commands, or commands unreleated to registers
% do not produce an error, just return |false| and so print
% |bad input| here. Note also that commands defined by |\newbox| work and
% return the number of the box register even though the actual command
% holding this number is a |\chardef| defined token (there is no
% |\boxdef|).
%
% \section{Implementation}
%
% Note that the current implementation uses |\directlua| and a
%
%    \begin{macrocode}
%<*package>
%    \end{macrocode}
% Stop if not lua\TeX.
%    \begin{macrocode}
\ifx\directlua\@undefined
\PackageError{luaregisternames}
{\luaregisternames may only be used with luatex}
{skipping package}
\expandafter\endinput
\fi
%    \end{macrocode}
%
%
% We need a declared register of each type, preferably 0.
% the \LaTeX\ format has predeclared registers for everything except
% lua attributes and muskip.
%    \begin{macrocode}
\luatexattributedef\reserved@a=0
\muskipdef\reserved@b=0
%    \end{macrocode}
%
% Test  the luatex version 0.80 is the texlive 2015 release.
%    \begin{macrocode}
\ifnum\luatexversion>79
%    \end{macrocode}
%
% This package only defines lua functions, no TeX code.
% (it might be better to distribute it as a lua file to be used with
% |require|. We'll see\ldots.
%    \begin{macrocode}
\directlua {
%    \end{macrocode}
%
% The |.mode| field in |newtoken| isn't currently documented but Hans
% Hagen pointed it out this possibility to me on the luatex list as a
% better alternative to the |hashtokens| below.
%
% First store the starting value of each register type in the internal
% command tables.
%    \begin{macrocode}
luaregisterbasetable={}
luaregisterbasetable["assign_glue"]=
  newtoken.create("skip@").mode
luaregisterbasetable["assign_mu_glue"]=
  newtoken.create("reserved@b").mode
luaregisterbasetable["assign_dimen"]=
  newtoken.create("dimen@").mode
luaregisterbasetable["assign_toks"]=
  newtoken.create("toks@").mode
luaregisterbasetable["assign_int"]=
  newtoken.create("c@page").mode
luaregisterbasetable["assign_attr"]=
  newtoken.create("reserved@a").mode
luaregisterbasetable["math_given"]=
  newtoken.create("@m").mode - 1000
luaregisterbasetable["char_given"]=
  newtoken.create("@inputcheck").mode
%    \end{macrocode}
%
% Now a function that determines the command type of a supplies csname
% and if it is one of the above types, return the relevant number,
% otherwise return |false|.
%    \begin{macrocode}
function registernumber (name) 
  local nt = newtoken.create(name)
  if(luaregisterbasetable[nt.cmdname]) then
    return nt.mode - luaregisterbasetable[nt.cmdname]
  else
    return false
  end
end
%    \end{macrocode}
% 
%    \begin{macrocode}
}
%    \end{macrocode}
% 
%    \begin{macrocode}
\else
%    \end{macrocode}
%
% Prior to 0.80 lua\TeX\ did not have athe |newtoken| class, however
% equivalent information could be obtained by peeking into the table
% returned by |hashtokens|.
% 
%    \begin{macrocode}
\directlua {
%    \end{macrocode}
%
%    \begin{macrocode}
luaregisterbasetable={}
luaregisterbasetable[tex.hashtokens()
  ["skip@"][1]]=tex.hashtokens()["skip@"][2]
luaregisterbasetable[tex.hashtokens()
  ["reserved@b"][1]]=tex.hashtokens()["reserved@b"][2]
luaregisterbasetable[tex.hashtokens()
  ["dimen@"][1]]=tex.hashtokens()["dimen@"][2]
luaregisterbasetable[tex.hashtokens()
  ["toks@"][1]]=tex.hashtokens()["toks@"][2]
luaregisterbasetable[tex.hashtokens()
  ["c@page"][1]]=tex.hashtokens()["c@page"][2]
luaregisterbasetable[tex.hashtokens()
  ["reserved@a"][1]]=tex.hashtokens()["reserved@a"][2]
luaregisterbasetable[tex.hashtokens()
  ["@m"][1]]=tex.hashtokens()["@m"][2] - 1000
luaregisterbasetable[tex.hashtokens()
  ["@inputcheck"][1]]=tex.hashtokens()["@inputcheck"][2]
%    \end{macrocode}
% 
%    \begin{macrocode}
function registernumber (name) 
  local nt = tex.hashtokens()[name]
  if(nt and luaregisterbasetable[nt[1]]) then
    return nt[2] - luaregisterbasetable[nt[1]]
  else
    return false
  end
end
%    \end{macrocode}
% 
%    \begin{macrocode}
}
%    \end{macrocode}
% 
%    \begin{macrocode}
\fi
%    \end{macrocode}
% 
% Just undo the two temporary definitions made at the start.
%    \begin{macrocode}
\let\reserved@a\relax
\let\reserved@b\relax
%    \end{macrocode}
% 
%
%    \begin{macrocode}
%</package>
%    \end{macrocode}
%
% \Finale
%


--------------020201020305050602090205
Content-Type: text/plain; charset=windows-1252;
 name="catcodestack.tex"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
 filename="catcodestack.tex"

\documentclass{article}


\usepackage{ltluatex,luaregisternames}
\directlua{require("latex2e.lua")}


\begin{document}

\makeatletter

% setup copied more or less from luatexbase
\def\setluatexcatcodetable#1#2{%
  \begingroup
  #2%
  \luatexsavecatcodetable#1%
  \endgroup}


\def\SetCatcodeRange#1#2#3{%
  \@tempcnta=#1\relax
  \@tempcntb=#2\relax
  \count@=#3\relax
  \loop\unless\ifnum\@tempcnta>\@tempcntb
    \catcode\@tempcnta=\count@
    \advance\@tempcnta\@ne
  \repeat}


\newluatexcatcodetable\CatcodeTableIniTeX

\newluatexcatcodetable\CatcodeTableString
\setluatexcatcodetable\CatcodeTableString{%
  \luatexcatcodetable\CatcodeTableIniTeX
  \catcode0   12 % nul
  \catcode13  12 % carriage return
  \catcode37  12 % percent
  \SetCatcodeRange{65}{90}{12}% A-Z
  \SetCatcodeRange{97}{122}{12}% a-z
  \catcode92  12 % backslash
  \catcode127 12 }

\newluatexcatcodetable\CatcodeTableOther
\setluatexcatcodetable\CatcodeTableOther{%
  \luatexcatcodetable\CatcodeTableString
  \catcode32 12 }

\newluatexcatcodetable\CatcodeTableLaTeX
\setluatexcatcodetable\CatcodeTableLaTeX{%
  \luatexcatcodetable\CatcodeTableIniTeX
  \SetCatcodeRange{0}{31}{15}%
  \catcode9   10 % tab
  \catcode10  12 % newline
  \catcode12  13 % form feed
  \catcode13  5  % carriage return
  \catcode35  6  % hash
  \catcode36  3  % dollar
  \catcode38  4  % ampersand
  \catcode94  7  % circumflex
  \catcode95  8  % underscore
  \catcode123 1  % brace left
  \catcode125 2  % brace right
  \catcode126 13 % tilde
  \catcode127 15 }

\newluatexcatcodetable\CatcodeTableLaTeXAtLetter
\setluatexcatcodetable\CatcodeTableLaTeXAtLetter{%
  \luatexcatcodetable\CatcodeTableLaTeX
  \catcode64=11 }

\newluatexcatcodetable\CatcodeTableExpl
\setluatexcatcodetable\CatcodeTableExpl{%
  \luatexcatcodetable\CatcodeTableLaTeX
  \catcode126 10 % tilde is a space char
  \catcode32  9  % space is ignored
  \catcode9   9  % tab also ignored
  \catcode95  11 % underscore letter
  \catcode58  11 % colon letter
  }


% new simple stack code

% list of free tables to use in stack (extended on demand)
\def\@catcodetablelist{}

% currently active stack of tables (perhaps should initilse as {0})
\def\@catcodetablestack{}

% take a free table (allocate if needed)
%  and save current settings in it
% push on to stack.
\def\pushcurrentcatcodes{%
\ifx\@catcodetablelist\@empty
  \edef\reserved@a{\csname ct@\the\count258\endcsname}%
  \expandafter\newluatexcatcodetable\reserved@a
  \xdef\@catcodetablelist{\reserved@a}%
\fi
\expandafter\@pushctbl\@catcodetablelist\@nil}

\def\@pushctbl#1#2\@nil{%
\gdef\@catcodetablelist{#2}%
\xdef\@catcodetablestack{#1\@catcodetablestack}%
\luatexsavecatcodetable#1}

% pop a table off the stack and use to set current
% settings
\def\popcurrentcatcodes{%
\ifx\@catcodetablestack\@empty
\ERROR
\fi
 \expandafter\@popctbl\@catcodetablestack\@nil}

\def\@popctbl#1#2\@nil{%
  \gdef\@catcodetablestack{#2}%
  \gdef\@catcodetablelist{\@catcodetablestack#1}%
  \luatexcatcodetable#1}


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

% tests...

\def\showcs#1{\typeout{#1:
@=\the\catcode`\@, 
space=\the\catcode32,
Z=\the\catcode`\Z}}

\showcs{A1}

\pushcurrentcatcodes

\catcode`\Z=13

\showcs{B1}

\pushcurrentcatcodes

\luatexcatcodetable\CatcodeTableExpl

\showcs{C1}

\catcode`\Z=4

\showcs{C2}


\popcurrentcatcodes

\showcs{B3}

\popcurrentcatcodes

\showcs{A4}


\end{document}

--------------020201020305050602090205
Content-Type: text/plain; charset=windows-1252;
 name="regalloctest.tex"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
 filename="regalloctest.tex"

\documentclass{article}


\usepackage{ltluatex,luaregisternames}
\directlua{require("latex2e.lua")}

\newluatexcatcodetable\ctbl
\newluatexattribute\lattr

\begin{document}

\directlua{print("\string\nctbl" .. " is register:  " 
.. registernumber("ctbl") )}
\directlua{print("\string\nlattr" .. " is register:"
.. registernumber("lattr") )}

\newbox\mybox
{
\setbox2\hbox{xyz}
\lattr=5
\setbox0\hbox{abc}
\lattr=6
\setbox\mybox\hbox{123}

\directlua{
local n0=tex.getbox(0)
local n2=tex.getbox(2)
local nmybox=tex.getbox(registernumber("mybox"))
local a = registernumber("lattr")
print("\string\n test lattr on box 0: " 
.. (node.has_attribute(n0,a) or "unset"))
print("\string\n test lattr on box 2: "
 .. (node.has_attribute(n2,a) or "unset"))
print("\string\n test lattr on box mybox: " 
.. (node.has_attribute(nmybox,a) or "unset"))
}
}

\directlua{
function zzzX (s)
return string.gsub(s,"X", "[Capital x was here]")
end
function zzzC (s)
return string.gsub(s,"C", "[Capital c was here]")
end
latex.add_to_callback('process_input_buffer',zzzX,"testing X")
latex.add_to_callback('process_input_buffer',zzzC,"testing C")
}

\typeout{123XYZ456ABC}


\end{document}

--------------020201020305050602090205
Content-Type: text/x-tex;
 name="ltluatex.dtx"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
 filename="ltluatex.dtx"

% \iffalse meta-comment
%
% Copyright 2015
% The LaTeX3 Project and any individual authors listed elsewhere
% in this file.
%
% This file is part of the LaTeX base system.
% -------------------------------------------
%
% It may be distributed and/or modified under the
% conditions of the LaTeX Project Public License, either version 1.3c
% of this license or (at your option) any later version.
% The latest version of this license is in
%    http://www.latex-project.org/lppl.txt
% and version 1.3c or later is part of all distributions of LaTeX
% version 2005/12/01 or later.
%
% This file has the LPPL maintenance status "maintained".
%
% The list of all files belonging to the LaTeX base distribution is
% given in the file `manifest.txt'. See also `legal.txt' for additional
% information.
%
% The list of derived (unpacked) files belonging to the distribution
% and covered by LPPL is defined by the unpacking scripts (with
% extension .ins) which are part of the distribution.
%
% \fi
% \iffalse
%
%<*driver>
% \fi
\ProvidesFile{ltluatex.dtx}
  [2015/07/02 v1.0a LaTeX Kernel (LuaTeX support)]
% \iffalse
\documentclass{ltxdoc}
\AtBeginDocument{%
  \begingroup\lccode`~=`_%
  \lowercase{\endgroup\let~}_%
  \catcode`_=12
}
\GetFileInfo{ltluatex.dtx}
\begin{document}
\title{\filename\\(Lua\TeX{}-specific support)}
\author{David Carlisle and Joseph Wright\footnote{Significant portions
  of the code here are adapted/simplified from the packages \textsf{luatex} and
  \textsf{luatexbase} written by Heiko Oberdiek, \'{E}lie Roux,
  Manuel P\'{e}gouri\'{e}-Gonnar and Philipp Gesang.}}
\date{\filedate}
\MaintainedByLaTeXTeam{latex}
\maketitle
\DocInput{\filename}
\end{document}
%</driver>
% \fi
% 
% \CheckSum{0}
%
%    \begin{macrocode}
%<*2ekernel>
%    \end{macrocode}
%
% Lua\TeX{} adds a number of engine-specific functions to \TeX{}. Several of
% these require set up that is best done in the kernel or need related support
% functions. This file provides \emph{basic} support for Lua\TeX{} at the
% \LaTeXe{} kernel level.
% 
% This file contains code for both \TeX{} (to be stored as part of the format)
% and Lua (to be loaded at the start of each job). In the Lua code, the kernel
% uses the namespace |latex| which is \emph{reserved for use by the team}.
% 
% The following |\count| registers are used here for register allocation:
% \begin{itemize}
%  \item[\texttt{258}] Attributes
%  \item[\texttt{259}] Category code tables
%  \item[\texttt{260}] Lua functions
% \end{itemize}
% (|\count256| is used for |\newmarks| allocation and |\count257|
% is used for |\newXeTeXintercharclass| with Xe\TeX{}, with code defined in
% \texttt{ltfinal.dtx}).
% With any \LaTeXe{} kernel from 2015 onward these registers are part of
% the block in the extended area reserved by the kernel (prior to 2015 the
% \LaTeXe{} kernel did not provide any functionality for the extended
% allocation area).
% 
% \DescribeMacro{\newattribute}
% |\newluatexattribute{|\meta{attribute}|}|
% Defines a named \cs{attribute}, indexed from~$1$
% (\emph{i.e.}~|\luatexattribue0| is never defined). The attribute will be
% initialised with the marker value |-"7FFFFFFF| (`unset').
% 
% \DescribeMacro{\newcatcodetable}
% |\newluatexcatcodetable{|\meta{catcodetable}|}|
% Defines a named \cs{catcodetable}, indexed from~$1$
% (|\luatexcatcodetable0| never assigned). A new catcode table will be
% populated with exactly those values assigned by Ini\TeX{} (as described
% in the Lua\TeX{} manual).
% 
% \DescribeMacro{\newluafunction}
% |\newluafunction{|\meta{function}|}|
% Defines a named \cs{luafunction}, indexed from~$1$ (Lua indexes from
% $1$ so |\luafunction0| is not available).
% 
% \DescribeMacro{latex.add_to_callback}
% |latex.add_to_callback(|\meta{callback}, \meta{function}, \meta{description}|)|
% Registers the \meta{function} into the \meta{callback} with a textual
% \meta{description} of the function. Functions are inserted into the callback
% in the order loaded.
% 
% \DescribeMacro{latex.original_callback_register}
% A copy of the built-in function |callback.register| as direct use of the
% latter is disabled. This copy is available such that more sophisticated
% callback management may be provided at a package level.
% 
% \StopEventually{}
%
% \section{Minimum Lua\TeX{} version}
%
% Lua\TeX{} has changed a lot over time. In the kernel support for ancient
% versions is not provided: trying to build a format with a very old binary
% therefore gives some information in the log and loading stops.
%    \begin{macrocode}
\ifnum\luatexversion<37 %
  \wlog{***************************************************}
  \wlog{* LuaTeX version too old for kernel-level support *}
  \wlog{***************************************************}
  \expandafter\endinput
\fi
%    \end{macrocode}
%
% \section{Attributes}
%
% \begin{macro}{\newattribute}
% \changes{v1.0a}{2015/07/02}{Macro added}
%   Note that unlike other \TeX{} registers these
%   need to be `unset'  to be initialised correctly. Unlike other allocators we
%   start from $1$ here: some code assumes that |\attribute0| is never used!
%    \begin{macrocode}
\def\newattribute#1{%
  \e@alloc\attribute\luatexattributedef
    {\count258}\m@ne\e@alloc@top#1%
  #1=-"7FFFFFFF\relax
}
\count258=\z@
%    \end{macrocode}
% \end{macro}
%
% \section{Category code tables}
%
% Category code table management normally requires a stack: that is not covered
% here but rather is left to a higher-level package.
%
% \begin{macro}{\newcatcodetable}
% \changes{v1.0a}{2015/07/02}{Macro added}
%   Category code tables are allocated with a limit half of that used by Lua\TeX{}
%   for everything else. At the end of allocation there needs to be an
%   initialisation step. Table $0$ is already taken (it's the global one for
%   current use) so the allocation starts at~$1$.
%    \begin{macrocode}
\def\newcatcodetable#1{%
  \e@alloc\catcodetable\chardef
    {\count259}\m@ne\ctabletop#1%
  \luatexinitcatcodetable\allocationnumber
}
\chardef\ctabletop"8000 % TEMP
\count259=\z@
%    \end{macrocode}
% \end{macro}
%
% \section{Named Lua functions}
%
% \begin{macro}{\newluafunction}
% \changes{v1.0a}{2015/07/02}{Macro added}
%   Much the same story for allocating Lua\TeX{} functions except here they are
%   just numbers. No previous approaches exist for this allocator, so there is
%   less to worry about. Lua indexes from $1$ so the allocator starts from zero.
%    \begin{macrocode}
\def\newluafunction{%
  \e@alloc\luafunction\e@alloc@chardef
    {\count260}\m@ne\e@alloc@top
}
\count260=\z@
%    \end{macrocode}
% \end{macro}
%
% \section{Lua loader}
%
% Load the Lua code at the start of every job
%
%    \begin{macrocode}
\everyjob\expandafter{%
  \the\everyjob
  \directlua{require(latex2e.lua)}%
}
%    \end{macrocode}
%
%    \begin{macrocode}
%</2ekernel>
%    \end{macrocode}
%
% \section{Lua callback management}
%
% The native mechanism for callbacks in Lua allows only one per function.
% That is extremely restrictive and so a mechanism is needed to add and
% remove callbacks from the appropriate hooks.
%
%    \begin{macrocode}
%<*2elua>
%    \end{macrocode}
%
% \subsubsection{Basics and support}
%
% \begin{macro}{latex}
% \changes{v1.0a}{2015/07/02}{Table added}
%   Set up the table for the returned functions. This is used to expose
%   all of the public functions.
%    \begin{macrocode}
latex       = latex or { }
local latex = latex
%    \end{macrocode}
% \end{macro}
% 
% Some Lua best practice: use local versions of functions where possible.
%    \begin{macrocode}
local string_gsub = string.gsub
local write_nl = texio.write_nl
%    \end{macrocode}
%
% There are various warnings and errors that need to be given. For warnings
% we can get exactly the same formatting as from \TeX{}. For errors we have to
% make some changes. Here we give the text of the error in the \LaTeX{} format
% then force an error from Lua to halt the run. Splitting the message text is
% done using |\n| which takes the place of |\MessageBreak|.
%    \begin{macrocode}
local function msg_format(text, leader)
  local cont = string_gsub(leader, ".", " ")
  return leader .. " " .. string_gsub(text, "\n", "\n" .. cont .. " ") .. "\n"
end
local function latex_warning(text)
  local i
  for _,i in ipairs(msg_format(text, "LaTeX Warning:"):explode("\n")) do
    write_nl("term and log", i)
  end
end
local function latex_error(text)
  local i
  for _,i in ipairs(msg_format(text, "! LaTeX Error:"):explode("\n")) do
    write_nl("term and log", i)
  end
  write_nl("term and log", "\n")
  error("See LaTeX Error")
end
%    \end{macrocode}
%
% \subsubsection{Housekeeping}
%
%   The main table: keys are callback names, and values are the associated lists
%   of functions. More precisely, the entries in the list are tables holding the
%   actual function as |func| and the identifying description as |description|.
%   Only callbacks with a non-empty list of functions have an entry in this
%   list.
%    \begin{macrocode}
local callbacklist = callbacklist or { }
%    \end{macrocode}
%
%   Numerical codes for callback types, and name-to-value association (the
%   table keys are strings, the values are numbers).
%    \begin{macrocode}
local list, data, exclusive, simple = 1, 2, 3, 4
local types = {
  list      = list,
  data      = data,
  exclusive = exclusive,
  simple    = simple,
}
%    \end{macrocode}
% 
%  Now, list all predefined callbacks with their current type, based on the
%  Lua\TeX{} manual version 0.80. A full list of the currently-available
%  callbacks can be obtained using
%  \begin{verbatim}
%    \directlua{
%      for i,_ in pairs(callback.list()) do
%        texio.write_nl("- " .. i)
%      end
%    }
%    \bye
%  \end{verbatim}
%  in plain Lua\TeX{}. (Some undocumented callbacks are omitted as they are
%  to be removed.)
%    \begin{macrocode}
local callbacktypes = callbacktypes or {
%    \end{macrocode}
%   Section 4.1.1: file discovery callbacks.
%    \begin{macrocode}
  find_read_file     = exclusive,
  find_write_file    = exclusive,
  find_font_file     = data,
  find_output_file   = data,
  find_format_file   = data,
  find_vf_file       = data,
  find_map_file      = data,
  find_enc_file      = data,
  find_sfd_file      = data,
  find_pk_file       = data,
  find_data_file     = data,
  find_opentype_file = data,
  find_truetype_file = data,
  find_type1_file    = data,
  find_image_file    = data,
%    \end{macrocode}
%   Section 4.1.2: file reading callbacks.
%    \begin{macrocode}
  open_read_file     = exclusive,
  read_font_file     = exclusive,
  read_vf_file       = exclusive,
  read_map_file      = exclusive,
  read_enc_file      = exclusive,
  read_sfd_file      = exclusive,
  read_pk_file       = exclusive,
  read_data_file     = exclusive,
  read_truetype_file = exclusive,
  read_type1_file    = exclusive,
  read_opentype_file = exclusive,
%    \end{macrocode}
%   Section 4.1.3: data processing callbacks.
%    \begin{macrocode}
  process_input_buffer  = data,
  process_output_buffer = data,
  process_jobname       = data,
  token_filter          = exclusive,
%    \end{macrocode}
%   Section 4.1.4: node list processing callbacks.
%    \begin{macrocode}
  buildpage_filter      = simple,
  pre_linebreak_filter  = list,
  linebreak_filter      = list,
  post_linebreak_filter = list,
  hpack_filter          = list,
  vpack_filter          = list,
  pre_output_filter     = list,
  hyphenate             = simple,
  ligaturing            = simple,
  kerning               = simple,
  mlist_to_hlist        = list,
%    \end{macrocode}
%   Section 4.1.5: information reporting callbacks.
%    \begin{macrocode}
  pre_dump            = simple,
  start_run           = simple,
  stop_run            = simple,
  start_page_number   = simple,
  stop_page_number    = simple,
  show_error_hook     = simple,
  show_error_message  = simple,
  show_lua_error_hook = simple,
  start_file          = simple,
  stop_file           = simple,
%    \end{macrocode}
%   Section 4.1.6: PDF-related callbacks.
%    \begin{macrocode}
  finish_pdffile = data,
  finish_pdfpage = data,
%    \end{macrocode}
%   Section 4.1.7: font-related callbacks.
%    \begin{macrocode}
  define_font = first,
%    \end{macrocode}
%   Undocumented callbacks which are likely to get documented.
%    \begin{macrocode}
  find_cidmap_file           = data,
  pdf_stream_filter_callback = data,
}
%    \end{macrocode}
%
% \begin{macro}{original_callback_register}
% \changes{v1.0a}{2015/07/02}{Function modified}
% \begin{macro}{callback.register}
% \changes{v1.0a}{2015/07/02}{Function modified}
%   Save the original function for registering callbacks and prevent the
%   original being used. The original is saved in a place that remains
%   available so other more sophisticated code can override the approach
%   taken by the kernel is desired.
%    \begin{macrocode}
original_callback_register = original_callback_register or callback.register
latex.original_callback_register = original_callback_register
function callback.register()
  latex_error("Attempt to use callback.register() directly.")
end
%    \end{macrocode}
% \end{macro}
% \end{macro}
%
% \subsubsection{Handlers}
%
% The handler function is registered into the callback when the
% first function is added to this callback's list. Then, when the callback
% is called, then handler takes care of running all functions in the list.
% When the last function is removed from the callback's list, the handler
% is unregistered.
%
% More precisely, the functions below are used to generate a specialized
% function (closure) for a given callback, which is the actual handler.
%
% Handler for |data| callbacks.
%    \begin{macrocode}
local function data_handler(name)
  return function(data, ...)
    local i
    for _,i in ipairs(callbacklist[name]) do
      data = i.func(data)
    end
    return data
  end
end
%    \end{macrocode}
% Handler for |exclusive| callbacks. We can assume |callbacklist[name]| is not
% empty: otherwise, the function wouldn't be registered in the callback any
% more.
%    \begin{macrocode}
local function exclusive_handler(name)
  return function(...)
    return callbacklist[name][1].func(...)
  end
end
%    \end{macrocode}
% Handler for |list| callbacks.
%    \begin{macrocode}
local function list_handler(name)
  return function(head, ...)
    local ret
    local alltrue = true
    local i
    for _i in ipairs(callbacklist[name]) do
      ret = i.func(head, ...)
      if ret == false then
        latex_warning(
          "Function `i.description' returned false\n"
            .. "in callback `name'"
         )
         break
      end
      if ret ~= true then
        alltrue = false
        head = ret
      end
    end
    return alltrue and true or head
  end
end
%    \end{macrocode}
% Handler for |simple| callbacks.
%    \begin{macrocode}
local function simple_handler(name)
  return function(...)
    local i
    for _,i in ipairs(callbacklist[name]) do
      i.func(...)
    end
  end
end
%    \end{macrocode}
%    
% Keep a handlers table for indexed access.
%    \begin{macrocode}
local handlers = {
  [data]      = data_handler,
  [exclusive] = exclusive_handler,
  [list]      = list_handler,
  [simple]    = simple_handler,
}
%    \end{macrocode}
%
% \subsubsection{Public functions for callback management}
%
% \begin{macro}{latex.add_to_callback}
% \changes{v1.0a}{2015/07/02}{Function added}
%   Add a function to a callback. First check arguments.
%    \begin{macrocode}
function add_to_callback(name, func, description)
  if
    not name or
    name == "" or
    not callbacktypes[name] or
    type(func) ~= "function" or
    not description or 
    description == "" then
    latex_error(
      "Unable to register callback.\n\n"
        .. "Correct usage:\n"
        .. "add_to_callback(<callback>, <function>, <description>)"
    )
  end
%    \end{macrocode}
%   Then test if this callback is already in use. If not, initialise its list
%   and register the proper handler.
%    \begin{macrocode}
  local l = callbacklist[name]
  if not l then
    l = { }
    callbacklist[name] = l
    original_callback_register(name, handlers[callbacktypes[name]](name))
  end
%    \end{macrocode}
%  Actually register the function and give an error if more than one
%  |exclusive| one is registered.
%    \begin{macrocode}
  local f = {
    func        = func,
    description = description,
  }
  local priority = #l + 1
  if callbacktypes[name] == exclusive then
    if #l == 1 then
      latex_error(
        "Cannot add second callback to exclusive function `" .. name .. "'."
      )
    end
  end
  table.insert(l, priority, f)
%    \end{macrocode}
%  Keep user informed.
%    \begin{macrocode}
  write_nl(
    "Inserting callback `" .. description .. "' at position "
      .. priority .. " in `" .. name .. "'."
  )
end
latex.add_to_callback = add_to_callback
%    \end{macrocode}
% \end{macro}
%
%    \begin{macrocode}
%</2elua>
%    \end{macrocode}
%
% \Finale

--------------020201020305050602090205
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

_______________________________________________
Latex-team mailing list
Latex-team@latex-project.org
https://lists.dante.de/mailman/listinfo/latex-team

--------------020201020305050602090205--