User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-GB; rv:1.9.2.17)
            Gecko/20110414 Thunderbird/3.1.10
MIME-Version: 1.0
References: <4DF7CBEF.6080402@arcor.de>
            <20110614221243.GA9520@oberdiek.my-fqdn.de>
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
Message-ID:  <4DF8B044.2040903@morningstar2.co.uk>
Date:         Wed, 15 Jun 2011 14:14:44 +0100
Reply-To: Mailing list for the LaTeX3 project
              <LATEX-L@LISTSERV.UNI-HEIDELBERG.DE>
Sender: Mailing list for the LaTeX3 project <LATEX-L@LISTSERV.UNI-HEIDELBERG.DE>
From: Joseph Wright <joseph.wright@MORNINGSTAR2.CO.UK>
Subject: Re: xparse and hyperref: Token not allowed in a PDF string (PDFDocEncoding)
To: LATEX-L@LISTSERV.UNI-HEIDELBERG.DE
In-Reply-To:  <20110614221243.GA9520@oberdiek.my-fqdn.de>
Precedence: list
Status: R

On 14/06/2011 23:12, Heiko Oberdiek wrote:
> Yes. \NewDocumentCommand defines the macro using e-TeX's \protected.
> The result of \show\fooa:
> | \fooa=\protected macro:
> | ->FooA.
> 
> \pdfstringdef uses \edef to expand the string, \protected
> prevents this and the unexpandable macro is then filtered out:
> | Package hyperref Warning: Token not allowed in a PDF string (PDFDocEncoding):
> | (hyperref)                removing `\fooa' on input line 9.
> 
>> Would it be possible to make
>> \NewDocumentCommand macros work with hyperref bookmarks?
> 
> No, even worse, you cannot provide a bookmark string
> (\texorpdfstring) inside the definition, because it will not
> be expanded and \texorpdfstring is not seen.
> 
> 
>> I'd prefer
>> using \NewDocumentCommand macros because of the mandatory argument of
>> \DeclareExpandableDocumentCommand macros which I don't need in my context.
> 
> I don't see that mandatory arguments matter here.
> But probably you are refering to "1.7 Fully-expandable document commands":
> | * The function must have at least one mandatory argument, and in
> |   particular the last argument must be one of the mandatory types
> |   (l, m or u).
> 
> The section 1.7 contains advice against \DeclareExpandableDocumentCommand:
> | do not use these functions!
> One translation is: Good bye bookmarks.
> 
> Of course there are macros, whose definition text makes it
> difficult for bookmarks anyway (assignments, ...). But
> the expandable macros are excluded using \DeclareDocumentCommand.
> 
> Workarounds:
> * \pdfstringdefDisableCommands can be used, but
>   an expandable macro needs to be added for each \protected
>   macro that might appear in bookmarks. Then the redefinition
>   takes place for each bookmark string conversion, not very
>   efficient.
> * Expandable wrapper command that expands to the bookmark string
>   if inside bookmarks and calls the unexpandable macro otherwise.
> * LuaTeX would add access to node lists, thus the
>   bookmark string could be set in a \hbox and the node list
>   examined and converted to a text string. Some cases are
>   probably quite easy, other cases are very difficult and
>   would cry for OCR methods.

Clearly, dealing with bookmarks is something that will need to be
addressed in the longer term in a more systematic way: it is not the
intention to say 'Good bye bookmarks'!

To me, it seems that:
 - only some document commands need to work inside bookmarks, although
   it takes a human to decide which ones.
 - the definition for use within bookmarks needs to be carefully worked
   out, so again automation is out.
 - definitions for expandable contexts will in general be different from
   non-expandable ones.

There are a number of ways we could handle this. One way would be to
have a \DeclareBookmarkableCommand (or similar) macro, which would work
in the form

   \DeclareBookmarkableCommand\foo{m}
     {Standard code}
     {Bookmark code}

and then to define \foo as Heiko suggests

  \def\foo{\texorpdfstring{\InternalFoo}{Bookmark Code}}

although I'm not sure what happens in other expandable contexts (I
assume anything other than a bookmark context uses branch 1).

A second alternative would be to go with Frank's idea of separating out
interface and implementation

  \DeclareDocumentCommandInterface
  \DeclareDocumentCommandImplementation

and to provide a way to switch implementations.

I think I prefer the first approach, but in general expandable versus
non-expandable document commands are still a problem, as are variations
in implementation. So we may need to think a bit more widely about this.
--
Joseph Wright