markdown: handle inline code spans safely

Escape XML-special characters in Markdown-style inline backtick
spans and convert them before paragraph and list handling so
literals such as <Log> and <List> work correctly in .autodoc
files.

Refactor the inline-backtick logic into local Markdown helpers and
cover the behavior with focused misc tests.

Co-authored-by: Codex <codex@openai.com>

Author: fingolfin

doc/Comments.autodoc

@@ -26,7 +26,7 @@ Inside of &AutoDoc; comments, <E>&AutoDoc; commands</E>
 starting with <C>@</C> can be used to control the output &AutoDoc; produces.
 Any comment line that does <E>not</E> start with an &AutoDoc; command is treated
 as regular documentation text and may contain (almost) arbitrary &GAPDoc; XML
-markup, such as <C>&lt;Ref&gt;</C>, <C>&lt;A&gt;</C>, <C>&lt;List&gt;</C>, and similar tags.
+markup, such as `<Ref>`, `<A>`, `<List>`, and similar tags.
 This lets you use the normal &GAPDoc; formatting toolbox directly inside
 &AutoDoc; comments.
 
@@ -287,7 +287,7 @@ first entry in the group. See <Ref Sect="Section_Groups"/> for more information.
 @Index "@BeginExample" `@BeginExample`
 @Index "@EndExample" `@EndExample`
 <C>@BeginExample</C> marks the start of an example to be put into the manual.
-It differs from &GAPDoc;'s <C>&lt;Example></C> (see <Ref Subsect="Log" BookName="gapdoc"/>),
+It differs from &GAPDoc;'s `<Example>` (see <Ref Subsect="Log" BookName="gapdoc"/>),
 in that it expects actual code (not in a comment) interspersed with comments,
 to allow for examples files that can be both executed by &GAP;, and parsed
 by &AutoDoc;. To achieve this, &GAP; commands are not preceded by a comment,
@@ -323,7 +323,7 @@ Section <Ref Sect="Subsection_Tut:IntegrateExisting:EverythingThere"/>).
 @Index "@EndExampleSession" `@EndExampleSession`
 <C>@BeginExampleSession</C> marks the start of an example to be put into the manual,
 while <C>@EndExampleSession</C> ends the example block.
-It is the direct analog of &GAPDoc;'s <C>&lt;Example></C> (see <Ref Subsect="Log" BookName="gapdoc"/>).
+It is the direct analog of &GAPDoc;'s `<Example>` (see <Ref Subsect="Log" BookName="gapdoc"/>).
 
 To illustrate this command, consider this input:
 ```@listing
@@ -368,7 +368,7 @@ The &AutoDoc; command <C>@Log</C> is an alias of <C>@BeginLog</C>.
 @Index "@EndLogSession" `@EndLogSession`
 Works just like the <C>@BeginExampleSession</C> command, but the example
 will not be tested if manual examples are run. It is the direct analog of
-&GAPDoc;'s <C>&lt;Log></C> (see <Ref Subsect="Log" BookName="gapdoc"/>).
+&GAPDoc;'s `<Log>` (see <Ref Subsect="Log" BookName="gapdoc"/>).
 The &AutoDoc; command <C>@LogSession</C> is an alias of <C>@BeginLogSession</C>.
 
 @Subsection <C>@DoNotReadRestOfFile</C>
@@ -483,7 +483,7 @@ in the HTML and text versions of the manual or worksheet, but not in the PDF ver
 @Index "@Index" <C>@Index <A>key</A> [<A>entry text</A>]</C>
 Adds an index entry to the current documentation text.
 The command <C>@Index key entry text</C> generates
-<C>&lt;Index Key="key"&gt;entry text&lt;/Index&gt;</C>.
+`<Index Key="key">entry text</Index>`.
 If no entry text is provided, then the entry text is empty.
 To use keys containing spaces, wrap the key in double quotes, e.g.
 <C>@Index "my key" entry text</C>.
@@ -589,7 +589,7 @@ written in <F>.autodoc</F> format.
 @Subsection First Subsection
 
 This text belongs directly to the manual chapter.
-It can use XML tags such as <A>arg</A> or <C>&lt;Ref ...&gt;</C>.
+It can use XML tags such as <A>arg</A> or <Ref .../>.
 
 @BeginExampleSession
 gap> S5 := SymmetricGroup(5);
@@ -792,7 +792,7 @@ One can insert verbatim code blocks by placing the code between lines
    fence may optionally be followed by an info string. The values
    <C>@listing</C>, <C>@example</C>, and <C>@log</C> select the corresponding
    GAPDoc element; any other value is currently ignored. If nothing is specified
-   the default is to generate <C>&lt;Listing&gt;</C>. Example:
+   the default is to generate `<Listing>`. Example:
 
 ````@listing
 #! ```@listing

doc/Tutorials.autodoc

@@ -132,7 +132,7 @@ DeclareOperation( "ToricVariety", [ IsConvexObject ] );
 In these comment lines, you can freely use normal &GAPDoc; XML markup
 (with the usual exceptions for lines starting with <C>@</C>, which are
 interpreted as &AutoDoc; commands). So, for instance, tags like
-<C>&lt;Ref&gt;</C>, <C>&lt;A&gt;</C>, <C>&lt;K&gt;</C>, <C>&lt;List&gt;</C>, etc. can be written
+`<Ref>`, `<A>`, `<K>`, `<List>`, etc. can be written
 directly in <C>#!</C> comment text.
 
 For chapter text, tutorial material, worked examples, and similar prose, some
@@ -511,27 +511,27 @@ of the package info record are taken into account:
 <List>
 
 <Mark>PackageName</Mark><Item>
-This is used to set the <C>&lt;Title></C> element of the
+This is used to set the `<Title>` element of the
 title page.
 </Item>
 
 <Mark>Subtitle</Mark><Item>
-This is used to set the <C>&lt;Subtitle></C> element of the
+This is used to set the `<Subtitle>` element of the
 title page.
 </Item>
 
 <Mark>Version</Mark><Item>
-This is used to set the <C>&lt;Version></C> element of the
+This is used to set the `<Version>` element of the
 title page, with the string <Q>Version </Q> prepended.
 </Item>
 
 <Mark>Date</Mark><Item>
-This is used to set the <C>&lt;Date></C> element of the
+This is used to set the `<Date>` element of the
 title page.
 </Item>
 
 <Mark>Persons</Mark><Item>
-This is used to generate <C>&lt;Author></C> elements in the
+This is used to generate `<Author>` elements in the
 generated title page.
 </Item>
 
@@ -562,7 +562,7 @@ SetPackageInfo( rec(
   )
 ) );
 ```
-This inserts <C>&lt;Copyright></C> and <C>&lt;Acknowledgements></C> blocks into
+This inserts `<Copyright>` and `<Acknowledgements>` blocks into
 the generated <F>title.xml</F>.
 
 <C>PkgInfo.AutoDoc.TitlePage</C> has exactly the same meaning as

gap/Markdown.gd

@@ -5,5 +5,4 @@
 #
 # SPDX-License-Identifier: GPL-2.0-or-later
 
-DeclareGlobalFunction( "INSERT_IN_STRING_WITH_REPLACE" );
 DeclareGlobalFunction( "CONVERT_LIST_OF_STRINGS_IN_MARKDOWN_TO_GAPDOC_XML" );

gap/Markdown.gi

@@ -6,7 +6,7 @@
 # SPDX-License-Identifier: GPL-2.0-or-later
 
 ##
-InstallGlobalFunction( INSERT_IN_STRING_WITH_REPLACE,
+BindGlobal( "INSERT_IN_STRING_WITH_REPLACE",
   function( string, new_string, position, nr_letters_to_be_replaced )
     return Concatenation(
                string{[ 1 .. position - 1 ]},
@@ -15,6 +15,70 @@ InstallGlobalFunction( INSERT_IN_STRING_WITH_REPLACE,
            );
 end );
 
+##
+BindGlobal( "AUTODOC_EscapeXMLTextForInlineCode",
+  function( string )
+    local escaped_string, split_pos;
+
+    escaped_string := "";
+    while Length(string) > 0 do
+        split_pos := PositionProperty( string, c -> c in "&\"<>" );
+        if split_pos = fail then
+            Append( escaped_string, string );
+            break;
+        fi;
+        if split_pos > 1 then
+            Append( escaped_string, string{ [ 1 .. split_pos - 1 ] } );
+        fi;
+        if string[ split_pos ] = '&' then
+            Append( escaped_string, "&amp;" );
+        elif string[ split_pos ] = '"' then
+            Append( escaped_string, "&quot;" );
+        elif string[ split_pos ] = '<' then
+            Append( escaped_string, "&lt;" );
+        else
+            Append( escaped_string, "&gt;" );
+        fi;
+        string := string{ [ split_pos + 1 .. Length( string ) ] };
+    od;
+
+    return escaped_string;
+end );
+
+##
+BindGlobal( "AUTODOC_ConvertInlineBackticksInLine",
+  function( string, keyword_set )
+    local opening_pos, closing_pos, inline_content, tag_name;
+
+    while PositionSublist( string, "`" ) <> fail do
+        opening_pos := PositionSublist( string, "`" );
+        closing_pos := PositionSublist( string, "`", opening_pos + 1 );
+        if closing_pos = fail then
+            Error( "did you forget some `" );
+        fi;
+
+        if opening_pos + 1 <= closing_pos - 1 then
+            inline_content := string{ [ opening_pos + 1 .. closing_pos - 1 ] };
+        else
+            inline_content := "";
+        fi;
+        if inline_content in keyword_set then
+            tag_name := "Keyword";
+        else
+            tag_name := "Code";
+        fi;
+        string := Concatenation(
+            string{ [ 1 .. opening_pos - 1 ] },
+            "<", tag_name, ">",
+            AUTODOC_EscapeXMLTextForInlineCode( inline_content ),
+            "</", tag_name, ">",
+            string{ [ closing_pos + 1 .. Length( string ) ] }
+        );
+    od;
+
+    return string;
+end );
+
 ##
 InstallGlobalFunction( CONVERT_LIST_OF_STRINGS_IN_MARKDOWN_TO_GAPDOC_XML,
   function( string_list )
@@ -23,8 +87,7 @@ InstallGlobalFunction( CONVERT_LIST_OF_STRINGS_IN_MARKDOWN_TO_GAPDOC_XML,
           commands, position_of_command, insert, beginning_whitespaces, temp, string_list_temp, skipped,
           already_inserted_paragraph, in_list, in_item, converted_string_list,
           fence_char, fence_length, trimmed_line, code_block, info_string,
-          fence_element, keyword_set, opening_pos, closing_pos, inline_content,
-          tag_name;
+          fence_element, keyword_set;
 
     converted_string_list := [ ];
     i := 1;
@@ -94,6 +157,25 @@ InstallGlobalFunction( CONVERT_LIST_OF_STRINGS_IN_MARKDOWN_TO_GAPDOC_XML,
     od;
     string_list := converted_string_list;
 
+    # Convert inline backticks before list detection so literal tags such as
+    # `<List>` inside code spans do not look like structural GAPDoc tags.
+    keyword_set := Set( ALL_KEYWORDS() );
+    skipped := false;
+    for i in [ 1 .. Length( string_list ) ] do
+        if PositionSublist( string_list[ i ], "<![CDATA[" ) <> fail then
+            skipped := true;
+        fi;
+        if PositionSublist( string_list[ i ], "]]>" ) <> fail then
+            skipped := false;
+            continue;
+        fi;
+        if skipped = true then
+            continue;
+        fi;
+        string_list[ i ] :=
+            AUTODOC_ConvertInlineBackticksInLine( string_list[ i ], keyword_set );
+    od;
+
     ## Check for paragraphs by turning an empty string into <P/>
 
     already_inserted_paragraph := false;
@@ -218,45 +300,6 @@ InstallGlobalFunction( CONVERT_LIST_OF_STRINGS_IN_MARKDOWN_TO_GAPDOC_XML,
                                        [ "$", "Math" ],
                                        [ "**", "Emph" ],
                                        [ "__", "Emph" ] ];
-    keyword_set := Set( ALL_KEYWORDS() );
-
-    skipped := false;
-    for i in [ 1 .. Length( string_list ) ] do
-        if PositionSublist( string_list[ i ], "<![CDATA[" ) <> fail then
-            skipped := true;
-        fi;
-        if PositionSublist( string_list[ i ], "]]>" ) <> fail then
-            skipped := false;
-            continue;
-        fi;
-        if skipped = true then
-            continue;
-        fi;
-
-        while PositionSublist( string_list[ i ], "`" ) <> fail do
-            opening_pos := PositionSublist( string_list[ i ], "`" );
-            closing_pos := PositionSublist( string_list[ i ], "`", opening_pos + 1 );
-            if closing_pos = fail then
-                Error( "did you forget some `" );
-            fi;
-
-            if opening_pos + 1 <= closing_pos - 1 then
-                inline_content := string_list[ i ]{ [ opening_pos + 1 .. closing_pos - 1 ] };
-            else
-                inline_content := "";
-            fi;
-            if inline_content in keyword_set then
-                tag_name := "Keyword";
-            else
-                tag_name := "Code";
-            fi;
-            string_list[ i ] := Concatenation(
-                string_list[ i ]{ [ 1 .. opening_pos - 1 ] },
-                "<", tag_name, ">", inline_content, "</", tag_name, ">",
-                string_list[ i ]{ [ closing_pos + 1 .. Length( string_list[ i ] ) ] }
-            );
-        od;
-    od;
 
     ## special handling for \$
     for i in [ 1 .. Length( string_list ) ] do

tst/manual.expected/_Chapter_Comments.xml

@@ -29,7 +29,7 @@ Inside of &AutoDoc; comments, <E>&AutoDoc; commands</E>
 starting with <C>@</C> can be used to control the output &AutoDoc; produces.
 Any comment line that does <E>not</E> start with an &AutoDoc; command is treated
 as regular documentation text and may contain (almost) arbitrary &GAPDoc; XML
-markup, such as <C>&lt;Ref&gt;</C>, <C>&lt;A&gt;</C>, <C>&lt;List&gt;</C>, and similar tags.
+markup, such as <Code>&lt;Ref&gt;</Code>, <Code>&lt;A&gt;</Code>, <Code>&lt;List&gt;</Code>, and similar tags.
 This lets you use the normal &GAPDoc; formatting toolbox directly inside
 &AutoDoc; comments.
 <P/>
@@ -344,7 +344,7 @@ first entry in the group. See <Ref Sect="Section_Groups"/> for more information.
 <Index Key="@BeginExample"><Code>@BeginExample</Code></Index>
 <Index Key="@EndExample"><Code>@EndExample</Code></Index>
 <C>@BeginExample</C> marks the start of an example to be put into the manual.
-It differs from &GAPDoc;'s <C>&lt;Example></C> (see <Ref Subsect="Log" BookName="gapdoc"/>),
+It differs from &GAPDoc;'s <Code>&lt;Example&gt;</Code> (see <Ref Subsect="Log" BookName="gapdoc"/>),
 in that it expects actual code (not in a comment) interspersed with comments,
 to allow for examples files that can be both executed by &GAP;, and parsed
 by &AutoDoc;. To achieve this, &GAP; commands are not preceded by a comment,
@@ -384,7 +384,7 @@ Section <Ref Sect="Subsection_Tut:IntegrateExisting:EverythingThere"/>).
 <Index Key="@EndExampleSession"><Code>@EndExampleSession</Code></Index>
 <C>@BeginExampleSession</C> marks the start of an example to be put into the manual,
 while <C>@EndExampleSession</C> ends the example block.
-It is the direct analog of &GAPDoc;'s <C>&lt;Example></C> (see <Ref Subsect="Log" BookName="gapdoc"/>).
+It is the direct analog of &GAPDoc;'s <Code>&lt;Example&gt;</Code> (see <Ref Subsect="Log" BookName="gapdoc"/>).
 <P/>
 To illustrate this command, consider this input:
 <Listing><![CDATA[
@@ -437,7 +437,7 @@ The &AutoDoc; command <C>@Log</C> is an alias of <C>@BeginLog</C>.
 <Index Key="@EndLogSession"><Code>@EndLogSession</Code></Index>
 Works just like the <C>@BeginExampleSession</C> command, but the example
 will not be tested if manual examples are run. It is the direct analog of
-&GAPDoc;'s <C>&lt;Log></C> (see <Ref Subsect="Log" BookName="gapdoc"/>).
+&GAPDoc;'s <Code>&lt;Log&gt;</Code> (see <Ref Subsect="Log" BookName="gapdoc"/>).
 The &AutoDoc; command <C>@LogSession</C> is an alias of <C>@BeginLogSession</C>.
 <P/>
 </Subsection>
@@ -576,7 +576,7 @@ in the HTML and text versions of the manual or worksheet, but not in the PDF ver
 <Index Key="@Index"><C>@Index <A>key</A> [<A>entry text</A>]</C></Index>
 Adds an index entry to the current documentation text.
 The command <C>@Index key entry text</C> generates
-<C>&lt;Index Key="key"&gt;entry text&lt;/Index&gt;</C>.
+<Code>&lt;Index Key=&quot;key&quot;&gt;entry text&lt;/Index&gt;</Code>.
 If no entry text is provided, then the entry text is empty.
 To use keys containing spaces, wrap the key in double quotes, e.g.
 <C>@Index "my key" entry text</C>.
@@ -935,7 +935,7 @@ One can insert verbatim code blocks by placing the code between lines
    fence may optionally be followed by an info string. The values
    <C>@listing</C>, <C>@example</C>, and <C>@log</C> select the corresponding
    GAPDoc element; any other value is currently ignored. If nothing is specified
-   the default is to generate <C>&lt;Listing&gt;</C>. Example:
+   the default is to generate <Code>&lt;Listing&gt;</Code>. Example:
 <P/>
 <Listing><![CDATA[
 #! ```@listing

tst/manual.expected/_Chapter_Tutorials.xml

@@ -152,7 +152,7 @@ DeclareOperation( "ToricVariety", [ IsConvexObject ] );
 In these comment lines, you can freely use normal &GAPDoc; XML markup
 (with the usual exceptions for lines starting with <C>@</C>, which are
 interpreted as &AutoDoc; commands). So, for instance, tags like
-<C>&lt;Ref&gt;</C>, <C>&lt;A&gt;</C>, <C>&lt;K&gt;</C>, <C>&lt;List&gt;</C>, etc. can be written
+<Code>&lt;Ref&gt;</Code>, <Code>&lt;A&gt;</Code>, <Code>&lt;K&gt;</Code>, <Code>&lt;List&gt;</Code>, etc. can be written
 directly in <C>#!</C> comment text.
 <P/>
 For chapter text, tutorial material, worked examples, and similar prose, some
@@ -557,23 +557,23 @@ of the package info record are taken into account:
 <P/>
 <List>
 <Mark>PackageName</Mark><Item>
-This is used to set the <C>&lt;Title></C> element of the
+This is used to set the <Code>&lt;Title&gt;</Code> element of the
 title page.
 </Item>
 <Mark>Subtitle</Mark><Item>
-This is used to set the <C>&lt;Subtitle></C> element of the
+This is used to set the <Code>&lt;Subtitle&gt;</Code> element of the
 title page.
 </Item>
 <Mark>Version</Mark><Item>
-This is used to set the <C>&lt;Version></C> element of the
+This is used to set the <Code>&lt;Version&gt;</Code> element of the
 title page, with the string <Q>Version </Q> prepended.
 </Item>
 <Mark>Date</Mark><Item>
-This is used to set the <C>&lt;Date></C> element of the
+This is used to set the <Code>&lt;Date&gt;</Code> element of the
 title page.
 </Item>
 <Mark>Persons</Mark><Item>
-This is used to generate <C>&lt;Author></C> elements in the
+This is used to generate <Code>&lt;Author&gt;</Code> elements in the
 generated title page.
 </Item>
 <Mark>PackageDoc</Mark><Item>
@@ -602,7 +602,7 @@ SetPackageInfo( rec(
   )
 ) );
 ]]></Listing>
-This inserts <C>&lt;Copyright></C> and <C>&lt;Acknowledgements></C> blocks into
+This inserts <Code>&lt;Copyright&gt;</Code> and <Code>&lt;Acknowledgements&gt;</Code> blocks into
 the generated <F>title.xml</F>.
 <P/>
 <C>PkgInfo.AutoDoc.TitlePage</C> has exactly the same meaning as