diff mbox series

[v2,07/21] docs/qapidoc: fix nested parsing under untagged sections

Message ID 20240626222128.406106-8-jsnow@redhat.com (mailing list archive)
State New, archived
Headers show
Series qapi: convert "Note" and "Example" sections to rST | expand

Commit Message

John Snow June 26, 2024, 10:21 p.m. UTC 
Sphinx does not like sections without titles, because it wants to
convert every section into a reference. When there is no title, it
struggles to do this and transforms the tree inproperly.

Depending on the rST used, this may result in an assertion error deep in
the docutils HTMLWriter.

(Observed when using ".. admonition:: Notes" under such a section - When
this is transformed with its own <title> element, Sphinx is fooled into
believing this title belongs to the section and incorrect mutates the
docutils tree, leading to errors during rendering time.)

When parsing an untagged section (free paragraphs), skip making a hollow
section and instead append the parse results to the prior section.

Many Bothans died to bring us this information.

Signed-off-by: John Snow <jsnow@redhat.com>
Acked-by: Markus Armbruster <armbru@redhat.com>
---
 docs/sphinx/qapidoc.py | 16 +++++++++++-----
 1 file changed, 11 insertions(+), 5 deletions(-)

Comments

Markus Armbruster June 28, 2024, 7:54 a.m. UTC | #1 
John Snow <jsnow@redhat.com> writes:

> Sphinx does not like sections without titles, because it wants to
> convert every section into a reference. When there is no title, it
> struggles to do this and transforms the tree inproperly.
>
> Depending on the rST used, this may result in an assertion error deep in
> the docutils HTMLWriter.
>
> (Observed when using ".. admonition:: Notes" under such a section - When
> this is transformed with its own <title> element, Sphinx is fooled into
> believing this title belongs to the section and incorrect mutates the
> docutils tree, leading to errors during rendering time.)
>
> When parsing an untagged section (free paragraphs), skip making a hollow
> section and instead append the parse results to the prior section.
>
> Many Bothans died to bring us this information.
>
> Signed-off-by: John Snow <jsnow@redhat.com>
> Acked-by: Markus Armbruster <armbru@redhat.com>

Generated HTML changes, but the diff is hard to review due to id
attribute changes all over the place.

Generated qemu-ga-ref.7 also changes:

    diff -rup old/qemu-ga-ref.7 new/qemu-ga-ref.7
    --- old/qemu-ga-ref.7	2024-06-27 10:42:21.466096276 +0200
    +++ new/qemu-ga-ref.7	2024-06-27 10:45:36.502414099 +0200
    @@ -397,6 +397,7 @@ shutdown request, with no guarantee of s
     .B \fBmode\fP: \fBstring\fP (optional)
     \(dqhalt\(dq, \(dqpowerdown\(dq (default), or \(dqreboot\(dq
     .UNINDENT
    +.sp
     This command does NOT return a response on success.  Success
     condition is indicated by the VM exiting with a zero exit status or,
     when running with \-\-no\-shutdown, by issuing the query\-status QMP
    @@ -1348,6 +1349,7 @@ the new password entry string, base64 en
     .B \fBcrypted\fP: \fBboolean\fP
     true if password is already crypt()d, false if raw
     .UNINDENT
    +.sp
     If the \fBcrypted\fP flag is true, it is the caller\(aqs responsibility to
     ensure the correct crypt() encryption scheme is used.  This command
     does not attempt to interpret or report on the encryption scheme.

We add vertical space.  Visible when viewed with man.  Looks like an
improvement to me.

Here's the first of these two spots in HTML:

    -<section id="qapidoc-31">
    -<h3><a class="toc-backref" href="#id13" role="doc-backlink"><code class="docutils literal notranslate"><span class="pre">guest-shutdown</span></code> (Command)</a><a class="headerlink" href="#qapidoc-31" title="Permalink to this heading"></a></h3>
    +<section id="qapidoc-30">
    +<h3><a class="toc-backref" href="#id13" role="doc-backlink"><code class="docutils literal notranslate"><span class="pre">guest-shutdown</span></code> (Command)</a><a class="headerlink" href="#qapidoc-30" title="Permalink to this heading"></a></h3>
     <p>Initiate guest-activated shutdown.  Note: this is an asynchronous
     shutdown request, with no guarantee of successful shutdown.</p>
     <section id="qapidoc-28">
    @@ -502,22 +502,20 @@ shutdown request, with no guarantee of s
     </dd>
     </dl>
     </section>
    -<section id="qapidoc-29">
     <p>This command does NOT return a response on success.  Success
     condition is indicated by the VM exiting with a zero exit status or,
     when running with –no-shutdown, by issuing the query-status QMP
     command to confirm the VM status is “shutdown”.</p>
    -</section>
    -<section id="qapidoc-30">
    -<h4>Since<a class="headerlink" href="#qapidoc-30" title="Permalink to this heading"></a></h4>
    +<section id="qapidoc-29">
    +<h4>Since<a class="headerlink" href="#qapidoc-29" title="Permalink to this heading"></a></h4>
     <p>0.15.0</p>
     </section>
     </section>

The id changes muddy the waters.  With them manually removed:

     <section id="qapidoc-31">
     <h3><a class="toc-backref" href="#id13" role="doc-backlink"><code class="docutils literal notranslate"><span class="pre">guest-shutdown</span></code> (Command)</a><a class="headerlink" href="#qapidoc-31" title="Permalink to this heading"></a></h3>
     <p>Initiate guest-activated shutdown.  Note: this is an asynchronous
     shutdown request, with no guarantee of successful shutdown.</p>
     <section id="qapidoc-28">
    @@ -502,22 +502,20 @@ shutdown request, with no guarantee of s
     </dd>
     </dl>
     </section>
    -<section id="qapidoc-29">
     <p>This command does NOT return a response on success.  Success
     condition is indicated by the VM exiting with a zero exit status or,
     when running with –no-shutdown, by issuing the query-status QMP
     command to confirm the VM status is “shutdown”.</p>
    -</section>
     <section id="qapidoc-30">
     <h4>Since<a class="headerlink" href="#qapidoc-30" title="Permalink to this heading"></a></h4>
     <p>0.15.0</p>
     </section>
     </section>

Makes no visual difference in my browser.

Do these differences match your expectations?
John Snow June 28, 2024, 3:10 p.m. UTC | #2 
On Fri, Jun 28, 2024, 3:55 AM Markus Armbruster <armbru@redhat.com> wrote:

> John Snow <jsnow@redhat.com> writes:
>
> > Sphinx does not like sections without titles, because it wants to
> > convert every section into a reference. When there is no title, it
> > struggles to do this and transforms the tree inproperly.
> >
> > Depending on the rST used, this may result in an assertion error deep in
> > the docutils HTMLWriter.
> >
> > (Observed when using ".. admonition:: Notes" under such a section - When
> > this is transformed with its own <title> element, Sphinx is fooled into
> > believing this title belongs to the section and incorrect mutates the
> > docutils tree, leading to errors during rendering time.)
> >
> > When parsing an untagged section (free paragraphs), skip making a hollow
> > section and instead append the parse results to the prior section.
> >
> > Many Bothans died to bring us this information.
> >
> > Signed-off-by: John Snow <jsnow@redhat.com>
> > Acked-by: Markus Armbruster <armbru@redhat.com>
>
> Generated HTML changes, but the diff is hard to review due to id
> attribute changes all over the place.
>
> Generated qemu-ga-ref.7 also changes:
>
>     diff -rup old/qemu-ga-ref.7 new/qemu-ga-ref.7
>     --- old/qemu-ga-ref.7       2024-06-27 10:42:21.466096276 +0200
>     +++ new/qemu-ga-ref.7       2024-06-27 10:45:36.502414099 +0200
>     @@ -397,6 +397,7 @@ shutdown request, with no guarantee of s
>      .B \fBmode\fP: \fBstring\fP (optional)
>      \(dqhalt\(dq, \(dqpowerdown\(dq (default), or \(dqreboot\(dq
>      .UNINDENT
>     +.sp
>      This command does NOT return a response on success.  Success
>      condition is indicated by the VM exiting with a zero exit status or,
>      when running with \-\-no\-shutdown, by issuing the query\-status QMP
>     @@ -1348,6 +1349,7 @@ the new password entry string, base64 en
>      .B \fBcrypted\fP: \fBboolean\fP
>      true if password is already crypt()d, false if raw
>      .UNINDENT
>     +.sp
>      If the \fBcrypted\fP flag is true, it is the caller\(aqs
> responsibility to
>      ensure the correct crypt() encryption scheme is used.  This command
>      does not attempt to interpret or report on the encryption scheme.
>
> We add vertical space.  Visible when viewed with man.  Looks like an
> improvement to me.
>
> Here's the first of these two spots in HTML:
>
>     -<section id="qapidoc-31">
>     -<h3><a class="toc-backref" href="#id13" role="doc-backlink"><code
> class="docutils literal notranslate"><span
> class="pre">guest-shutdown</span></code> (Command)</a><a class="headerlink"
> href="#qapidoc-31" title="Permalink to this heading"></a></h3>
>     +<section id="qapidoc-30">
>     +<h3><a class="toc-backref" href="#id13" role="doc-backlink"><code
> class="docutils literal notranslate"><span
> class="pre">guest-shutdown</span></code> (Command)</a><a class="headerlink"
> href="#qapidoc-30" title="Permalink to this heading"></a></h3>
>      <p>Initiate guest-activated shutdown.  Note: this is an asynchronous
>      shutdown request, with no guarantee of successful shutdown.</p>
>      <section id="qapidoc-28">
>     @@ -502,22 +502,20 @@ shutdown request, with no guarantee of s
>      </dd>
>      </dl>
>      </section>
>     -<section id="qapidoc-29">
>      <p>This command does NOT return a response on success.  Success
>      condition is indicated by the VM exiting with a zero exit status or,
>      when running with –no-shutdown, by issuing the query-status QMP
>      command to confirm the VM status is “shutdown”.</p>
>     -</section>
>     -<section id="qapidoc-30">
>     -<h4>Since<a class="headerlink" href="#qapidoc-30" title="Permalink to
> this heading"></a></h4>
>     +<section id="qapidoc-29">
>     +<h4>Since<a class="headerlink" href="#qapidoc-29" title="Permalink to
> this heading"></a></h4>
>      <p>0.15.0</p>
>      </section>
>      </section>
>
> The id changes muddy the waters.  With them manually removed:
>
>      <section id="qapidoc-31">
>      <h3><a class="toc-backref" href="#id13" role="doc-backlink"><code
> class="docutils literal notranslate"><span
> class="pre">guest-shutdown</span></code> (Command)</a><a class="headerlink"
> href="#qapidoc-31" title="Permalink to this heading"></a></h3>
>      <p>Initiate guest-activated shutdown.  Note: this is an asynchronous
>      shutdown request, with no guarantee of successful shutdown.</p>
>      <section id="qapidoc-28">
>     @@ -502,22 +502,20 @@ shutdown request, with no guarantee of s
>      </dd>
>      </dl>
>      </section>
>     -<section id="qapidoc-29">
>      <p>This command does NOT return a response on success.  Success
>      condition is indicated by the VM exiting with a zero exit status or,
>      when running with –no-shutdown, by issuing the query-status QMP
>      command to confirm the VM status is “shutdown”.</p>
>     -</section>
>      <section id="qapidoc-30">
>      <h4>Since<a class="headerlink" href="#qapidoc-30" title="Permalink to
> this heading"></a></h4>
>      <p>0.15.0</p>
>      </section>
>      </section>
>
> Makes no visual difference in my browser.
>
> Do these differences match your expectations?
>

Yep!

It does change the output just a little, but Sphinx really doesn't like
title-less sections.

I thought the change looked fine, and I'm still planning on removing this
old generator anyway, so...

>
John Snow June 28, 2024, 3:28 p.m. UTC | #3 
On Fri, Jun 28, 2024, 11:10 AM John Snow <jsnow@redhat.com> wrote:

>
>
> On Fri, Jun 28, 2024, 3:55 AM Markus Armbruster <armbru@redhat.com> wrote:
>
>> John Snow <jsnow@redhat.com> writes:
>>
>> > Sphinx does not like sections without titles, because it wants to
>> > convert every section into a reference. When there is no title, it
>> > struggles to do this and transforms the tree inproperly.
>> >
>> > Depending on the rST used, this may result in an assertion error deep in
>> > the docutils HTMLWriter.
>> >
>> > (Observed when using ".. admonition:: Notes" under such a section - When
>> > this is transformed with its own <title> element, Sphinx is fooled into
>> > believing this title belongs to the section and incorrect mutates the
>> > docutils tree, leading to errors during rendering time.)
>> >
>> > When parsing an untagged section (free paragraphs), skip making a hollow
>> > section and instead append the parse results to the prior section.
>> >
>> > Many Bothans died to bring us this information.
>> >
>> > Signed-off-by: John Snow <jsnow@redhat.com>
>> > Acked-by: Markus Armbruster <armbru@redhat.com>
>>
>> Generated HTML changes, but the diff is hard to review due to id
>> attribute changes all over the place.
>>
>> Generated qemu-ga-ref.7 also changes:
>>
>>     diff -rup old/qemu-ga-ref.7 new/qemu-ga-ref.7
>>     --- old/qemu-ga-ref.7       2024-06-27 10:42:21.466096276 +0200
>>     +++ new/qemu-ga-ref.7       2024-06-27 10:45:36.502414099 +0200
>>     @@ -397,6 +397,7 @@ shutdown request, with no guarantee of s
>>      .B \fBmode\fP: \fBstring\fP (optional)
>>      \(dqhalt\(dq, \(dqpowerdown\(dq (default), or \(dqreboot\(dq
>>      .UNINDENT
>>     +.sp
>>      This command does NOT return a response on success.  Success
>>      condition is indicated by the VM exiting with a zero exit status or,
>>      when running with \-\-no\-shutdown, by issuing the query\-status QMP
>>     @@ -1348,6 +1349,7 @@ the new password entry string, base64 en
>>      .B \fBcrypted\fP: \fBboolean\fP
>>      true if password is already crypt()d, false if raw
>>      .UNINDENT
>>     +.sp
>>      If the \fBcrypted\fP flag is true, it is the caller\(aqs
>> responsibility to
>>      ensure the correct crypt() encryption scheme is used.  This command
>>      does not attempt to interpret or report on the encryption scheme.
>>
>> We add vertical space.  Visible when viewed with man.  Looks like an
>> improvement to me.
>>
>> Here's the first of these two spots in HTML:
>>
>>     -<section id="qapidoc-31">
>>     -<h3><a class="toc-backref" href="#id13" role="doc-backlink"><code
>> class="docutils literal notranslate"><span
>> class="pre">guest-shutdown</span></code> (Command)</a><a class="headerlink"
>> href="#qapidoc-31" title="Permalink to this heading"></a></h3>
>>     +<section id="qapidoc-30">
>>     +<h3><a class="toc-backref" href="#id13" role="doc-backlink"><code
>> class="docutils literal notranslate"><span
>> class="pre">guest-shutdown</span></code> (Command)</a><a class="headerlink"
>> href="#qapidoc-30" title="Permalink to this heading"></a></h3>
>>      <p>Initiate guest-activated shutdown.  Note: this is an asynchronous
>>      shutdown request, with no guarantee of successful shutdown.</p>
>>      <section id="qapidoc-28">
>>     @@ -502,22 +502,20 @@ shutdown request, with no guarantee of s
>>      </dd>
>>      </dl>
>>      </section>
>>     -<section id="qapidoc-29">
>>      <p>This command does NOT return a response on success.  Success
>>      condition is indicated by the VM exiting with a zero exit status or,
>>      when running with –no-shutdown, by issuing the query-status QMP
>>      command to confirm the VM status is “shutdown”.</p>
>>     -</section>
>>     -<section id="qapidoc-30">
>>     -<h4>Since<a class="headerlink" href="#qapidoc-30" title="Permalink
>> to this heading"></a></h4>
>>     +<section id="qapidoc-29">
>>     +<h4>Since<a class="headerlink" href="#qapidoc-29" title="Permalink
>> to this heading"></a></h4>
>>      <p>0.15.0</p>
>>      </section>
>>      </section>
>>
>> The id changes muddy the waters.  With them manually removed:
>>
>>      <section id="qapidoc-31">
>>      <h3><a class="toc-backref" href="#id13" role="doc-backlink"><code
>> class="docutils literal notranslate"><span
>> class="pre">guest-shutdown</span></code> (Command)</a><a class="headerlink"
>> href="#qapidoc-31" title="Permalink to this heading"></a></h3>
>>      <p>Initiate guest-activated shutdown.  Note: this is an asynchronous
>>      shutdown request, with no guarantee of successful shutdown.</p>
>>      <section id="qapidoc-28">
>>     @@ -502,22 +502,20 @@ shutdown request, with no guarantee of s
>>      </dd>
>>      </dl>
>>      </section>
>>     -<section id="qapidoc-29">
>>      <p>This command does NOT return a response on success.  Success
>>      condition is indicated by the VM exiting with a zero exit status or,
>>      when running with –no-shutdown, by issuing the query-status QMP
>>      command to confirm the VM status is “shutdown”.</p>
>>     -</section>
>>      <section id="qapidoc-30">
>>      <h4>Since<a class="headerlink" href="#qapidoc-30" title="Permalink
>> to this heading"></a></h4>
>>      <p>0.15.0</p>
>>      </section>
>>      </section>
>>
>> Makes no visual difference in my browser.
>>
>> Do these differences match your expectations?
>>
>
> Yep!
>
> It does change the output just a little, but Sphinx really doesn't like
> title-less sections.
>
> I thought the change looked fine, and I'm still planning on removing this
> old generator anyway, so...
>

Oh, pro tip: try using the xml builder before and after for a cleaner
comparison.

--js

>
Markus Armbruster July 1, 2024, 9:11 a.m. UTC | #4 
John Snow <jsnow@redhat.com> writes:

> On Fri, Jun 28, 2024, 3:55 AM Markus Armbruster <armbru@redhat.com> wrote:
>
>> John Snow <jsnow@redhat.com> writes:
>>
>> > Sphinx does not like sections without titles, because it wants to
>> > convert every section into a reference. When there is no title, it
>> > struggles to do this and transforms the tree inproperly.
>> >
>> > Depending on the rST used, this may result in an assertion error deep in
>> > the docutils HTMLWriter.
>> >
>> > (Observed when using ".. admonition:: Notes" under such a section - When
>> > this is transformed with its own <title> element, Sphinx is fooled into
>> > believing this title belongs to the section and incorrect mutates the
>> > docutils tree, leading to errors during rendering time.)
>> >
>> > When parsing an untagged section (free paragraphs), skip making a hollow
>> > section and instead append the parse results to the prior section.
>> >
>> > Many Bothans died to bring us this information.
>> >
>> > Signed-off-by: John Snow <jsnow@redhat.com>
>> > Acked-by: Markus Armbruster <armbru@redhat.com>
>>
>> Generated HTML changes, but the diff is hard to review due to id
>> attribute changes all over the place.
>>
>> Generated qemu-ga-ref.7 also changes:
>>
>>     diff -rup old/qemu-ga-ref.7 new/qemu-ga-ref.7
>>     --- old/qemu-ga-ref.7       2024-06-27 10:42:21.466096276 +0200
>>     +++ new/qemu-ga-ref.7       2024-06-27 10:45:36.502414099 +0200
>>     @@ -397,6 +397,7 @@ shutdown request, with no guarantee of s
>>      .B \fBmode\fP: \fBstring\fP (optional)
>>      \(dqhalt\(dq, \(dqpowerdown\(dq (default), or \(dqreboot\(dq
>>      .UNINDENT
>>     +.sp
>>      This command does NOT return a response on success.  Success
>>      condition is indicated by the VM exiting with a zero exit status or,
>>      when running with \-\-no\-shutdown, by issuing the query\-status QMP
>>     @@ -1348,6 +1349,7 @@ the new password entry string, base64 en
>>      .B \fBcrypted\fP: \fBboolean\fP
>>      true if password is already crypt()d, false if raw
>>      .UNINDENT
>>     +.sp
>>      If the \fBcrypted\fP flag is true, it is the caller\(aqs
>> responsibility to
>>      ensure the correct crypt() encryption scheme is used.  This command
>>      does not attempt to interpret or report on the encryption scheme.
>>
>> We add vertical space.  Visible when viewed with man.  Looks like an
>> improvement to me.
>>
>> Here's the first of these two spots in HTML:

[...]

>> The id changes muddy the waters.  With them manually removed:

[...]

>> Makes no visual difference in my browser.
>>
>> Do these differences match your expectations?
>>
>
> Yep!
>
> It does change the output just a little, but Sphinx really doesn't like
> title-less sections.
>
> I thought the change looked fine, and I'm still planning on removing this
> old generator anyway, so...

I'll add to the commit message

    The resulting output changes are basically invisible.

Thanks!
diff mbox series

Patch

diff --git a/docs/sphinx/qapidoc.py b/docs/sphinx/qapidoc.py
index f9683444b14..efcd84656fa 100644
--- a/docs/sphinx/qapidoc.py
+++ b/docs/sphinx/qapidoc.py
@@ -272,14 +272,20 @@  def _nodes_for_sections(self, doc):
             if section.tag and section.tag == 'TODO':
                 # Hide TODO: sections
                 continue
+
+            if not section.tag:
+                # Sphinx cannot handle sectionless titles;
+                # Instead, just append the results to the prior section.
+                container = nodes.container()
+                self._parse_text_into_node(section.text, container)
+                nodelist += container.children
+                continue
+
             snode = self._make_section(section.tag)
-            if section.tag and section.tag.startswith('Example'):
+            if section.tag.startswith('Example'):
                 snode += self._nodes_for_example(dedent(section.text))
             else:
-                self._parse_text_into_node(
-                    dedent(section.text) if section.tag else section.text,
-                    snode,
-                )
+                self._parse_text_into_node(dedent(section.text), snode)
             nodelist.append(snode)
         return nodelist