Asciidoctor: Support for "open in widget" (#627)

* Asciidoctor: Support for "open in widget"

This'll take a while to describe properly. Let's start with the goal
first, talk about the AsciiDoc implementation, then move on to the
Asciidoctor implementation, then talk about why we need a compatibility
layer, then describe the compatibity layer, and finally round out this
book of a commit message by describing some of the more esoteric usages
of this that this change currently supports and our plan for dropping
this support in the future.

== The goal

Certain snippets in Elastic's docs are special and we'd like to decorate
them with buttons. These buttons allow opening the snippets in developer
tools or transforming them into `cURL` commands. I'm calling all of the
stuff that makes these buttons an "open in widget".

== AsciiDoc implementation

Because of AsciiDoc's fairly limited ability to customize it we
triggered these snippets by adding special comments after the code
blocks that we'd like to decorate that look like this:
```
[source,js]
----
GET /
----
// CONSOLE   <----- This is the comment
```

We customize AsciiDoc to emit all comments as `<remark>` elements in the
generated xml and then use custom xslt to recognize remarks that look
like `<remark>CONSOLE</remark>` to trigger the widget.

These buttons need the contents of the snippet, `GET /` in the example
above, to be accessible in a file. We implement this by post-processing
the generated html in the perl code to extra these snippets to files,
rewriting the html that generates the widget to point to the file.

== Asciidoctor implementation

Asciidoctor feels strongly that comments shouldn't have semantic
meaning. I'm on board with that. So, to trigger the widget in
Asciidoctor you use:
```
[source,console]
----
GET /
----
```

Note that the language, which was `js` in AsciiDoc is now `console`.
This language is what triggers the widget. This feels good to me because
all snippets that are in the "console" language really do want this
decoration. And because the snippets really *aren't* javascript. They
just often have json in them.

We recognize these `:listing` blocks using a treepreprocessor and built
the snippet file when processing the Asciidoctor AST. We also rewrite
the docbook that is generated by these blocks to contain a link to the
extract snippet. Finally, we also use custom xslt to extract that link
and render the widget. The xslt is less hairy than the one used to find
the `<remark>`s.

== Why we need compatibility

It'd be fairly simple to change `// CONSOLE` to `[source,console]` on a
page by page basis, but we have thousands of uses of `// CONSOLE` across
dozens of books and dozens of branches. In addition, AsciiDoc doesn't
understand `[source,console]` which'd make changing these a thing that
you'd have to do at the same time as you switched the book from AsciiDoc
to Asciidoctor. Beyond *that*, Elasticsearch automatically extracts
snippets with the `// CONSOLE` comment and turns them into tests.

There are just too many moving parts for a hard cut over from
`// CONSOLE` to `[source,console]`. So I built a compatiblity layer in
Asciidoctor

== How the compatibilty layer works

We use two customizations in Asciidoctor to make the compatibility layer
go, one that recognizes comments shaped like `// CONSOLE` and turns them
into a literal `// CONSOLE` using the `pass` macro like so:
`pass:[// CONSOLE]`. The next one picks up source blocks that are
immediately followed by `pass:[// CONSOLE]` and switches the language to
`console`.

Something like this *begs* for away to tell the user "you have 1232
warnings that you have to fix". Like linting but for your docs. I'll be
thinking more about this in the coming weeks. But for now there is not
warning when you use the compatibility layer.

== Esoteric forms of the "open in widget"

Kibana's docs have snippets like:
```
[source,js]
--------------------------------------------------
GET api/logstash/pipeline/hello-world
--------------------------------------------------
// KIBANA
```

Note the `// KIBANA`. This is *like* `// CONSOLE` but it is for snippets
that should be sent to Kibana's API instead of Elasticsearch's. It is
debateable if `kibana` is really a different language than `console`,
but it is at least a different dialect. Either way, Asciidoctor triggers
the "open in widget" for these snippets with `[source,kibana]`.

Older versions of the documentation have snippets like this:
```
[source,js]
--------------------------------------------------
GET /
--------------------------------------------------
// AUTOSENSE
```

These open the command in an older tool named "sense" instead of the
developer console. The developer console "grew out of" sense but hasn't
been called that for a long time. In any case, it exists for
compatibility with old books.

The Definitive Guide has snippets that look like:
```
[source,js]
--------------------------------------------------
GET /
--------------------------------------------------
// SENSE:a/path/to/some.json
```

These are snippets that render as `GET /` but when you open them in
"sense" they have the contents of `a/path/to/some.json` which is
supposed to be similar. This isn't widely used and I personally think it
is super confusing, at least the way that we've implemented it now. So I
log a warning whenever you try to do this which should prevent folks
from doing it accidentally.

== Follow ups

This work begs for at least two follow up changes:
1. A warnings management system so old books can continue to use the
backwards compatibility features but new books will be forced to use
native Asciidoctor features.
2. More in depth integration testing. Right now we build
README.asciidoc with AsciiDoc and Asciidoctor which is great, but it is
difficult to assert that the results are "close enough" with the tools
that we have. We can do better and this change makes it obvious that we
must do better.

* Word

* Clean up after merge

* Missing command

* speeling

* Shift to error

And shift the warning for being hard to read to after the copy.

* Test cleanups

* Style

Author: nik9000

README.asciidoc

@@ -1014,12 +1014,26 @@ footnote to a particular line of code:
 === View in Console
 
 Code blocks can be followed by a "View in Console" link which, when clicked,
-will open the code snippet in Console.  The snippet can either be taken directly
-from the code block (`CONSOLE`), or be a link to a custom snippet.
+will open the code snippet in Console. There are two ways to do this, the
+"AsciiDoc" way and the "Asciidoctor" way. The "AsciiDoc" way is preferred in
+the Elaticsearch repository because it can recognize it to make tests. The
+"Asciidoctor" way is preferred in other books, but only if they are built with
+"Asciidoctor". Try it first and if it works then use it. Otherwise, use the
+"AsciiDoc" way.
 
-.Code block with CONSOLE link
+////
+
+The tricks that we pull to make ascidoctor support // CONSOLE are windy
+and force us to add subs=+macros when we render an asciidoc snippet.
+We *don't* require that for normal snippets, just those that contain
+asciidoc.
+
+////
+
+.Code block with CONSOLE link (AsciiDoc way)
 ==================================
-[source,asciidoc]
+ifdef::asciidoctor[[source,asciidoc,subs=+macros]]
+ifndef::asciidoctor[[source,asciidoc]]
 --
 [source,js]
 ----------------------------------
@@ -1035,6 +1049,24 @@ GET /_search
 ==================================
 <1> The `// CONSOLE` line must follow immediately after the code block, before any callouts.
 
+.Code block with CONSOLE link (Asciidoctor way)
+==================================
+[source,asciidoc]
+--
+[source,console]
+----------------------------------
+GET /_search
+{
+    "query": "foo bar" \<1>
+}
+----------------------------------
+
+\<1> Here's the explanation
+--
+==================================
+
+Both render as:
+
 [source,js]
 ----------------------------------
 GET /_search
@@ -1060,38 +1092,6 @@ The local web browser can be stopped with `Ctrl-C`.
 
 ================================
 
-==== Custom Console snippets
-
-Sometimes you will want to show a small amount of code in the code block, but
-to provide a full recreation in the Console snippet.  In this case, you need to:
-
-* Save the snippet file in the `./snippets/` directory in the root docs directory.
-* Under the code block, specify the name of the snippet file with
-+
-    // CONSOLE: path/to/snippet.json
-
-For instance, to add a custom snippet to the file `./one/two/three.asciidoc`, save the snippet
-to `./snippets/one/two/three/example_1.json`, then add the `CONSOLE` link below the code block:
-
-.Code block with custom CONSOLE link
-==================================
-[source,asciidoc]
---
-[source,js]
-----------------------------------
-GET /_search
-{
-    "query": "foo bar" \<1>
-}
-----------------------------------
-// CONSOLE:one/two/three/example_1.json <1>
-
-\<1> Here's the explanation
---
-<1> The path should not contain the initial `snippets` directory
-==================================
-
-
 [[admon-blocks]]
 === Admonition blocks
 

integtest/Makefile

@@ -63,14 +63,15 @@ experimental_expected_files: /tmp/experimental_asciidoc
 %_same_files: /tmp/%_asciidoc /tmp/%_asciidoctor
 	diff \
 		<(cd /tmp/$*_asciidoc    && find * -type f | sort \
-			| grep -v snippets/blocks \
+			| grep -v 'snippets/' \
 		) \
-		<(cd /tmp/$*_asciidoctor && find * -type f | sort)
-	# The grep -v below are for known issues with asciidoctor
-	for file in $$(cd /tmp/$*_asciidoc && find * -type f -name '*.html' \
-			| grep -v 'blocks'); do \
+		<(cd /tmp/$*_asciidoctor && find * -type f | sort \
+			| grep -v 'snippets/' \
+		)
+	for file in $$(cd /tmp/$*_asciidoc && find * -type f -name '*.html'); do \
 		./html_diff /tmp/$*_asciidoc/$$file /tmp/$*_asciidoctor/$$file; \
 	done
+	# TODO validate the snippets have the same contents even if the files aren't the same
 
 # Build the docs into the target
 define BD=

integtest/html_diff

@@ -35,10 +35,13 @@ def normalize_html(html):
     # is between words. They are actively nice to have but asciidoc doesn't
     # make them.
     html = html.replace('\u2014\u200b', '\u2014')
-    # Temporary workaround for known issues
-    html = re.sub(
-        r'(?m)^\s+<div class="console_widget" data-snippet="[^"]+">'
-        r'\s+</div>\n', '', html)
+    # We intentionally changed lang-js to lang-console because in Asciidoctor
+    # because that is more accurate
+    html = html.replace('"programlisting prettyprint lang-js"',
+                        '"programlisting prettyprint lang-console"')
+    # The URL for the console snippets has changed
+    html = re.sub(r'data-snippet="[^"]+"', 'data-snippet="snippet"', html)
+    # Temporary work around for known issue
     html = html.replace('\\&lt;1&gt;', '&lt;1&gt;')
     return html
 

lib/ES/Template.pm

@@ -47,6 +47,7 @@ sub apply {
     my $self = shift;
     my $dir  = shift;
     my $lang = shift || die "No lang specified";
+    my $asciidoctor = shift;
 
     my $map = $self->_map;
 
@@ -61,7 +62,7 @@ sub apply {
         $contents =~ s/\s*$/\n/;
 
         # Extract AUTOSENSE snippets
-        $contents = $self->_autosense_snippets( $file, $contents );
+        $contents = $self->_autosense_snippets( $file, $contents ) unless $asciidoctor;
 
         # Fill in template
         my @parts  = @{ $self->_parts };

lib/ES/Util.pm

@@ -154,7 +154,7 @@ sub build_chunked {
     my ($chunk_dir) = grep { -d and /\.chunked$/ } $dest->children
         or die "Couldn't find chunk dir in <$dest>";
 
-    finish_build( $index->parent, $chunk_dir, $lang );
+    finish_build( $index->parent, $chunk_dir, $lang, $asciidoctor );
     extract_toc_from_index($chunk_dir);
     for ( $chunk_dir->children ) {
         run( 'mv', $_, $dest );
@@ -287,7 +287,7 @@ sub build_single {
             or die "Couldn't rename <$src> to <index.html>: $!";
     }
 
-    finish_build( $index->parent, $dest, $lang );
+    finish_build( $index->parent, $dest, $lang, $asciidoctor );
 }
 
 #===================================
@@ -369,10 +369,10 @@ sub build_pdf {
 #===================================
 sub finish_build {
 #===================================
-    my ( $source, $dest, $lang ) = @_;
+    my ( $source, $dest, $lang, $asciidoctor ) = @_;
 
     # Apply template to HTML files
-    $Opts->{template}->apply( $dest, $lang );
+    $Opts->{template}->apply( $dest, $lang, $asciidoctor );
 
     my $snippets_dest = $dest->subdir('snippets');
     my $snippets_src;

resources/asciidoctor/lib/elastic_compat_preprocessor/extension.rb

@@ -89,12 +89,31 @@
 # Because Asciidoc permits these mismatches but asciidoctor does not. We'll
 # emit a warning because, permitted or not, they are bad style.
 #
+# With the help of ElasticCompatTreeProcessor turns
+#   [source,js]
+#   ----
+#   foo
+#   ----
+#   // CONSOLE
+#
+# Into
+#   [source,console]
+#   ----
+#   foo
+#   ----
+# Because Elastic has thousands of these constructs but Asciidoctor feels
+# strongly that comments should not convey meaning. This is a totally
+# reasonable stance and we should migrate away from these comments in new
+# docs when it is possible. But for now we have to support the comments as
+# well.
+#
 class ElasticCompatPreprocessor < Asciidoctor::Extensions::Preprocessor
   include Asciidoctor::Logging
 
   INCLUDE_TAGGED_DIRECTIVE_RX = /^include-tagged::([^\[][^\[]*)\[(#{Asciidoctor::CC_ANY}+)?\]$/.freeze
   SOURCE_WITH_SUBS_RX = /^\["source", ?"[^"]+", ?subs="(#{Asciidoctor::CC_ANY}+)"\]$/.freeze
   CODE_BLOCK_RX = /^-----*$/.freeze
+  SNIPPET_RX = %r{//\s*(?:AUTOSENSE|KIBANA|CONSOLE|SENSE:[^\n<]+)}.freeze
 
   def process(_document, reader)
     reader.instance_variable_set :@in_attribute_only_block, false
@@ -142,13 +161,22 @@ def reader.process_line(line)
             @code_block_start = line
           end
         end
+
         supported = 'added|beta|coming|deprecated|experimental'
         # First convert the "block" version of these macros. We convert them
         # to block macros because they are at the start of the line....
         line&.gsub!(/^(#{supported})\[([^\]]*)\]/, '\1::[\2]')
         # Then convert the "inline" version of these macros. We convert them
         # to inline macros because they are *not* at the start of the line....
         line&.gsub!(/(#{supported})\[([^\]]*)\]/, '\1:[\2]')
+
+        # Transform Elastic's traditional comment based marking for
+        # AUTOSENSE/KIBANA/CONSOLE snippets into a marker that we can pick
+        # up during tree processing to turn the snippet into a marked up
+        # CONSOLE snippet. Asciidoctor really doesn't recommend this sort of
+        # thing but we have thousands of them and it'll take us some time to
+        # stop doing it.
+        line&.gsub!(SNIPPET_RX, 'pass:[\0]')
       end
     end
     reader

resources/asciidoctor/lib/elastic_compat_tree_processor/extension.rb

@@ -23,14 +23,62 @@
 #   <1> The count of categories that were matched
 #   <2> The categories retrieved
 #
+# Turns
+#   [source,js]
+#   --------------------------------------------------
+#   GET / <1>
+#   --------------------------------------------------
+#   pass:[// CONSOLE]
+#   <1> The count of categories that were matched
+#   <2> The categories retrieved
+#
+# Into
+#   [source,console]
+#   --------------------------------------------------
+#   GET / <1>
+#   --------------------------------------------------
+#   <1> The count of categories that were matched
+#   <2> The categories retrieved
+#
 class ElasticCompatTreeProcessor < TreeProcessorScaffold
+  include Asciidoctor::Logging
+
   def process_block(block)
-    if block.context == :listing && block.style == "source" &&
-          block.subs.include?(:specialcharacters) == false
-      # callouts have to come *after* special characters
-      had_callouts = block.subs.delete(:callouts)
-      block.subs << :specialcharacters
-      block.subs << :callouts if had_callouts
-    end
+    return unless block.context == :listing && block.style == 'source'
+
+    process_subs block
+    process_lang_override block
+  end
+
+  def process_subs(block)
+    return if block.subs.include? :specialcharacters
+
+    # callouts have to come *after* special characters
+    had_callouts = block.subs.delete(:callouts)
+    block.subs << :specialcharacters
+    block.subs << :callouts if had_callouts
+  end
+
+  LANG_MAPPING = {
+    'AUTOSENSE' => 'sense',
+    'CONSOLE' => 'console',
+    'KIBANA' => 'kibana',
+    'SENSE' => 'sense',
+  }.freeze
+
+  def process_lang_override(block)
+    next_block = block.next_adjacent_block
+    return unless next_block && next_block.context == :paragraph
+    return unless next_block.source =~ %r{pass:\[//\s*([^:\]]+)(?::\s*([^\]]+))?\]}
+
+    lang = LANG_MAPPING[$1]
+    snippet = $2
+    return unless lang # Not a language we handle
+
+    block.set_attr 'language', lang
+    block.set_attr 'snippet', snippet
+
+    block.parent.blocks.delete next_block
+    block.parent.reindex_sections
   end
 end

resources/asciidoctor/lib/extensions.rb

@@ -8,6 +8,7 @@
 require_relative 'elastic_compat_tree_processor/extension'
 require_relative 'elastic_compat_preprocessor/extension'
 require_relative 'elastic_include_tagged/extension'
+require_relative 'open_in_widget/extension'
 
 Asciidoctor::Extensions.register CareAdmonition
 Asciidoctor::Extensions.register ChangeAdmonition
@@ -20,5 +21,6 @@
   treeprocessor CopyImages::CopyImages
   treeprocessor EditMe
   treeprocessor ElasticCompatTreeProcessor
+  treeprocessor OpenInWidget
   include_processor ElasticIncludeTagged
 end