Skip to content

Commit b3fd445

Browse files
nik9000nik9000
authored
Improvements for book sources (#1250)
This adds two features that are sort of interlinked in their implementation but not so much in their use. We want them both though, I think: 1. Allows books to explictly depend on files in the docs repo so they are rebuilt when those files are changed. 2. Create attributes for the "root" of each repository. This should make a lot of the `../../../../` paths unneeded. So the second feature allows you to write includes like: ``` include::{elasticsearch-root}/docs/reference/glossary.asciidoc[] ``` instead of: ``` include::{docdir}/../../../../elasticsearch/docs/reference/glossary.asciidoc[] ``` The first feature combined with the second feature lets you write this to include the shared files: ``` include::{docs-root}/shared/versions/stack/{source_branch}.asciidoc[] include::{docs-root}/shared/attributes.asciidoc[] ``` instead of: ``` include::{asciidoc-dir}/../../shared/versions/stack/{source_branch}.asciidoc[] include::{asciidoc-dir}/../../shared/attributes.asciidoc[] ``` Which is nice. In addition, the first feature lets you write: ``` - repo: docs path: shared/attributes.asciidoc - repo: docs path: shared/versions/stack/{version}.asciidoc ``` in `conf.yaml` to cause a book to be rebuilt if either: * `attributes.asciidoc` changes * `shared/versions/stack/master.asciidoc` changes. Assuming we're building the master branch of the book. It'll be `7.x.asciidoc` if we're building `7.x`, etc. You can also declare a dependency on `shared/versions/stack/current.asciidoc` and we'll cause a rebuild if either `current.asciidoc` changes to point to a new file *or* the file that it points to changes.
1 parent 8dfd388 commit b3fd445

16 files changed

+655
-126
lines changed

‎README.asciidoc

+55-6
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
= Docs HOWTO
22

3-
include::{asciidoc-dir}/../../shared/versions/stack/current.asciidoc[]
4-
include::{asciidoc-dir}/../../shared/attributes.asciidoc[]
3+
include::{docs-root}/shared/versions/stack/current.asciidoc[]
4+
include::{docs-root}/shared/attributes.asciidoc[]
55

66
////
77
This file contains the sequence `_` to escape from around Elastic's
@@ -760,17 +760,43 @@ attributes file] to resolve links:
760760
= My Book Title
761761
762762
# Use this if your repo is versioned with the Elastic stack:
763-
\include::{asciidoc-dir}/../../shared/versions/stack/{source_branch}.asciidoc[]
763+
\include::{docs-root}/shared/versions/stack/{source_branch}.asciidoc[]
764764
# Or use this to alway point to the most recent version of the stack
765-
\include::{asciidoc-dir}/../../shared/versions/stack/current.asciidoc[]
766-
\include::{asciidoc-dir}/../../shared/attributes.asciidoc[]
765+
\include::{docs-root}/shared/versions/stack/current.asciidoc[]
766+
# Either way, you'll want to include a reference to the attributes file
767+
# which builds the links from the versions.
768+
\include::{docs-root}/shared/attributes.asciidoc[]
767769
768770
Here is a link to the {ref}/search.html[search page]
769771
----------------------------------
770772
771773
Here is a link to the {ref}/search.html[search page]
772774
==================================
773775

776+
Books that use the attributes file should declare a dependency on it in
777+
https://github.com/elastic/docs/blob/master/conf.yaml[`conf.yaml`] like this:
778+
779+
[source,yaml]
780+
----
781+
-
782+
repo: docs
783+
path: shared/versions/stack/{version}.asciidoc
784+
-
785+
repo: docs
786+
path: shared/attributes.asciidoc
787+
----
788+
789+
or
790+
791+
[source,yaml]
792+
----
793+
-
794+
repo: docs
795+
path: shared/versions/stack/current.asciidoc
796+
-
797+
repo: docs
798+
path: shared/attributes.asciidoc
799+
----
774800

775801
For books that don't use the
776802
https://github.com/elastic/docs/blob/master/shared/attributes.asciidoc[shared
@@ -1253,12 +1279,35 @@ include them where appropriate:
12531279
[source,asciidoc]
12541280
----------------------------------
12551281
\include::myfolder/mydoc.asciidoc[]
1256-
12571282
----------------------------------
12581283

12591284
Paths are relative to the file which
12601285
contains the `include` statement.
12611286

1287+
[[cross-repo-includes]]
1288+
=== Across repositories
1289+
1290+
If you have to include files in a different repository then use its `-root`
1291+
attribute to locate the files:
1292+
1293+
[source,asciidoc]
1294+
----------------------------------
1295+
\include::{elasticsearch-root}/docs/foo.asciidoc[]
1296+
----------------------------------
1297+
1298+
Books that reference another repository should register that reference in
1299+
https://github.com/elastic/docs/blob/master/conf.yaml[`conf.yaml`].
1300+
1301+
[source,yaml]
1302+
----
1303+
-
1304+
repo: elasticsearch
1305+
path: docs/foo.asciidoc
1306+
----
1307+
1308+
The path should be as specific as possible because we skip rebuilding books if
1309+
changes to the referenced repository don't change the referenced path.
1310+
12621311
[[changes]]
12631312
== Additions and deprecations
12641313

‎build_docs.pl

+66-35
Original file line numberDiff line numberDiff line change
@@ -46,6 +46,7 @@ BEGIN
4646
use Sys::Hostname;
4747

4848
use ES::BranchTracker();
49+
use ES::DocsRepo();
4950
use ES::Repo();
5051
use ES::Book();
5152
use ES::TargetRepo();
@@ -87,7 +88,8 @@ sub build_local {
8788
$Opts->{resource}
8889
= [ map { dir($_)->absolute($Old_Pwd) } @{ $Opts->{resource} || [] } ];
8990

90-
_guess_opts_from_file($index);
91+
_guess_opts( $index );
92+
$Opts->{roots}{docs} = '/docs_build' unless $Opts->{roots}{docs};
9193

9294
my @alternatives;
9395
if ( $Opts->{alternatives} ) {
@@ -131,30 +133,41 @@ sub build_local {
131133
}
132134

133135
#===================================
134-
sub _guess_opts_from_file {
136+
sub _guess_opts {
135137
#===================================
136138
my $index = shift;
137139

138-
my %edit_urls = ();
139-
my $doc_toplevel = _find_toplevel($index->parent);
140-
unless ( $doc_toplevel ) {
140+
$Opts->{edit_urls} = {};
141+
$Opts->{roots} = {};
142+
my $toplevel = _find_toplevel($index->parent);
143+
unless ( $toplevel ) {
141144
$Opts->{root_dir} = $index->parent;
142145
$Opts->{branch} = 'master';
143146
# If we can't find the edit url for the document then we're never
144147
# going to find it for anyone.
145148
return;
146149
}
147-
$Opts->{root_dir} = $doc_toplevel;
148-
my $edit_url = _guess_edit_url($doc_toplevel);
149-
$edit_urls{ $doc_toplevel } = $edit_url if $edit_url;
150+
my $remote = _pick_best_remote( $toplevel );
151+
my $branch = _guess_branch( $toplevel );
152+
my $repo_name = _guess_repo_name( $remote );
153+
$Opts->{branch} = $branch;
154+
$Opts->{root_dir} = $toplevel;
155+
$Opts->{roots}{ $repo_name } = $toplevel;
156+
$Opts->{edit_urls}{ $toplevel } = ES::Repo::edit_url_for_url_and_branch(
157+
$remote || 'unknown', $branch
158+
);
150159
for my $resource ( @{ $Opts->{resource} } ) {
151-
my $resource_toplevel = _find_toplevel($resource);
152-
next unless $resource_toplevel;
153-
154-
my $resource_edit_url = _guess_edit_url($resource_toplevel);
155-
$edit_urls{ $resource_toplevel } = $resource_edit_url if $resource_edit_url;
160+
$toplevel = _find_toplevel( $resource );
161+
next unless $toplevel;
162+
163+
$remote = _pick_best_remote( $toplevel );
164+
$branch = _guess_branch( $toplevel );
165+
$repo_name = _guess_repo_name( $remote );
166+
$Opts->{roots}{ $repo_name } = $toplevel;
167+
$Opts->{edit_urls}{ $toplevel } = ES::Repo::edit_url_for_url_and_branch(
168+
$remote || 'unknown', $branch
169+
);
156170
}
157-
$Opts->{edit_urls} = { %edit_urls };
158171
}
159172

160173
#===================================
@@ -171,34 +184,31 @@ sub _find_toplevel {
171184
}
172185

173186
#===================================
174-
sub _guess_edit_url {
187+
sub _pick_best_remote {
175188
#===================================
176189
my $toplevel = shift;
177190

178191
local $ENV{GIT_DIR} = dir($toplevel)->subdir('.git');
179192
my $remotes = eval { run qw(git remote -v) } || '';
180-
my $remote;
181193
if ($remotes =~ m|\s+(\S+[/:]elastic(?:search-cn)?/\S+)|) {
182-
$remote = $1;
183194
# We prefer either an elastic or elasticsearch-cn organization. All
184195
# but two books are in elastic but elasticsearch-cn is special.
185-
} else {
186-
say "Couldn't find edit url because there isn't an Elastic remote";
187-
if ($remotes =~ m|\s+(\S+[/:]\S+/\S+)|) {
188-
$remote = $1;
189-
} else {
190-
$remote = 'unknown';
191-
}
196+
return $1;
197+
}
198+
say "Couldn't an Elastic remote for $toplevel";
199+
if ($remotes =~ m|\s+(\S+[/:]\S+/\S+)|) {
200+
return $1;
192201
}
193-
my $branch = eval { run qw(git rev-parse --abbrev-ref HEAD) } || 'master';
194-
$Opts->{branch} = _guess_branch( $branch ) unless $Opts->{branch};
195-
return ES::Repo::edit_url_for_url_and_branch($remote, $branch);
202+
return 0;
196203
}
197204

198205
#===================================
199206
sub _guess_branch {
200207
#===================================
201-
my $real_branch = shift;
208+
my $toplevel = shift;
209+
210+
local $ENV{GIT_DIR} = dir($toplevel)->subdir('.git');
211+
my $real_branch = eval { run qw(git rev-parse --abbrev-ref HEAD) } || 'master';
202212

203213
# Detects common branch patterns like:
204214
# 7.x
@@ -218,6 +228,17 @@ sub _guess_branch {
218228
return 'master';
219229
}
220230

231+
#===================================
232+
sub _guess_repo_name {
233+
#===================================
234+
my ( $remote ) = @_;
235+
236+
$remote = dir( $remote )->basename;
237+
$remote =~ s/\.git$//;
238+
239+
return $remote;
240+
}
241+
221242
#===================================
222243
sub build_local_pdf {
223244
#===================================
@@ -265,7 +286,9 @@ sub build_all {
265286
}
266287
else {
267288
say "Building docs";
268-
build_entries( $raw_build_dir, $build_dir, $temp_dir, $toc, @$contents );
289+
build_entries(
290+
$raw_build_dir, $build_dir, $temp_dir, $toc, $tracker, @$contents
291+
);
269292

270293
say "Writing main TOC";
271294
$toc->write( $raw_build_dir, $build_dir, $temp_dir, 0 );
@@ -288,7 +311,7 @@ sub build_all {
288311
say "Checking links";
289312
check_links($build_dir);
290313
}
291-
$tracker->prune_out_of_date( @$contents );
314+
$tracker->prune_out_of_date;
292315
push_changes( $build_dir, $target_repo, $tracker ) if $Opts->{push};
293316
serve_local_preview( $build_dir, $redirects, 0, 0 ) if $Opts->{open};
294317

@@ -383,7 +406,7 @@ sub check_kibana_links {
383406
#===================================
384407
sub build_entries {
385408
#===================================
386-
my ( $raw_build, $build, $temp_dir, $toc, @entries ) = @_;
409+
my ( $raw_build, $build, $temp_dir, $toc, $tracker, @entries ) = @_;
387410

388411
while ( my $entry = shift @entries ) {
389412
my $title = $entry->{title}
@@ -395,8 +418,8 @@ sub build_entries {
395418
my $sub_build = $build->subdir($base_dir);
396419
my $section_toc = build_entries(
397420
$raw_sub_build, $sub_build, $temp_dir,
398-
ES::Toc->new( $title, $entry->{lang} ),
399-
@$sections
421+
ES::Toc->new( $title, $entry->{lang}, ),
422+
$tracker, @$sections
400423
);
401424
if ($base_dir) {
402425
$section_toc->write( $raw_sub_build, $sub_build, $temp_dir );
@@ -418,6 +441,7 @@ sub build_entries {
418441
%$entry
419442
);
420443
$toc->add_entry( $book->build( $Opts->{rebuild} ) );
444+
$tracker->allowed_book( $book );
421445
}
422446

423447
return $toc;
@@ -531,7 +555,7 @@ sub init_target_repo {
531555
my ( $repos_dir, $temp_dir, $reference_dir ) = @_;
532556

533557
my $target_repo = ES::TargetRepo->new(
534-
dir => $repos_dir,
558+
git_dir => $repos_dir->subdir('target_repo.git'),
535559
user => $Opts->{user},
536560
url => $Opts->{target_repo},
537561
reference => $reference_dir,
@@ -571,13 +595,15 @@ sub init_repos {
571595
$pm->finish;
572596
}
573597
for my $name (@repo_names) {
598+
next if $name eq 'docs';
599+
574600
my $url = $conf->{$name};
575601
# We always use ssh-style urls regardless of conf.yaml so we can use
576602
# our ssh key for the cloning.
577603
$url =~ s|https://([^/]+)/|git\@$1:|;
578604
my $repo = ES::Repo->new(
579605
name => $name,
580-
dir => $repos_dir,
606+
git_dir => $repos_dir->subdir("$name.git"),
581607
tracker => $tracker,
582608
user => $Opts->{user},
583609
url => $url,
@@ -613,6 +639,11 @@ sub init_repos {
613639
say "Removing old repo <" . $dir->basename . ">";
614640
$dir->rmtree;
615641
}
642+
643+
# Setup the docs repo
644+
# We support configuring the remote for the docs repo for testing
645+
ES::DocsRepo->new( $tracker, $conf->{docs} || '/docs_build' );
646+
616647
return $tracker;
617648
}
618649

‎conf.yaml

+6
Original file line numberDiff line numberDiff line change
@@ -109,6 +109,12 @@ contents:
109109
-
110110
repo: elasticsearch
111111
path: docs/reference
112+
-
113+
repo: docs
114+
path: shared/attributes.asciidoc
115+
-
116+
repo: docs
117+
path: shared/versions/stack/{version}.asciidoc
112118
-
113119
title: Stack Overview
114120
prefix: en/elastic-stack-overview

0 commit comments

Comments
 (0)