generated CRD reference docs: self-hosting crds.dev

[fhennig]

Context

Currently our documentation does not contain any complete CRD reference. For Spark there is a manually maintained page. Such a reference would list all the CRD config options and what they are for, how they relate to other options, what the default is (if any) and if it is mandatory. This is an essential part of the documentation and completements tutorials, guides and conceptual page

Objective

The reference documentation for all our CRDs needs to be generated (maintaining it by hand is not feasible) and hosted. Whether this can be integrated into the existing Antora docs or whether it will be separate is not so important for now.

We need to control the generation and hosting though.

Ideally the docs would be static generated files.

Evaluated options so far

Various generators

#238. We found that a lot of the tools to generate docs were not well maintained or didn't work for Rust code.

crds.dev

@sbernauer found https://github.com/crdsdev/doc , which generates docs, i.e. https://doc.crds.dev/github.com/stackabletech/[email protected]

• 192 GitHub stars.
• Last significant dev activity 2 years ago
• no documentation. Especially no docs on how to build it or host it.
• Hosted service, looks like it requires storage or a backend DB - not an ideal fit for us, it would be better if we got static files to host.
• Generated docs look good and useful!

Next Steps

crds.dev

• further evaluate crds.dev and explore/spike different options of hosting/running it.
  • should we host it? It is an "active" service with storage, we need a different hoster that goes beyond static sites.
  • can we somehow turn it into a site generator, and pre-generate the docs for our CRDs? This could either be generated adoc pages for antora, or just pregenerated HTML that we can host, possibly on a different subdomain (i.e. apidocs.stackable.tech).
• set it up
• optional: style it with Stackable branding

build custom generator

Should we look into building our own generator? Would be interesting to spike this.

Upsides: we get more info in the Rust code than in the CRD: We know that there are common structs (i.e. S3Bucket) and we can generate docs for these structs specifically.

It would be cool to have such an open source tool for the community

Downside: probably a lot of work/more work than we think.

[NickLarsenNZ]

There is also https://www.manifests.io/ which is just for the well known Kubernetes resources, but could be massaged to work how we wanted if we couldn't find a suitable alternative.

source code

[fhennig]

Interesting! Thanks for posting

[fhennig]

Update

I managed to turn the crds.dev docs code into a tool to statically generate docs for a predefined list of repositories and tags.
Now the question is, how to proceed?

How to host

There are two ways to do it, either we host the generated pages with minor styling changes on a separate domain, or we generate something that we can integrate into our Antora docs.

I think, hosting it under a separate subdomain is probably the better option. We have to do very little changes to the generated pages, which is easier for us. The Antora menu structure and table of contents doesn't make sense for the reference docs. It's fine to have a separate subdomain for this different type of documentation, and it's not uncommon. Actually, these code-level docs are often separated out. Also, it is not clear to me how we would handle the different API versions of the CRDs in Antora. So I think, Antora is just not the right tool for API docs.

Next steps

A brainstormed list of things to do next

• clean up the code and document the repo a bit more
• clean up styling, add a little bit of the stackable brand colors etc.
• Add a better menu (not sure what it should look like yet)
• Come up with a plan to deploy it to a new subdomain (what should the domain be?) and how we integrate it into the release process.

[fhennig]

Update: It's now deployed

The docs are now deployed at: https://crds.stackable.tech/

The source for the template and which repos are generated are located in this repo: https://github.com/stackabletech/crddocs . The generator is included as a submodule, it is located in this repo: https://github.com/stackabletech/crddocs-generator

Netlify is used to deploy. Whenever a commit to main is pushed into the crddocs repo, the docs will be rebuilt.

[NickLarsenNZ]

Nice!

Just a note, the dark-mode theme seems to work on the first page, but then it's light-mode for the rest.

---

Edit: after browsing back to the main page, it is also in light-mode.

[fhennig]

Yeah I removed light mode because it was implemented with a cookie that the server was reading, and that didn't work well with static site generating. Sorry if you liked dark mode! 😬

[fhennig]

Update

• Some CRDs didn't get generated, because the validator flagged them as not valid. I've disabled validation, but the build script still prints "WARNING" for these, so we should look into them. The docs however look fine.
• the main branch docs are now displayed too, under the 'nightly' version
• I've added a github action to rebuild the docs monday to friday at night.
• The site now also has a footer

[fhennig]

Update

• I've made an operator-rs 0.57.0 release that adds doc link templating.
• druid PR for more docs is up: More CRD docs! druid-operator#494

Next steps

• I would like to get the new landing page merged too: Landing page and new operator overview page #492
• all the operators need to be documented better -> can be a follow-up ticket. they all should use operator-rs 0.57
• maybe a CRD docs style guide
• The CRD docs should also be linked from the individual operator docs in the antora sidebar nav.

We're also waiting for kube-rs 0.88 for our doc attribute to write top-level doc strings for the CRDs. Also this PR: GREsau/schemars#251

[fhennig]

New druid docs: https://crds.stackable.tech/stackabletech/druid-operator/nightly/druid.stackable.tech/druidcluster/v1alpha1/

[fhennig]

I consider this ticket done now.

I've made a new one to polish the actual documentation next: #500