Defining a text bloc for group_for_docs sections

[lud]

Hello,

I would like to know if it is currently possible to inject some text in the generated docs below the section headers that are generated from groups_for_docs.

If not, is this something you'd consider adding?

Thank you.

[josevalim]

Just to make sure we are on the same page, can you send a screenshot showing where you'd want to add the text?

[garazdawi]

If I understand the request correctly, this is something that we would also like to have in our docs. That is the possibility to add some text below this heading: https://www.erlang.org/doc/apps/stdlib/string.html#obsolete-api-functions

[josevalim]

In that case, a PR is definitely welcome.

[lud]

Yes, exactly here.

Below the "Schema Definition Utilities" header, that is a section header after the Summary section with signatures.

In that case, a PR is definitely welcome.

I'll give it a try. Any direction or general rule to consider before starting?

[josevalim]

I'll give it a try. Any direction or general rule to consider before starting?

No, go for it!

[garazdawi]

Any direction or general rule to consider before starting?

As it is now possible to create groups via two mechanism, that is doc metadata and configuration file, IMO it should also be possible to add the "group description" with those two mechanisms. The syntax for writing the description should IMO be markdown (and not just text as deprections are), so that we can have links and other things in there.

[lud]

To me the groups are defined like so:

  defp groups_for_docs do
    [
      "Helpers": &(&1[:section] == :helpers)
    ]
  end

Or a catchall

default_group_for_doc: fn metadata ->
  if group = metadata[:group] do
    "Functions: #{group}"
  end
end

In both cases you can return a binary like :"Helpers" or a binary like "Functions: some group".

Are you mentioning another mechanism?

I feel like the markdown should be attached to that returned value, and not metadata in the module file, because instead of &1[:section] == :helpers you could match on other metadata entries. The :section key is nothing special. (The :group is though, but just because it's a default).

So in this example, the markdown should be retrieved from the :"Helpers" key. That could be yet another callback for mix.exs / project / :docs like this:

description_for_group: fn
  :"Helpers" -> 
    """
    Hello world....

    * hello
    * goodbye
    """
  _ -> 
    nil
end

But admitedly this is a lot of indirection.

We could pass the module too:

description_for_module_group: fn
  module, :"Helpers" -> 
    """
    Hello world....

    * hello
    * goodbye
    """
  _, _ -> 
    nil
end

Which would allow us to provide a default implementation:

description_for_module_group: fn module, section_header -> 
  case Code.fetch_docs(module) do
    {:docs_v1, _, _, _, _, %{^section_header => doctext}, _} -> doctext
    _ -> nil
end

Which would finally make it possible to define the doc like this:

defmodule SomeModule do
  @moduledoc """
  The real module doc
  """
  
  @moduledoc "Helpers": """
    Hello world....

    * hello
    * goodbye
  """
end

What do you think about this?

[josevalim]

I think you covered the options well but all of them come with trade-offs that I am not particularly excited about. Storing it as metadata in the moduledoc can be interesting, so we can keep all of the markup in one place, but you can also have reused groups (and perhaps reused descriptions).

[lud]

If you want to reuse I guess instead of using the defaults, you provide :description_for_module_group like this:

description_for_module_group: fn
  _module, :"Helpers" -> MyApp.Docs.description_for(:"Helpers")
  _module, :"Deprecated" -> """
    Some old stuff..
    """
  _, _ -> nil
end

I think it's possible to call app code from mix.exs right? As it is evaluated by a mix task.

[josevalim]

Yes, still a lot of indirection (as you said).

[lud]

Yes.

We could take shortcuts:

If groups_for_docs returns iodata then it's a match and the text is going to be used as the description. If it returns true, false or nil it behaves as usual (no description).

defp groups_for_docs do
  [
    Helpers:
      &(&1[:section] == :helpers &&
          """
          Some docs here
          """),
    Deprecations: &(&1[:section] == :old_stuff)
  ]
end

In the catchall you can return a {header, text} tuple, or a text/nil as usual.

default_group_for_doc: fn metadata ->
  case metadata[:section] do
    :helpers ->
      {"Helpers",
        """
        Some docs here
        """}

    :old_stuff ->
      "Deprecations"

    _ ->
      nil
  end
end

Less indirection, but no support for per-module text (but there is no support for per module header currently. I guess people just use different section names for different modules as I do), and possible weird syntax:

# Big chunk of text in the middle of a function
:helpers ->
    {"Helpers",
      """
      Some docs here
      """}

# Big chunk of text in mix.exs
:helpers ->
    {"Helpers", @helpers_doctext}

# Indirection strikes back
:helpers ->
    {"Helpers", MyApp.Docs.helpers_description()}     

[josevalim]

Oh, I like this direction. Let's start with default_group_for_doc, as it is the most general form, and allow it to return either: "Title" or [title: "Title", description: "some markdown"]? Would this work for you @garazdawi?

[garazdawi]

What I would like to be able to do in the end is what was proposed by @lub above:

defmodule SomeModule do
@moduledoc """
 The real module doc
 """
 
 @moduledoc "Helpers": """
   Hello world....

   * hello
   * goodbye
 """
end

I want to keep both the group titles and descriptions in the file that we are documenting and not in the config file for the application. Would that be possible with this approach?

Are you mentioning another mechanism?

What I meant was the default_group_for_docs + group_for_docs.

[josevalim]

I want to keep both the group titles and descriptions in the file that we are documenting and not in the config file for the application. Would that be possible with this approach?

I don't think so because we only receive the function metadata and not the module one. :( We could perhaps change it to pass both but I am not convinced.

About your approach, I am worried about polluting the .beam files with metadata which is ultimately for the docs generation and likely won't be used by any other documentation tooling (such as IDEs, terminal, etc). Thoughts? Perhaps if we want it to be used consistently, it should be an "update" to EEP 48?

[garazdawi]

For groups we already use the metadata in the terminal to group things visually, I don't see it as very unlikely that it should be used also to display that text. And yes, I suppose in that case it would be an extension to EEP-48 as well.

I get the need for not pushing too much stuff into the doc metadata, but as the groups are already there, to me it feel natural to also have this.

[lud]

I admit that I like the first proposal more.

In the first proposal you have to define a function that accepts a module and a section header, and you only have to return text or nil. A provided default implementation will let you store the text into moduledoc (we may have to force cast the header to be an atom so it can always match @moduledoc "some_quoted_atom": "some text").

In the second proposal the bahaviour is less clear because in one place you may return either a boolean, a doc text or nil, and in another place you may return a header, a header/doctext tuple (ok keyword as in your example), or nil. Which makes a lot more different data types.

[lud]

@josevalim I realise that the metadata passed to groups_for_docs functions and default_group_for_doc function bears the module name.

So we can do the second proposal that you prefer with a default callback that attempts to fetch from moduledoc metadata. It's not solving the polluting problem but I guess everyone should know that putting some big chunk of text in their modules has an impact on code size.

[josevalim]

I guess if we pass the module we can also shove the module_metadata somewhere too, so you don't have to fetch it. Btw, I realized there is another benefit for having it in the module, which is that the module can then provide some ordering of the groups. If you use @doc group: "foo", the groups appear in alphabetical order. But if you want the obsolete ones to come last, then only groups_for_docs support it. But if we have something like:

@moduledoc groups: [
  "Foo",
  %{title: "Obsolete", description: "Do not use this"}
]

Then we have per-module ordering.

Another question is: how large do you folks think these descriptions to be?

[lud]

Ah yes, nice!

And so the doc metadata would have all of it in a :moduledoc or :module_meta key.

Good idea.

For my current use case I wanted to display a short paragraph and a code example (that would of course be ran by doctest :D )

Are you worried about the size of the docs? I don't know much about the BEAM format but if I recall there is a strip docs options right?

To me this is part of the moduledoc. For now, as that feature is not available I just have dedicated sections in the general @moduledoc which takes the same place I guess.

Or is it that key/value pairs given to @moduledoc are not currently stripped or something like that?

edit

It could be nice to have the following layout too:

defmodule SomeModule do
  @moduledoc """
  Some doc
  """

  # bunch of functions
  #
  #

  @moduledoc group: "Foo"

  # bunch of functions
  #
  #

  @moduledoc group: %{
               title: "Obsolete",
               description: """
               Do not use this because

               Bla bla bla
               """
             }

  # bunch of functions
  #
  #
end

Which will probably make me and other people do something similar:

@section """
Do not use this because

Bla bla bla
"""
@moduledoc group: %{title: "Obsolete", description: @section}

[josevalim]

I don't think the design above would work with Elixir's current implementation. :( So I am thinking we should go ahead with @moduledoc groups: [...] that I previously proposed, if @garazdawi is happy with it.

[garazdawi]

You proposal is almost exactly what I originally had for Erlang/OTP, only I did it with a map instead of proplist so had to order the titles in the doc configuration script. So I am happy with that solution.

[josevalim]

@lud would you like to try a PR exploring that direction then?

[lud]

I can't start until monday but from there I'd be happy to give it a try!

[lud]

Looking at it a bit this morning... Should we keep the "Types" and "Callbacks" groups always first? I feel like so. But that means splitting the default_groups (~w(Types Callbacks Functions)) into two separate things.

[josevalim]

No, I don't think we need to keep them first (and we don't keep them first anyway in projects like Nx: https://hexdocs.pm/nx/Nx.html).

[lud]

Alright :)