Convert RST files to Markdown.

[paulyoung]

As requested by @lattner on the mailing list.

This is currently the result of running the following command in the root directory:

find ./ -iname "*.rst" -type f -exec sh -c 'pandoc "${0}" -o "${0%.rst}.md"' {} \;

[jckarter]

This looks like detritus from the automatic conversion. You might need to check how other .. annotations got mangled too.

[paulyoung]

What's the preferred way to resolve those? Just remove them?

[alexwlchan]

@paulyoung I think the orphan directive can just be removed.

When Sphinx builds the RST files into HTML, it emits a warning if it finds any files that don’t appear in the table of contents. The :orphan: directive suppresses this warning. See Miscellaneous markup.

Some other directives (e.g. :note:) affect the formatting of a block of text. Perhaps those could be blockquotes with "Note:" written at the top? e.g.

.. note::

   This is a note.

becomes:

> **Note:** This is a note.

[paulyoung]

I'm seeing a couple of similar artifacts that weren't handled well in the conversion, so will address those too. Unfortunately the diff is too big for GitHub, but an example can be seen here.

[jrose-apple]

Yep, a pandoc conversion is just a starting point; after that, we need to manually fix up the output to look good in GitHub-Flavored Markdown again. There are a few docs that make extensive use of ReST features and we definitely don't want to regress usability for those—possibly even to the point of leaving them as ReST for now.

(Why GitHub-Flavored Markdown? Because most likely the most common way to view them will be on GitHub. If we can also get Kramdown compatibility that's great, because that's what our website uses, but that's just a bonus.)

[paulyoung]

Could I get some advice on how to convert the following type of thing?

.. default-role:: term
.. title:: Library Evolution Support in Swift ("Resilience")

:Author: Jordan Rose
:Author: John McCall

[jrose-apple]

:Author: can just turn into regular text, and title…doesn't really fit on the GitHub page, although maybe it would work well in a Jekyll-style YAML front matter block.

default-role is a little more important: it means that everything in backticks in the document is being treated as a ReST term for the glossary at the bottom. Those can be turned into manual links and anchors.

(I wish GitHub Flavored Markdown supported definition lists in a non-HTML way, but we should probably reformat without them.)

[paulyoung]

Speaking of manual links and anchors, I just implemented some of that here.

I made some adjustments to respect the existing line limit. If that's not really a requirement anymore, please let me know.

[paulyoung]

@jrose-apple please let me know what you think of my efforts here.

[paulyoung]

This type of thing is also possible:

title	author	author
Library Evolution Support in Swift ("Resilience")	Jordan Rose	John McCall

but much more code:

| title                                             | author      | author      |
| ------------------------------------------------- | ----------- | ----------- |
| Library Evolution Support in Swift ("Resilience") | Jordan Rose | John McCall |

when compared to front matter:

---
title: Library Evolution Support in Swift ("Resilience")
authors: Jordan Rose, John McCall
---

[jrose-apple]

It is a weekend today. :-) I'll come back and look through this carefully on Monday. Thanks for taking this on!

[paulyoung]

@jrose-apple thanks for reminding me 😄

In no way did I expect any immediate feedback.

I was actually going to leave this for the day anyway. Enjoy the weekend!

[alexwlchan]

@paulyoung The YAML front matter is probably preferable – that’s a fairly standard format for specifying info at the top of documents. Markdown tables are just icky.

[jrose-apple]

All right, I got through the "A" docs and I have a fair number of comments already:

• ReST lists using (1), (2), etc are coming through unformatted.
• Several internal links are missing. In some cases this is also leaving a dangling trailing underscore (example).
• note:: blocks are coming through with "note" in lowercase (example). It should probably at least be uppercase, but I'd actually suggest reflowing this to > **Note:** This rule...
• Definition lists aren't supported, which already stinks, but it's specifically causing formatting issues for multi-line definitions (example).
• Odd line breaks in blockquotes (example).
• Inline CSS is just showing up as text (example). (You also missed some front matter there.)
• GitHub Markdown doesn't support footnotes (example). I don't really know what we're going to do about this one.
• (nitpick) URLs are no longer autolinked, so using Markdown <url> syntax would be nice.

Thanks for taking this on. Since I'm being fairly nitpicky about this conversion, it's okay if at any point you want to set it down and leave it to someone else. Another possibility might be to do this work in a public branch, so that others can help, but leave the main repo as ReST for now so that there remains only one way to build the docs for a normal user.

[paulyoung]

@jrose-apple yeah, I really only did some manual work on ABI.md and LibraryEvolution.md up until now.

I was actually thinking it may be best to do this one document at a time to make it easier to review. It would also (hopefully) have a shorter turnaround time and reduce the probability of conflicts.

How does that sound to you?

[lattner]

Another suggestion is that it is probably best to tackle one or two documents at a time, to reduce conflicts with other parallel changes to the documents.

[jrose-apple]

I'm happy to do things that way, but again I don't want to have the master branch have a mix of ReST and Markdown for an extended period of time. If both of you agree I'll make an rst-to-markdown branch in the main repository and we can do the docs one by one there. (And I'd suggest starting with the docs that haven't been significantly changed recently!)

What do you think?

[paulyoung]

@jrose-apple rst-to-markdown in the main repo sounds good to me.

[jrose-apple]

I've added the branch. Let's start doing small pull requests there. Thanks, @paulyoung!

[flying-sheep]

a question: why?

rST and ASCIIdoc are due to their extrensible nature a natural fit for technical documentation. creating yet another incompatible dialect of markdown is something nobody wants to see.

they easily allow to specify internal links that can be displayed with meaningful formatting, as well as external ones using intersphinx.

using rST or ASCIIdoc leads to works of beauty like this. observe how functions, classes, code samples, and links differ visually.

using markdown leads to generic, unlinked stuff like this. i want to click on Eq and Hash and so on!

[jrose-apple]

Good question. ReST is harder to write*, isn't what we're using for anything else (doc comments, proposals, website), and doesn't buy us much when most people will be reading the docs directly on GitHub anyway, which only supports very basic ReST. Most of the docs don't refer to the stdlib as it is today, nor to each other, meaning auto-cross-linking isn't so useful; they are designs, rationales, notes, and proposals.

It'd be a little different if we generated the docs somewhere for people to read, but even then I bet quite a few of them would end up looking at the version on GitHub anyway.

* My number 1 pet peeve with ReST is not being able to nest markup easily. Number 2 is that Sphinx exits early if you have any errors in your syntax.

[dabrahams]

On Dec 17, 2015, at 1:28 PM, Jordan Rose [email protected] wrote:

Good question. ReST is harder to write*, isn't what we're using for anything else (doc comments, proposals, website), and doesn't buy us much when most people will be reading the docs directly on GitHub anyway, which only supports very basic ReST. Most of the docs don't refer to the stdlib as it is today, nor to each other, meaning auto-cross-linking isn't so useful; they are designs, rationales, notes, and proposals.

It'd be a little different if we generated the docs somewhere for people to read, but even then I bet quite a few of them would end up looking at the version on GitHub anyway.

• My number 1 pet peeve with ReST is not being able to nest markup easily.

That is annoying, but honestly the need for it is super-rare, and where I need it the most (markup within code, to emphasize certain parts), ReST is capable while Markdown is incapable.
Number 2 is that Sphinx exits early if you have any errors in your syntax.

That is a tunable behavior if I’m not mistaken. We used to have it continue, but IMO that was worse, because silent errors crept into our docs.

IMO the main argument for Markdown is that it’s more broadly known and adopted.

-Dave

[flying-sheep]

it’s sad to see semantic information being sacrificed that could have been used to improve quality and visuals.

e.g. a ..note:: becomes a > **Note**, which means no more than “bold text in a blockquote”. who’s being quoted? what does the bold here mean? only humans know.

i think hosting the docs somewhere different and linking to them from the repo’s readme and all the blog posts and whatever that will undoubtedly refer to them will serve well to make the official location pop up early on google.

about rST being hard to write: i only have problems with the inline link syntax, which matter more for things like forums/issue trackers, where you don’t have bigger sections of text with many link targets that are well-suited for the non-inline syntax. and the rest of the syntax is pretty nice, and, like markdown, learnable in a few minutes (“learnable” as in: you grasp enough to make meaningful changes)

Most of the docs don't refer to the stdlib as it is today, nor to each other,

oh? that explains why you don’t see the awesomeness that rST has to offer. e.g. here you could have created links with minimal effort once the domain roles for swift were written. instead of AnyObject, you could have written :type:AnyObject`` and gotten a link for almost free (only had to add :type:), with semantic information about it being a type attached.