Add wiki pages for docs.rs (rust-lang#310)

* docsrs: Add wiki pages

* docsrs: Update wiki pages

* Add syntax highlighting for example

Co-Authored-By: XAMPPRocky <[email protected]>

* Improve docs.rs landing page

* Link to team instead of website individual members

Co-Authored-By: XAMPPRocky <[email protected]>

* Link to website instead of GitHub

* infra/docs/docs-rs/ -> docs-rs/

* fix table of contents

* add back infra page

* including summary

Co-authored-by: XAMPPRocky <[email protected]>

Author: jyn514

src/SUMMARY.md

@@ -20,6 +20,12 @@
         - [How to run a design meeting](./compiler/steering-meeting/how-to-run-design.md)
 - [crates.io](./crates-io/README.md)
     - [Crate removal](./crates-io/crate-removal.md)
+- [docs.rs](./docs-rs/README.md)
+  - [Adding dependencies to the build environment](./docs-rs/add-dependencies.md)
+  - [Developing without docker-compose](./docs-rs/no-docker-compose.md)
+  - [Self-hosting a docs.rs instance](./docs-rs/self-hosting.md)
+  - [Maintenance procedures](./docs-rs/maintenance.md)
+  - [Migrating from a local database to S3](./docs-rs/migrate-s3.md)
 - [GitHub](./github.md)
 - [Governance](./governance/README.md)
 - [Infrastructure](./infra/README.md)

src/docs-rs/README.md

@@ -0,0 +1,17 @@
+# docs.rs
+
+[docs.rs](https://docs.rs/) is a website that hosts documentation for crates published to [crates.io](https://crates.io/).
+
+## External Links
+
+* Source code: [rust-lang/docs.rs][repo]
+* Hosted on: `docsrs.infra.rust-lang.org` (behind the bastion -- [how to connect][bastion-connect])
+* Maintainers: [docs.rs team]
+* [Instance metrics][grafana-instance] (only available to infra team members).
+* [Application metrics][grafana-app] (only available to infra team members).
+
+[repo]: https://github.com/rust-lang/docs.rs
+[grafana-instance]: https://grafana.rust-lang.org/d/rpXrFfKWz/instance-metrics?orgId=1&var-instance=docsrs.infra.rust-lang.org:9100
+[grafana-app]: https://grafana.rust-lang.org/d/-wWFg2cZz/docs-rs?orgId=1
+[bastion-connect]: https://github.com/rust-lang/infra-team/blob/master/docs/hosts/bastion.md#logging-into-servers-through-the-bastion
+[docs.rs team]: https://www.rust-lang.org/governance/teams/dev-tools#docs-rs

src/docs-rs/add-dependencies.md

@@ -0,0 +1,83 @@
+# Add a dependency to the build environment
+
+Rustwide internally uses `rustops/crates-build-env` as the build environment for the crate. If you want to add a system package for crates to link to, this is place you're looking for.
+
+## Getting started
+
+First, clone the crates-build-env repo:
+
+```sh
+git clone https://github.com/rust-lang/crates-build-env && cd crates-build-env
+```
+
+Next, add the package to `packages.txt`. This should be the name of a package in the **Ubuntu 18.04** Repositories. See [the package home page](https://packages.ubuntu.com/) for a full list/search bar, or use `apt search` locally.
+
+## Building the image
+
+Now build the image. This will take a very long time, probably 10-20 minutes.
+
+```sh
+docker build --tag build-env .
+```
+
+## Testing the image
+
+Use the image to build your crate.
+
+```sh
+cd /path/to/docs.rs
+docker-compose build
+# NOTE: this must be an absolute path, not a relative path
+# On platforms with coreutils, you can instead use `$(realpath ../relative/path)`
+YOUR_CRATE=/path/to/your/crate
+# avoid docker-compose creating the volume if it doesn't exist
+if [ -e "$YOUR_CRATE" ]; then
+  docker-compose run -e DOCS_RS_LOCAL_IMAGE=build-env \
+                     -v "$YOUR_CRATE":/opt/rustwide/workdir \
+    web build crate --local /opt/rustwide/workdir
+else
+  echo "$YOUR_CRATE does not exist";
+fi
+```
+
+## Making multiple changes
+
+If your build fails even after your changes, it will be annoying to rebuild the image from scratch just to add a single package. Instead, you can make changes directly to the Dockerfile so that the existing packages are cached. Be sure to move these new packages from the Dockerfile to `packages.txt` once you are sure they work.
+
+On line 7 of the Dockerfile, add this line: `RUN apt-get install -y your_second_package`.
+Rerun the build and start the container; it should take much less time now:
+
+```sh
+cd /path/to/crates-build-env
+docker build --tag build-env .
+cd /path/to/docs.rs
+docker-compose run -e DOCS_RS_LOCAL_IMAGE=build-env \
+                     -v "$YOUR_CRATE":/opt/rustwide/workdir \
+    web build crate --local /opt/rustwide/workdir
+```
+
+## Run the lint script
+
+Before you make a PR, run the shell script `ci/lint.sh` and make sure it passes. It ensures `packages.txt` is in order and will tell you exactly what changes you need to make if not.
+
+## Make a pull request
+
+Once you are sure your package builds, you can make a pull request to get it adopted upstream for docs.rs and crater. Go to https://github.com/rust-lang/crates-build-env and click 'Fork' in the top right. Locally, add your fork as a remote in git and push your changes:
+
+```sh
+git remote add personal https://github.com/<your_username_here>/crates-build-env
+git add -u
+git commit -m 'add packages necessary for <your_package_here> to compile'
+git push personal
+```
+
+Back on github, make a pull request:
+
+1. Go to https://github.com/rust-lang/crates-build-env/compare
+2. Click 'compare across forks'
+3. Click 'head repository' -> <your_username>/crates-build-env
+4. Click 'Create pull request'
+5. Add a description of what packages you added and what crate they fixed
+6. Click 'Create pull request' again in the bottom right.
+
+Hopefully your changes will be merged quickly! After that you can either publish a point release (rebuilds your docs immediately) or request for a member of the docs.rs team to schedule a new build (may take a while depending on their schedules).

src/docs-rs/maintenance.md

@@ -0,0 +1,101 @@
+# Common maintenance procedures
+
+## Temporarily remove a crate from the queue
+
+It might happen that a crate fails to build repeatedly due to a docs.rs bug,
+clogging up the queue and preventing other crates to build. In this case it's
+possible to temporarily remove the crate from the queue until the docs.rs's bug
+is fixed. To do that, log into the machine and open a PostgreSQL shell with:
+
+```
+$ psql
+```
+
+Then you can run this SQL query to remove the crate:
+
+```
+UPDATE queue SET attempt = 100 WHERE name = '<CRATE_NAME>';
+```
+
+To add the crate back in the queue you can run in the PostgreSQL shell this
+query:
+
+```
+UPDATE queue SET attempt = 0 WHERE name = '<CRATE_NAME>';
+```
+
+## Pinning a version of nightly
+
+Sometimes the latest nightly might be broken, causing doc builds to fail. In
+those cases it's possible to tell docs.rs to stop updating to the latest
+nightly and instead pin a specific release. To do that you need to edit the
+`/home/cratesfyi/.docs-rs-env` file, adding or changing this environment
+variable:
+
+```
+CRATESFYI_TOOLCHAIN=nightly-YYYY-MM-DD
+```
+
+Once the file changed docs.rs needs to be restarted:
+
+```
+systemctl restart docs.rs
+```
+
+To return to the latest nightly simply remove the environment variable and
+restart docs.rs again.
+
+## Rebuild a specific crate
+
+If a bug was recently fixed, you may want to rebuild a crate so that it builds with the latest version.
+From the docs.rs machine:
+
+```
+cratesfyi queue add <crate> <version>
+```
+
+This will add the crate with a lower priority than new crates by default, you can change the priority with the `-p` option.
+
+## Adding all the crates failed after a date back in the queue
+
+After an outage you might want to add all the failed builds back to the queue.
+To do that, log into the machine and open a PostgreSQL shell with:
+
+```
+psql
+```
+
+Then you can run this SQL query to add all the crates failed after `YYYY-MM-DD
+HH:MM:SS` back in the queue:
+
+```
+UPDATE queue SET attempt = 0 WHERE attempt >= 5 AND build_time > 'YYYY-MM-DD HH:MM:SS';
+```
+
+## Removing a crate from the website
+
+Sometimes it might be needed to remove all the content related to a crate from
+docs.rs (for example after receiving a DMCA). To do that, log into the server
+and run:
+
+```
+cratesfyi database delete-crate CRATE_NAME
+```
+
+The command will remove all the data from the database, and then remove the
+files from S3.
+
+## Blacklisting crates
+
+Occasionally it might be needed to prevent a crate from being built on docs.rs,
+for example if we can't legally host the content of those crates. To add a
+crate to the blacklist, preventing new builds for it, you can run:
+
+```
+cratesfyi database blacklist add <CRATE_NAME>
+```
+
+Other operations (such as `list` and `remove`) are also supported.
+
+> **Warning:** blacklisting a crate doesn't remove existing content from the
+> website, it just prevents new versions from being built!

src/docs-rs/migrate-s3.md

@@ -0,0 +1,32 @@
+# Migrating from a local database to S3
+
+If you're running your own instance of docs.rs for personal development, and you'd like to test out the S3 integration, there are a couple steps involved. It's mostly straightforward, but there's at least one major caveat that should be taken into account.
+
+## Requirements
+
+Since docs.rs uses a fixed bucket name to upload files, you'll need to set up an independent server that implements the Amazon S3 API. [Minio](https://min.io/) is an example of a server you can set up. Instructions for installing and configuring Minio (or any S3-compliant provider) are beyond the scope of this article.
+
+## Configuring your docs.rs instance
+
+Once you have your server and credentials set up, you need to tell the docs.rs server to use it. You'll need to add some extra environment variables to the environment file you use to configure docs.rs. It uses the `S3_ENDPOINT` variable to determine where to call, and the other variables to configure its access privileges. The available environment variables are documented in `rusoto`'s [`EnvironmentProvider`](https://docs.rs/rusoto_credential/0.40.0/rusoto_credential/struct.EnvironmentProvider.html).
+
+```text
+AWS_ACCESS_KEY_ID=<access key>
+AWS_SECRET_ACCESS_KEY=<access secret>
+S3_ENDPOINT=<endpoint url>
+```
+
+## Migrating files out of the database
+
+Before you restart the process to load the new credentials, the files that are currently in the local database need to be migrated out to S3. Once docs.rs sees that it has AWS credentials in its environment, it will stop reading from the `files` table entirely. Therefore, to properly transition to using the new file storage, you need to run an extra command to upload them. It's helpful to lock the build queue while this is happening, so that no new files will sneak in while the migration happens.
+
+```sh
+cratesfyi build lock
+# edit the environment file with the above variables if you haven't already
+cratesfyi database move-to-s3 # this will take a while, depending on the size of your database
+sudo systemctl restart cratesfyi # or however you manage your docs.rs service
+# verify that files are loading from S3/Minio/etc
+cratesfyi build unlock
+```
+
+Once this is done, new crate builds will upload files directly to your file storage service rather than into the database. The `move-to-s3` command will remove rows from the `files` table in the database as it uploads them, so once you're done, you can compact the database to shrink its on-disk size.

src/docs-rs/no-docker-compose.md

@@ -0,0 +1,111 @@
+# Developing locally without docker-compose
+
+These are instructions for developing docs.rs locally. For deploying in a production environment, see [Self-hosting a docs.rs instance](self-hosting.html).
+
+While the docker-compose allows for easier setup of the required dependencies and environment, here is a breakdown of how to use the service without an outer docker container. This is useful e.g. for quickly iterating during development.
+
+Note that this does not remove the docker dependency altogether, since docs.rs uses docker at runtime for sandboxing. This just allows you to run commands more quickly since you are building in debug mode instead of release and are also caching more of the build.
+
+## Requirements
+
+The commands and package names on this page will assume a Debian-like machine with `apt` installed, but most dependencies should be relatively easy to find on Linux. Do note however that this requires the host to be `x86_64-unknown-linux-gnu`.
+
+Docs.rs has a few basic requirements:
+
+* Rust
+* Git
+* CMake, GCC, G++, and `pkg-config` (to build dependencies for crates and docs.rs itself)
+* OpenSSL, zlib, curl, and `libmagic` (to link against)
+* PostgreSQL
+
+```console
+# apt install build-essential git curl cmake gcc g++ pkg-config libmagic-dev libssl-dev zlib1g-dev postgresql
+$ sudo -u postgres psql -c "CREATE USER cratesfyi WITH PASSWORD 'password';"
+$ sudo -u postgres psql -c "CREATE DATABASE cratesfyi OWNER cratesfyi;"
+```
+
+## Building the site
+
+Be warned - this builds over 350 crates! Depending on your computer, this may
+take upwards of 10 minutes.
+
+```console
+$ git clone https://github.com/rust-lang/docs.rs && cd docs.rs
+$ cargo build
+```
+
+## The "prefix" directory
+
+docs.rs stores several files in a "prefix" directory. This can be anywhere, but if you put it in the doc.rs repo, it should go under the ./ignored/ directory so that it is not seen by git or the docker daemon.
+
+```console
+$ mkdir -p ignored/cratesfyi-prefix
+$ cd ignored/cratesfyi-prefix
+$ mkdir -vp documentations public_html sources
+$ git clone https://github.com/rust-lang/crates.io-index && cd crates.io-index
+$ git branch crates-index-diff_last-seen
+```
+
+(That last command is used to set up the `crates-index-diff` crate, so we can start monitoring new crate releases.)
+
+## Docker group
+
+docs.rs needs to run docker containers for sandboxing. Therefore, you need to be in the 'docker' group to build crates. If you are already in the docker group, you can skip this step (you can check with `groups`).
+
+```console
+# usermod -a -G docker "$USER"
+$ # now logout and back in to your shell
+$ cd /path/to/docs.rs/ignored/cratesfyi-prefix
+```
+
+## Environment for the server
+
+To ensure that the docs.rs server is configured properly, we need to set a few environment variables. This is most easily done by making a shell script.
+
+```sh
+$ cd ..
+$ echo '
+export CRATESFYI_PREFIX=.
+# or add an appropriate username/password as necessary
+export CRATESFYI_DATABASE_URL=postgresql://cratesfyi:password@localhost
+export CRATESFYI_GITHUB_USERNAME=
+export CRATESFYI_GITHUB_ACCESSTOKEN=
+export RUST_LOG=cratesfyi,rustwide=info
+' > env.sh
+```
+
+This last command assumes you put the shell script in `./env.sh`,
+but you can name it anything as long as it is in the current directory.
+
+```console
+$ . ./env.sh
+$ cargo run database migrate
+$ cargo run database update-search-index
+$ cargo run database update-release-activity
+# This will take between 5 and 30 minutes on the first run, depending on your internet speed.
+# It downloads the rustops/crates-build-env crates which is almost 1 GB.
+# It does not currently display a progress bar, this is https://github.com/rust-lang/rustwide/issues/9
+# As a workaround, you can run `docker pull rustops/crates-build-env` in a separate terminal.
+$ cargo run build crate rand 0.5.5
+```
+
+## Building on platforms other than Linux
+
+This is not currently possible. We assume in several places that the rustup toolchain is x86_64-unknown-linux-gnu. As a result, the only way to build on Mac or Alpine is to use the docker-compose file.
+
+## Resetting the database
+
+Occasionally, if you make changes to the migrations in a PR, those migrations will be saved in the database but will not be in the code when you switch back to the master branch. In this case, there is no way to undo the migration without knowing exactly which version of the code made the change (`cargo migrate` will have no effect). Here is a convenient shell script to reset the database so you don't have to remember how to undo your changes.
+
+NOTE: DO NOT RUN THIS IN PRODUCTION.
+
+```sh
+#!/bin/sh
+set -euv
+
+. ./env.sh
+
+sudo -u postgres dropdb cratesfyi --if-exists
+sudo -u postgres psql -c 'CREATE DATABASE cratesfyi OWNER cratesfyi;'
+cargo run database migrate
+```