Miscellaneous doc cleanup (#3564)

* Miscellaneous doc cleanup

PBENCH-1282

This is just merging a random assortment of minor doc changes I've been
accumulating for several months, before we package "Server 1.0".

Author: dbutenhof

README.md

@@ -2,38 +2,43 @@
 A Benchmarking and Performance Analysis Framework
 
 The code base includes three sub-systems. The first is the collection agent,
-`pbench-agent`, responsible for providing commands for running benchmarks
-across one or more systems while properly collecting the configuration data for
-those systems as well as specified telemetry or data from various tools (`sar`,
-`vmstat`, `perf`, etc.).
-
-The second sub-system is the `pbench-server`, which is responsible for
-archiving result tar balls, indexing them, and managing access to their
-contents.
-It provides a RESTful API which can be used by client applications, such as
-the Pbench Dashboard, to curate the results as well as to explore their
-contents.
-
-The third sub-system is the Pbench Dashboard, which provides a web-based GUI
-for the Pbench Server allowing users to list and view public results.  After
-logging in, users can view their own results, make them available for others
-to view, or delete them.  On the _User Profile_ page, a logged-in user can
-generate API keys for use with the Pbench Server API or with the Agent
-`pbench-results-push` command.  The Pbench Dashboard also serves as a platform
-for exploring and visualizing result data.
+[Pbench Agent](docs/Agent/agent.md), responsible for collecting configuration
+data for test systems, managing the collection of performance tool data from
+those systems (`sar`, `vmstat`, `perf`, etc.), and executing and postprocessing
+standardized or arbitrary benchmarked workloads (`uperf`, `fio`, `linpack`, as
+well as real system activity).
+
+The second sub-system is the [Pbench Server](docs/Server/server.md), which is
+responsible for archiving result tar balls and providing a secure
+[RESTful API](docs/Server/API/README.md) to client applications, such as the
+Pbench Dashboard. The API supports curation of results data, the ability to
+annotate results with arbitrary metadata, and to explore the results and
+collected data.
+
+The third sub-system is the [Pbench Dashboard](docs/Dashboard/user_guide.md), which
+provides a web-based GUI for the Pbench Server allowing users to list and view
+public results. After logging in, users can view their own results, publish
+results for others to view, and delete results which are no longer of use.
+On the _User Profile_ page, a logged-in user can generate API keys for use with
+the Pbench Server API or with the Agent `pbench-results-move` command. The
+Pbench Dashboard also serves as a platform for exploring and visualizing result
+data.
 
 ## How is it installed?
 Instructions for installing `pbench-agent`, can be found
 in the Pbench Agent [Getting Started Guide](
 https://distributed-system-analysis.github.io/pbench/gh-pages/start.html).
 
-For Fedora, CentOS, and RHEL users, we have made available [COPR
+For Fedora, CentOS, and RHEL users, we have made available [COPR RPM
 builds](https://copr.fedorainfracloud.org/coprs/ndokos/pbench/) for the
 `pbench-agent` and some benchmark and tool packages.
 
 You might want to consider browsing through the [rest of the documentation](
 https://distributed-system-analysis.github.io/pbench/gh-pages/doc.html).
 
+You can also use `podman` or `docker` to pull Pbench Agent containers from
+[Quay.io](https://quay.io/pbench/).
+
 ## How do I use pbench?
 Refer to the [Pbench Agent Getting Started Guide](
 https://distributed-system-analysis.github.io/pbench/gh-pages/start.html).

docs/Agent/agent.md

@@ -0,0 +1,11 @@
+# Pbench Agent
+
+The Pbench Agent manages performance experiments and collects the data for
+analysis. The Agent CLI controls the execution of predefined synthetic
+benchmarks (such as fio and uperf) and allows execution of generic workloads
+(via [pbench-user-benchmark](user-guide/man_page.md#pbench-user-benchmark))
+while gathering and managing additional selected performance data
+([pbench-register-tool-set](user-guide/man_page.md#pbench-register-tool-set)).
+
+For more information, see [Getting Started](user-guide/getting_started.md)
+and [Man Page](user-guide/man_page.md).

docs/Dashboard/user_guide.md

@@ -1,4 +1,4 @@
-# Pbench Dashboard 
+# Pbench Dashboard
 
 Pbench Dashboard is the web-based platform for consuming indexed performance benchmark data. It provides data curation capabilities for the performance datasets.
 
@@ -15,13 +15,10 @@ It has three components.
 - *Saved Runs* lists the saved runs which can be published to share with others
 - *Expiring Runs* lists the saved runs which will be deleted from the server within the next 20 days
 
-
 ![Overview](assets/images/Overview.png)
 
-
-The *User Profile* page can be used to view profile information from the OIDC authentication as well as to view and manage Pbench Server API keys. This page is accessed by selecting the *My profile* option from the dropdown menu activated by clicking on the username at the right end of the header bar. 
+The _User Profile_ page can be used to view profile information from the OIDC authentication as well as to view and manage Pbench Server API keys. This page is accessed by selecting the *My profile* option from the dropdown menu activated by clicking on the username at the right end of the header bar.
 
 From this page, Pbench Server API keys can be created by clicking on the *New API Key* button; existing keys are listed with their labels and creation dates; and, the keys can be copied or deleted using the icon buttons.
 
 ![User Profile](assets/images/UserProfile.png)
-

docs/Server/API/README.md

@@ -17,12 +17,35 @@ Any client, regardless of authentication, is able to READ a dataset with the
 `access` property set to `"public"`. Only the owner of the dataset, or a user with
 the administrator *role*, can UPDATE or DELETE the dataset.
 
+## User Identity
+
+The Pbench Server must be configured with an external OIDC identity provider,
+such as Red Hat, Google, or GitHub. The Pbench Server authenticates user access
+by validating signed OIDC JWT tokens. The token's encoded `sub` field is the
+primary "user ID" controlling ownership of and access to data. The Pbench
+Server caches each token's `sub` UUID and `preferred_username` to translate
+user ID values back into human readable form.
+
 ## Roles
 
 The Pbench Server access model allows assigning an `ADMIN` role to one or more
-user accounts through the OIDC identity provider. These users will be granted
-full CRUD access to all server data, including all datasets, server settings,
-and audit logs.
+user accounts. These users will be granted full CRUD access to all server data,
+including all datasets, server settings, and audit logs.
+
+In Pbench Server 1.0, there are two ways to assign the `ADMIN` role to a user:
+
+If you have administrative access to the OIDC client configuration, you cay add
+the `ADMIN` role token to the audience field that will be encoded into the JWT
+token under the `resource_access` field.
+
+Or you can add the `admin-role` configuration variable in the `pbench-server`
+section of the server's `/opt/pbench-server/lib/config/pbench-server.cfg` file,
+set to a list of OIDC `preferred_username` values: for example,
+
+```
+[pbench-server]
+admin-role = dave,webb
+```
 
 ## Groups
 

docs/Server/API/access_model.md

@@ -0,0 +1,25 @@
+# Pbench Server API documentation
+
+The Pbench Server API provides a RESTful HTTPS interface to curated Pbench
+performance data for use by the UI dashboard as well as any other web clients.
+
+The performance results data associated with an experiment is collectively
+referred to as a "dataset". The dataset resource is initially defined by the
+".tar.xz" tar ball uploaded to the Pbench Server. The server extracts and
+creates metadata describing the dataset, and (if appropriate) indexes synthetic
+benchmark data in Elasticsearch to support later analysis.
+
+The [V1 API](V1/README.md) provides access to the original tar ball, the
+original performance results file structure within the tarball, Pbench Server
+[metadata](metadata.md), and the Elasticsearch indexed data.
+
+The Pbench Server APIs accept parameters from a variety of sources. See the
+individual API documentation for details.
+1. Some parameters, especially "resource ids", are embedded in the URI, such as
+`/api/v1/datasets/<resource_id>`;
+2. Some parameters are passed as query parameters, such as
+`/api/v1/datasets?filter=server.benchmark:fio`;
+3. For `PUT` and `POST` APIs, some parameters may be passed as a JSON
+(`application/json` content type) request payload, such as
+`{"metadata": {"dataset.name": "new name"}}` or as an `application/octet-stream`
+byte stream, such as uploading a new results tarball file.

docs/Server/API/api.md

@@ -0,0 +1,5 @@
+# Pbench Server
+
+The Pbench Server stores performance experiment results data and provides a set
+set of [RESTful APIs](API/api.md) to support management and exploration of that
+data with access controls.