Include UTxO HD installation/usage instructions in READMEs

jorisdral · jorisdral · commit 02943b62ba63 · 2022-06-28T17:11:49.000+02:00
* Added installation/usage instructions for UTxO HD to top-level README,
installation instructions, and node config file instructions.
* More documentation for `defaultLMDBLimits`.
* Other change: Parsing behaviour for LMDB mapsize should now allow
parsing more types of units. The previous parsing behaviour would only
parse 1024-based units such Mebibytes, Gibibytes, etc. Now, we can also
provide 1000-based units, for example.
* Fixed Redhat/Fedora/CentOS installation instructions, which was
missing an instruction to install OpenSSL libraries, amongst others.
* Crudely updated the Windows installation instructions.
diff --git a/README.rst b/README.rst
@@ -98,6 +98,8 @@ The general synopsis is as follows:
                            [--host-ipv6-addr IPV6-ADDRESS]
                            [--port PORT]
                            [--config NODE-CONFIGURATION] [--validate-db]
+                           [ --in-memory-ledger-db-backend | --lmdb-ledger-db-backend [--lmdb-mapsize BIN]]
+
      Run the node.
 
 * ``--topology`` - Filepath to a topology file describing which peers the node should connect to.
@@ -128,6 +130,12 @@ The general synopsis is as follows:
 
 * ``--validate-db`` - Flag to revalidate all on-disk database files
 
+* ``--in-memory-ledger-db-backend`` - Optionally use the in-memory backend of the UTxO HD feature: store the complete ledger state in memory. Incompatible with ``--lmdb-ledger-db-backend``.
+
+* ``--lmdb-ledger-db-backend`` - Optionally use the LMDB backend of the UTxO HD feature: store parts of the ledger state on disk. Incompatible with ``--in-memory-ledger-db-backend``. The node uses the LMDB backend by default if no ``--*-db-backend`` flags are set.
+
+  * ``--lmdb-mapsize`` - Optionally set the mapsize (maximum database size) for the LMDB backend. By default, the mapsize (maximum database size) of the backend is set to 16 Gigabytes. Warning: if the database size exceeds the given mapsize, the node will abort. Therefore, the mapsize should be set to a value high enough to guarantee that the maximum database size will not be reached during the expected node uptime.
+
 Configuration ``.yaml`` files
 =============================
 
diff --git a/cardano-node/src/Cardano/Node/Configuration/LedgerDB.hs b/cardano-node/src/Cardano/Node/Configuration/LedgerDB.hs
@@ -28,6 +28,7 @@ data BackingStoreSelectorFlag =
 
 -- | Recommended settings for the LMDB backing store.
 --
+-- === @'lmdbMapSize'@
 -- The default @'LMDBLimits'@ uses an @'lmdbMapSize'@ of @16_000_000_000@
 -- bytes, or 16 GigaBytes. @'lmdbMapSize'@ sets the size of the memory map
 -- that is used internally by the LMDB backing store, and is also the
@@ -42,6 +43,29 @@ data BackingStoreSelectorFlag =
 -- @'lmdbMapSize'@. If this fatal error were to occur, we would expect that
 -- the node can continue normal operation if it is restarted with a higher
 -- @'lmdbMapSize'@ configured. Nonetheless, this situation should be avoided.
+--
+-- === @'lmdbMaxDatabases'@
+-- The @'lmdbMaxDatabases'@ is set to 10, which means that the LMDB backing
+-- store will allow up @<= 10@ internal databases. We say /internal/
+-- databases, since they are not exposed outside the backing store interface,
+-- such that from the outside view there is just one /logical/ database.
+-- Two of these internal databases are reserved for normal operation of the
+-- backing store, while the remaining databases will be used to store ledger
+-- tables. At the moment, there is at most one ledger table that will be
+-- stored in an internal database: the UTxO. Nonetheless, we set
+-- @'lmdbMaxDatabases'@ to @10@ in order to future-proof these limits.
+--
+-- === @'lmdbMaxReaders'@
+-- The @'lmdbMaxReaders'@ limit sets the maximum number of threads that can
+-- read from the LMDB database. Currently, there should only be a single reader
+-- active. Again, we set @'lmdbMaxReaders'@ to @16@ in order to future-proof
+-- these limits.
+--
+-- === References
+-- For more information about LMDB limits, one should inspect:
+-- * The @lmdb-simple@ and @haskell-lmdb@ forked repositories.
+-- * The official LMDB API documentation at
+--    <http://www.lmdb.tech/doc/group__mdb.html>.
 defaultLMDBLimits :: LMDBLimits
 defaultLMDBLimits = LMDBLimits {
     lmdbMapSize = 16_000_000_000
diff --git a/cardano-node/src/Cardano/Node/Configuration/POM.hs b/cardano-node/src/Cardano/Node/Configuration/POM.hs
@@ -331,7 +331,7 @@ instance FromJSON PartialNodeConfiguration where
              maybeMapSize :: Maybe String <- v .:? "LMDBMapSize"
              mapSize <- case maybeMapSize of
                Nothing -> return Nothing
-               Just s  -> case parseValue ParseBinary s of
+               Just s  -> case parseValue ParseExact s of
                  Left e      -> fail ("Malformed LMDBMapSize: " <> e)
                  Right units -> return $ Just units
              return . Just . LMDB $ mapSize
diff --git a/cardano-node/src/Cardano/Node/Parsers.hs b/cardano-node/src/Cardano/Node/Parsers.hs
@@ -240,7 +240,7 @@ parseLedgerDBBackend = parseInMemory <|> parseLMDB <*> optional parseMapSize
 
     parseMapSize :: Parser Int
     parseMapSize =
-      option (eitherReader (parseValue ParseBinary)) (
+      option (eitherReader (parseValue ParseExact)) (
            long "lmdb-mapsize"
         <> metavar "BIN"
         <> help "The maximum database size defined as a binary \
diff --git a/doc/getting-started/building-the-node-on-windows.md b/doc/getting-started/building-the-node-on-windows.md
@@ -1,42 +1,52 @@
 # Building the development version of the Cardano node on Windows
 
-This document explains how to build a __DEVELOPMENT__ version of the `cardano-node`.
-Note that this is *not* for building a __PRODUCTION__ version of the node. 
+This document explains how to build a __DEVELOPMENT__ version of the `cardano-node`. Note that this is *not* for building a __PRODUCTION__ version of the node.
 
-To start building on `Windows`, you will need to install and configure specific tools outlined below. We recommend using [chocolatey](https://chocolatey.org), which provides the `choco` command to install these tools.  
+We recommend installing and using `git-bash` for development purposes, which is `git` when installed with `choco install git`.
 
-You can run all the instructions that invoke `choco` in PowerShell with root privileges. We recommend installing and using `git-bash` for development purposes, which is `git` when installed with `choco install git`.
 
-## Install GHC
+## Install the Haskell environment
 
-The recommended way is to run the following command in PowerShell:
+The recommended way to install the Haskell tools is via [GHCup](https://www.haskell.org/ghcup/). Check [this page](https://www.haskell.org/ghcup/install/) for further explanation on the installation process.
 
-```PowerShell
-choco install --version 8.10.7 ghc
+Once GHCup is installed, open a new terminal (to get an updated environment) and run:
+
+```bash
+ghcup install ghc 8.10.7
+ghcup install cabal 3.6.2.0
+ghcup set ghc 8.10.7
+ghcup set cabal 3.6.2.0
 ```
 
+Alternatively, with `ghcup tui` you can pick the specific versions of the tools that you want to install, in particular you should have installed and set:
+- `cabal >= 3.6.2.0`
+- `GHC >= 8.10.7`
+
 ## Install pkg-config
 
-```PowerShell
+To start building on `Windows`, you will need to install and configure specific tools outlined below. We recommend using [chocolatey](https://chocolatey.org), which provides the `choco` command to install some of these tools. You can run all the instructions that invoke `choco` in bash with root privileges.
+
+```bash
 choco install pkgconfiglite
 ```
 
 ## Install vcpkg
 
-You will first need to install `vcpkg` to proceed with `libsodium` library installation (which is the next step). 
+You will first need to install `vcpkg` to proceed installing necessary libraries (which is the next step).
 
-For this, use `git`(which you can also install using `choco`) and follow [these
-instructions](https://github.com/microsoft/vcpkg#quick-start-windows).
+For this, use `git`(which you can also install using `choco`) and follow [these instructions](https://github.com/microsoft/vcpkg#quick-start-windows).
 
-You can now install `libsodium` with the following command:
+You can now install `libsodium`, `openssl`, `secp256k1` and `lmdb` with the following commands:
 ```bash
 ./vcpkg install --triplet x64-windows libsodium
+./vcpkg install --triplet x64-mingw-dynamic openssl
+./vcpkg install secp256k1
+./vcpkg install lmdb
 ```
 
-## Create libsodium.pc file
+## Create libsodium.pc file (and others)
 
-To find system dependencies like `libsodium`, `cabal` uses `pkg-config`. On Windows, you will need to create the `libsodium.pc` description file in a correct
-directory.
+To find system dependencies like `libsodium`, `cabal` uses `pkg-config`. On Windows, you will need to create the `libsodium.pc` description file in a correct directory.
 
 In one of the paths reported by:
 ```bash
@@ -54,20 +64,22 @@ Description: libsodium library
 Cflags: -I${includedir}/sodium
 Libs: -L${libdir} -llibsodium
 ```
-> Note that you need to replace `VCPKG_PATH` with the
-absolute path, where you use `vcpkg`, and `LIBSODIUM_VERSION` with the version
-number of `libsodium` which was installed on your system. Please verify that
-the paths above contain
-`libsodium.dll` file and headers.
+> Note that you need to replace `VCPKG_PATH` with the absolute path, where you use `vcpkg`, and `LIBSODIUM_VERSION` with the version number of `libsodium` which was installed on your system. Please verify that the paths above contain `libsodium.dll` file and headers.
+
+> Also, you cannot use `prefix=` in the `libsodium.pc` file. This might be changed for some other directory, `pkg-config` provides a switch to use the provided `prefix`, but there is no way to instruct `cabal` to do so.
+
+Follow the same procedure of creating `.pc` files for `openssl`, `secp256k1` and `lmdb`.
+
+Finally, ensure that
+```
+pkg-config --list-all
+```
+lists all packages: `libsodium`, `openssl`, `secp256k1` and `lmdb`.
 
-> Also, you cannot use `prefix=` in the `libsodium.pc` file. This might be changed for
-some other directory, `pkg-config` provides a switch to use the provided
-`prefix`, but there is no way to instruct `cabal` to do so.
 
 ## `cabal` configuration
 
-Go to the directory, where you cloned the `cardano-node` repository and add the command below
-to your `cabal.project.local` file (if you don't already have it, create one):
+Go to the directory, where you cloned the `cardano-node` repository and add the command below to your `cabal.project.local` file (if you don't already have it, create one):
 
 ```
 max-backjump: 5000
@@ -80,8 +92,7 @@ extra-lib-dirs: "VCPKG_PATH\\installed\\x64-windows\\bin"
 extra-include-dirs: "VCPKG_PATH\\installed\\x64-windows\\include"
 ```
 
-The final step is to add `VCPKG_PATH/installed/x64-windows/bin` to the `PATH`
-variable. Since we are using `git-bash`, you can do this with: 
+The final step is to add `VCPKG_PATH/installed/x64-windows/bin` to the `PATH` variable. Since we are using `git-bash`, you can do this with:
 
 ```
 export PATH="VCPKG_PATH/installed/x64-windows/bin:${PATH}"
@@ -93,12 +104,10 @@ You can now build the node with:
 ```bash
 cabal build --builddir /c/dist exe:cardano-node
 ```
-> Note: using `--builddir /c/dist` with a succinct directory protects you
-from exceeding the [maximal path
-size](https://docs.microsoft.com/en-us/windows/win32/fileio/maximum-file-path-limitation)
-on Windows (which is a limitation of the linker rather than `ghc` itself).
 
-You can now verify whether the node runs: 
+> Note: using `--builddir /c/dist` with a succinct directory protects you from exceeding the [maximal path size](https://docs.microsoft.com/en-us/windows/win32/fileio/maximum-file-path-limitation) on Windows (which is a limitation of the linker rather than `ghc` itself).
+
+You can now verify whether the node runs:
 ```bash
 cabal run --builddir /c/dist exe:cardano-node -- run --help
 ```
diff --git a/doc/getting-started/install.md b/doc/getting-started/install.md
@@ -27,6 +27,8 @@ To download the source code and build it, you need the following packages and to
 * developer libraries for `systemd`,
 * developer libraries for `ncurses`,
 * `ncurses` compatibility libraries,
+* developer libraries for `openssl`,
+* developer libraries for `lmdb`,
 * the Haskell build tool `cabal`,
 * the GHC Haskell compiler (version `8.10.7` or above).
 
@@ -35,14 +37,14 @@ In Redhat, Fedora, and Centos:
 ```bash
 sudo yum update -y
 sudo yum install git gcc gcc-c++ tmux gmp-devel make tar xz wget zlib-devel libtool autoconf -y
-sudo yum install systemd-devel ncurses-devel ncurses-compat-libs -y
+sudo yum install systemd-devel ncurses-devel ncurses-compat-libs which jq openssl-devel lmdb-devel -y
 ```
 
 For Debian/Ubuntu, use the following instead:
 
 ```bash
 sudo apt-get update -y
-sudo apt-get install automake build-essential pkg-config libffi-dev libgmp-dev libssl-dev libtinfo-dev libsystemd-dev zlib1g-dev make g++ tmux git jq wget libncursesw5 libtool autoconf -y
+sudo apt-get install automake build-essential pkg-config libffi-dev libgmp-dev libssl-dev libtinfo-dev libsystemd-dev zlib1g-dev make g++ tmux git jq wget libncursesw5 libtool autoconf liblmdb-dev -y
 ```
 
 If you are using a different flavor of Linux, you will need to use the correct package manager for your platform instead of `yum` or `apt-get`, and the names of the packages you need to install might differ.  On MacOSX, use the Homebrew (`brew`) installer.
diff --git a/doc/getting-started/understanding-config-files.md b/doc/getting-started/understanding-config-files.md
@@ -355,3 +355,19 @@ It is also possible to have more fine grained control over filtering of trace ou
 	  }
 	}
 ```
+
+#### UTxO HD
+
+The UTxO HD feature of the consensus layer allows parts of the ledger state to be stored on-disk, whereas consensus previously kept the whole ledger state in memory. The node can be configured to use any of two *backends*:
+
+* The *LMDB backend*, which is built around a Lightning Memory-Mapped Database (LMDB), stores parts of the ledger state on disk. Optionally, a configuration can be included that sets the mapsize (maximum size) of the on-disk database that is used to store parts of the ledger state. By default, the mapsize is set to 16 Gigabytes. **Warning**: if the database size exceeds the given mapsize, the node will abort. Therefore, the mapsize should be set to a value high enough to guarantee that the maximum database size will not be reached during the expected node uptime.
+  ```json
+  "LedgerDBBackend": "LMDB",
+  "LMDBMapSize": "16G",
+  ```
+* The *InMemory backend* stores parts of the ledger state in memory.
+  ```json
+  "LedgerDBBackend": "InMemory",
+  ```
+
+If no choice of backend is configured, then the node defaults to using the LMDB backend with the *default* mapsize. It is recommended not to opt for the in-memory backend if the machine that is running the node is constrained by the amount of available memory.
