Skip to content

Commit 93fbacc

Browse files
authored
Merge pull request #4079 from input-output-hk/jdral/CAD-4437-utxo-hd-readme-instructions
CAD-4437 Include UTxO HD installation/usage instructions in READMEs
2 parents b9ceb23 + db9461f commit 93fbacc

File tree

7 files changed

+237
-64
lines changed

7 files changed

+237
-64
lines changed

README.rst

+20
Original file line numberDiff line numberDiff line change
@@ -98,6 +98,8 @@ The general synopsis is as follows:
9898
[--host-ipv6-addr IPV6-ADDRESS]
9999
[--port PORT]
100100
[--config NODE-CONFIGURATION] [--validate-db]
101+
[ --in-memory-ledger-db-backend | --lmdb-ledger-db-backend [--lmdb-mapsize BIN]]
102+
101103
Run the node.
102104
103105
* ``--topology`` - Filepath to a topology file describing which peers the node should connect to.
@@ -128,6 +130,24 @@ The general synopsis is as follows:
128130

129131
* ``--validate-db`` - Flag to revalidate all on-disk database files
130132

133+
* ``--in-memory-ledger-db-backend`` - Use the in-memory backend to store the
134+
UTxO portion of the ledger state in memory. As a result, the whole ledger
135+
state will be stored in memory. Incompatible with
136+
``--lmdb-ledger-db-backend``.
137+
* ``--lmdb-ledger-db-backend`` - Use the LMDB backend to store the UTxO portion
138+
of the ledger state on disk. The remainder of the ledger state will still be
139+
stored in memory. Incompatible with ``--in-memory-ledger-db-backend``. The
140+
node uses the LMDB backend by default if no ``--*-db-backend`` flags are set.
141+
142+
* ``--lmdb-mapsize`` - Set the maximum database size (mapsize) for the LMDB
143+
backend. By default, the mapsize of the backend is set to 16 Gigabytes.
144+
Warning: if the database size exceeds the given mapsize, the node will
145+
abort. Therefore, the mapsize should be set to a value high enough to
146+
guarantee that the maximum database size will not be reached during the
147+
expected node uptime. The current default value is sufficient to offer this
148+
guarantee, and setting the mapsize to anything less is therefore not
149+
recommended.
150+
131151
Configuration ``.yaml`` files
132152
=============================
133153

cardano-node/src/Cardano/Node/Configuration/LedgerDB.hs

+24
Original file line numberDiff line numberDiff line change
@@ -28,6 +28,7 @@ data BackingStoreSelectorFlag =
2828

2929
-- | Recommended settings for the LMDB backing store.
3030
--
31+
-- === @'lmdbMapSize'@
3132
-- The default @'LMDBLimits'@ uses an @'lmdbMapSize'@ of @16_000_000_000@
3233
-- bytes, or 16 GigaBytes. @'lmdbMapSize'@ sets the size of the memory map
3334
-- that is used internally by the LMDB backing store, and is also the
@@ -42,6 +43,29 @@ data BackingStoreSelectorFlag =
4243
-- @'lmdbMapSize'@. If this fatal error were to occur, we would expect that
4344
-- the node can continue normal operation if it is restarted with a higher
4445
-- @'lmdbMapSize'@ configured. Nonetheless, this situation should be avoided.
46+
--
47+
-- === @'lmdbMaxDatabases'@
48+
-- The @'lmdbMaxDatabases'@ is set to 10, which means that the LMDB backing
49+
-- store will allow up @<= 10@ internal databases. We say /internal/
50+
-- databases, since they are not exposed outside the backing store interface,
51+
-- such that from the outside view there is just one /logical/ database.
52+
-- Two of these internal databases are reserved for normal operation of the
53+
-- backing store, while the remaining databases will be used to store ledger
54+
-- tables. At the moment, there is at most one ledger table that will be
55+
-- stored in an internal database: the UTxO. Nonetheless, we set
56+
-- @'lmdbMaxDatabases'@ to @10@ in order to future-proof these limits.
57+
--
58+
-- === @'lmdbMaxReaders'@
59+
-- The @'lmdbMaxReaders'@ limit sets the maximum number of threads that can
60+
-- read from the LMDB database. Currently, there should only be a single reader
61+
-- active. Again, we set @'lmdbMaxReaders'@ to @16@ in order to future-proof
62+
-- these limits.
63+
--
64+
-- === References
65+
-- For more information about LMDB limits, one should inspect:
66+
-- * The @lmdb-simple@ and @haskell-lmdb@ forked repositories.
67+
-- * The official LMDB API documentation at
68+
-- <http://www.lmdb.tech/doc/group__mdb.html>.
4569
defaultLMDBLimits :: LMDBLimits
4670
defaultLMDBLimits = LMDBLimits {
4771
lmdbMapSize = 16_000_000_000

cardano-node/src/Cardano/Node/Configuration/POM.hs

+1-1
Original file line numberDiff line numberDiff line change
@@ -331,7 +331,7 @@ instance FromJSON PartialNodeConfiguration where
331331
maybeMapSize :: Maybe String <- v .:? "LMDBMapSize"
332332
mapSize <- case maybeMapSize of
333333
Nothing -> return Nothing
334-
Just s -> case parseValue ParseBinary s of
334+
Just s -> case parseValue ParseExact s of
335335
Left e -> fail ("Malformed LMDBMapSize: " <> e)
336336
Right units -> return $ Just units
337337
return . Just . LMDB $ mapSize

cardano-node/src/Cardano/Node/Parsers.hs

+1-1
Original file line numberDiff line numberDiff line change
@@ -240,7 +240,7 @@ parseLedgerDBBackend = parseInMemory <|> parseLMDB <*> optional parseMapSize
240240

241241
parseMapSize :: Parser Int
242242
parseMapSize =
243-
option (eitherReader (parseValue ParseBinary)) (
243+
option (eitherReader (parseValue ParseExact)) (
244244
long "lmdb-mapsize"
245245
<> metavar "BIN"
246246
<> help "The maximum database size defined as a binary \
Original file line numberDiff line numberDiff line change
@@ -1,104 +1,201 @@
11
# Building the development version of the Cardano node on Windows
22

3-
This document explains how to build a __DEVELOPMENT__ version of the `cardano-node`.
4-
Note that this is *not* for building a __PRODUCTION__ version of the node.
3+
This document explains how to build a __DEVELOPMENT__ version of the `cardano-node`. Note that this is *not* for building a __PRODUCTION__ version of the node.
54

6-
To start building on `Windows`, you will need to install and configure specific tools outlined below. We recommend using [chocolatey](https://chocolatey.org), which provides the `choco` command to install these tools.
5+
See [install.md](./install.md) for a more thorough explanation of the dependencies and packages that have to be installed to run a node. Although the installation instructions in that document are aimed at building on Linux, large parts of the text are interesting to Windows users as well.
76

8-
You can run all the instructions that invoke `choco` in PowerShell with root privileges. We recommend installing and using `git-bash` for development purposes, which is `git` when installed with `choco install git`.
7+
## Install the Haskell environment
98

10-
## Install GHC
9+
The recommended way to install the Haskell tools is via [GHCup](https://www.haskell.org/ghcup/). Check [this page](https://www.haskell.org/ghcup/install/) for further explanation on the installation process.
1110

12-
The recommended way is to run the following command in PowerShell:
11+
Once GHCup is installed, open a new terminal (to get an updated environment) and run:
1312

14-
```PowerShell
15-
choco install --version 8.10.7 ghc
13+
```bash
14+
ghcup install ghc 8.10.7
15+
ghcup install cabal 3.6.2.0
16+
ghcup set ghc 8.10.7
17+
ghcup set cabal 3.6.2.0
1618
```
1719

18-
## Install pkg-config
20+
## Install MSYS2 and basic dependencies
1921

20-
```PowerShell
21-
choco install pkgconfiglite
22+
### GHCUP and MSYS2
23+
24+
Note that the GHCUP comes with its own installation for MSYS2, and it can be used to build `cardano-node`. You can start the MSYS2 shell with
25+
```cmd
26+
<ghcup-path>\msys64\msys2_shell.cmd -defterm -here -no-start -mingw64
27+
```
28+
29+
Make sure to enable `mingw64` subsystem. It is also possible to enable `mingw64` subsystem by
30+
31+
```bash
32+
source shell mingw64
33+
```
34+
35+
36+
The MSYS2 shell is bundled with `pacman` package manager. `GHC` is bundled with `gcc` for `mingw`. Configure your environment to use this tool-chain.
37+
38+
```bash
39+
pacman -Syu
40+
pacman -S --needed base-devel automake git
41+
export PATH=$PATH:/mingw64/bin:<ghcup-path>/bin:<ghcup-path>/ghc/<ghc-version>/mingw/bin
42+
```
43+
44+
Alternatively, you can update the `mingw64` tool-chain, that adds number of packages including latest `gcc`. This may give rise to `crt` linker error. Please read the [section - global cabal configuration](#global-cabal-configuration) to mitigate the error.
45+
46+
```bash
47+
pacman -S mingw-w64-x86_64-toolchain
2248
```
2349

24-
## Install vcpkg
50+
### Installing MSYS independently
2551

26-
You will first need to install `vcpkg` to proceed with `libsodium` library installation (which is the next step).
52+
Go to [the MSYS2 website](https://www.msys2.org/) and follow its instructions to install MSYS2. It will also install `pacman`. Then, update and upgrade all `pacman` packages and install basic dependencies:
53+
54+
```
55+
pacman -Syu
56+
pacman -S --needed base-devel mingw-w64-x86_64-toolchain git
57+
```
2758

28-
For this, use `git`(which you can also install using `choco`) and follow [these
29-
instructions](https://github.com/microsoft/vcpkg#quick-start-windows).
59+
## Installing and configuring third party libraries
60+
You can use `vcpkg`, or `pacman` or both to install third party libraries. Following are the dependencies for `cardano-node`.
61+
62+
| package | `vcpkg` | `pacman` | Remarks
63+
|-----------|-----------|----------------------------|--------------------------------------
64+
| libsodium | libsodium | mingw-w64-x86_64-lmdb |
65+
| lmdb | lmdb | mingw-w64-x86_64-libsodium |
66+
| secp256k1 | secp256k1 | __not available__ | `vcpkg` package is not complete.
67+
| openssl | openssl | mingw-w64-x86_64-openssl |
68+
|-----------|-----------|----------------------------|--------------------------------------
69+
70+
### Using __vcpkg__
71+
- Install `vcpkg` for MSYS2 with instruction given [here](https://vcpkg.io/en/docs/users/mingw.html)
72+
- Set following variables for installing libraries
73+
```bash
74+
export VCPKG_DEFAULT_TRIPLET=x64-mingw-static
75+
export VCPKG_DEFAULT_HOST_TRIPLET=x64-mingw-dynamic
76+
```
77+
- You can also use `x64-mingw-dynamic` for `VCPKG_DEFAULT_TRIPLET`, if you'd like to use shared libraries
78+
- Install libraries
79+
```bash
80+
vcpkg install libsodium
81+
vcpkg install openssl
82+
vcpkg install lmdb
83+
```
84+
- __cardano-crypto-class__ uses several features of `secp256k1` that are not compiled into `vcpkg` package. Hence it needs to be compiled separately.
85+
- `vcpkg` generates `pkg-config` when it can. It generates configuration for `openssl`. To avoid mixing configurations with existing system, you can use following to install packages
86+
```bash
87+
PATH="${PATH/:\/usr\/bin:\/bin:/:}:/ming64/bin" ./vcpkg.exe install <pkg-name>
88+
```
89+
+ Note that packages will be generated at `<vc-pkg>/installed/${VCPKG_DEFAULT_TRIPLET}/lib/pkgconfig`.
90+
91+
### Using __MSYS2 pacman__
92+
93+
You can search for packages [here](https://packages.msys2.org/)
94+
95+
- Install dependencies as below
96+
```bash
97+
pacman -S mingw-w64-x86_64-libsodium
98+
pacman -S mingw-w64-x86_64-lmdb
99+
pacman -S mingw-w64-x86_64-openssl
100+
```
101+
102+
### Install Secp256k1
103+
104+
You can use `secp256k1` either by downloading from hydra. Or by downloading from [github](https://github.com/bitcoin-core/secp256k1)
30105

31-
You can now install `libsodium` with the following command:
32-
```bash
33-
./vcpkg install --triplet x64-windows libsodium
106+
```
107+
pacman -S unzip
108+
curl https://hydra.iohk.io/job/Cardano/haskell-nix/windows-secp256k1/latest/download/1 -o secp256k1.zip
109+
mkdir secp256k1
110+
cd secp256k1/
111+
unzip ../secp256k1.zip
112+
cp -r * /mingw64
34113
```
35114

36-
## Create libsodium.pc file
115+
If `unzip` complains that `../secp256k1.zip` is not a zip file, unzip the
116+
file manually in the explorer by choosing the option to unzip from the
117+
right-click menu.
37118

38-
To find system dependencies like `libsodium`, `cabal` uses `pkg-config`. On Windows, you will need to create the `libsodium.pc` description file in a correct
39-
directory.
119+
### Managing package configuration
120+
Make sure that package configurations are correectly installed. If you are using chocolatey or other tool, it may be possible that more than one `pkg-config` tools are installed. You are encouraged to use `pkg-config` that __MSYS2__ has to minimize the error. Also make sure that the pkg configuration are discoverable by checking that `pkg-config --list-all` lists all the installed packages.
121+
122+
Note that the pkg-config will only look in the paths pointed to by `pkg-config --variable pc_path pkg-config`. Also check if pkg-config correctly displays library path, by checking `pkg-config --variable libdir <pkg-name>`. In case, the path is incorrectly displayed, modify the file `pkg-config --path <pkg-name>` to correct the path.
40123

41-
In one of the paths reported by:
42-
```bash
43-
pkg-config --variable pc_path pkg-config
44-
```
45-
create `libsodium.pc` file, which contains:
46124

47-
```
48-
libdir=VCPKG_PATH/installed/x64-windows/bin
49-
includedir=VCPKG_PATH/installed/x64-windows/include
50125

51-
Name: libsodium
52-
VERSION: LIBSODIUM_VERSION
53-
Description: libsodium library
54-
Cflags: -I${includedir}/sodium
55-
Libs: -L${libdir} -llibsodium
126+
## Global `cabal` configuration
127+
128+
Modify the following entries to your global `cabal` config file in your
129+
`\path\to\cabal\` directory, such that they look like this:
130+
56131
```
57-
> Note that you need to replace `VCPKG_PATH` with the
58-
absolute path, where you use `vcpkg`, and `LIBSODIUM_VERSION` with the version
59-
number of `libsodium` which was installed on your system. Please verify that
60-
the paths above contain
61-
`libsodium.dll` file and headers.
132+
-- extra-include-dirs:
133+
extra-lib-dirs: C:\ghcup\ghc\8.10.7\mingw\x86_64-w64-mingw32\lib
134+
extra-prog-path: C:\ghcup\bin,
135+
path\to\cabal\bin
136+
```
137+
138+
Depending on the installation procedure, the path to your `\path\to\cabal\` directory could be `C:\Users\<username>\AppData\Roaming\cabal`).
62139

63-
> Also, you cannot use `prefix=` in the `libsodium.pc` file. This might be changed for
64-
some other directory, `pkg-config` provides a switch to use the provided
65-
`prefix`, but there is no way to instruct `cabal` to do so.
140+
> Note: The instructions in this section are a workaround for a
141+
> [GHCup bug](https://github.com/msys2/MINGW-packages/issues/10837#issuecomment-1047105402).
66142
67-
## `cabal` configuration
143+
## Downloading the source code for cardano-node
68144

69-
Go to the directory, where you cloned the `cardano-node` repository and add the command below
70-
to your `cabal.project.local` file (if you don't already have it, create one):
145+
Create a working directory for your builds:
71146

147+
```bash
148+
mkdir -p ~/src
149+
cd ~/src
72150
```
73-
max-backjump: 5000
74-
reorder-goals: True
75151

76-
package cardano-crypt-praos
77-
flags: -external-libsodium-vrf
152+
Download the Cardano node sources:
78153

79-
extra-lib-dirs: "VCPKG_PATH\\installed\\x64-windows\\bin"
80-
extra-include-dirs: "VCPKG_PATH\\installed\\x64-windows\\include"
154+
```bash
155+
git clone https://github.com/input-output-hk/cardano-node.git
81156
```
82157

83-
The final step is to add `VCPKG_PATH/installed/x64-windows/bin` to the `PATH`
84-
variable. Since we are using `git-bash`, you can do this with:
158+
Change the working directory to the downloaded source code folder:
159+
160+
```bash
161+
cd cardano-node
162+
```
163+
164+
## Local `cabal` configuration
165+
166+
Add the following to the `cabal.project.local` file (if you don't already have it, create one) in the source code folder:
85167

86168
```
87-
export PATH="VCPKG_PATH/installed/x64-windows/bin:${PATH}"
169+
package lmdb
170+
extra-lib-dirs: "<lmdb-library-path> Or standard path E.g. C:\\msys64\\mingw64\\include or C:/tools/ghcup/msys64/mingw64/lib"
171+
extra-include-dirs: "<lmdb-library-path> Or standard path E.g. C:\\msys64\\mingw64\\include or C:/tools/ghcup/msys64/mingw64/include"
172+
173+
-- If pkg-config is available, use it.
174+
package HsOpenSSL
175+
flags: +use-pkg-config
176+
177+
package cardano-crypt-praos
178+
flags: -external-libsodium-vrf
88179
```
89-
*Remember to substitute `VCPKG_PATH` with the real path.*
180+
181+
## Building and installing the node
90182

91183
You can now build the node with:
92184

93185
```bash
186+
cabal update
94187
cabal build --builddir /c/dist exe:cardano-node
95188
```
96-
> Note: using `--builddir /c/dist` with a succinct directory protects you
97-
from exceeding the [maximal path
98-
size](https://docs.microsoft.com/en-us/windows/win32/fileio/maximum-file-path-limitation)
99-
on Windows (which is a limitation of the linker rather than `ghc` itself).
100189

101-
You can now verify whether the node runs:
190+
> Note: using `--builddir /c/dist` with a succinct directory protects you from exceeding the [maximal path size](https://docs.microsoft.com/en-us/windows/win32/fileio/maximum-file-path-limitation) on Windows (which is a limitation of the linker rather than `ghc` itself).
191+
192+
You can now verify whether the node runs:
102193
```bash
103194
cabal run --builddir /c/dist exe:cardano-node -- run --help
104195
```
196+
197+
## Notes
198+
199+
### Speed up compilation
200+
201+
`reorder-goals` can affect the compilation time negatively. You can disable `reorder-goals` by setting it to `False` in `cabal.project.local`.

doc/getting-started/install.md

+4-2
Original file line numberDiff line numberDiff line change
@@ -27,6 +27,8 @@ To download the source code and build it, you need the following packages and to
2727
* developer libraries for `systemd`,
2828
* developer libraries for `ncurses`,
2929
* `ncurses` compatibility libraries,
30+
* developer libraries for `openssl`,
31+
* developer libraries for `lmdb`,
3032
* the Haskell build tool `cabal`,
3133
* the GHC Haskell compiler (version `8.10.7` or above).
3234

@@ -35,14 +37,14 @@ In Redhat, Fedora, and Centos:
3537
```bash
3638
sudo yum update -y
3739
sudo yum install git gcc gcc-c++ tmux gmp-devel make tar xz wget zlib-devel libtool autoconf -y
38-
sudo yum install systemd-devel ncurses-devel ncurses-compat-libs -y
40+
sudo yum install systemd-devel ncurses-devel ncurses-compat-libs which jq openssl-devel lmdb-devel -y
3941
```
4042

4143
For Debian/Ubuntu, use the following instead:
4244

4345
```bash
4446
sudo apt-get update -y
45-
sudo apt-get install automake build-essential pkg-config libffi-dev libgmp-dev libssl-dev libtinfo-dev libsystemd-dev zlib1g-dev make g++ tmux git jq wget libncursesw5 libtool autoconf -y
47+
sudo apt-get install automake build-essential pkg-config libffi-dev libgmp-dev libssl-dev libtinfo-dev libsystemd-dev zlib1g-dev make g++ tmux git jq wget libncursesw5 libtool autoconf liblmdb-dev -y
4648
```
4749

4850
If you are using a different flavor of Linux, you will need to use the correct package manager for your platform instead of `yum` or `apt-get`, and the names of the packages you need to install might differ. On MacOSX, use the Homebrew (`brew`) installer.

0 commit comments

Comments
 (0)