Example YugabyteDB Setup

[LexLuthr]

We invite all Boost users to please add their YugabyteDB deployment experience and details including all configuration parameters to this discussion. This discussion will serve as a reference architecture for other users.

Please make sure to omit any IP and other sensitive information

A few key points to keep in mind while deploying:

1. Cassandra client does not have SSL enabled. So, if you enable security on Cassandra in YugabyteDB then you might see below error.
   Error: starting yugabyte store: creating cassandra keyspace idx: creating yugabyte cluster: gocql: unable to create session: unable to discover protocol version: EOF
2. If SSL is enabled for Postgres API then you would need to update your connect-string to enable SSL as well.
   postgresql://<username>:<password>@<yugabytedb>:5433?sslmode=disable
3. If SSL is not enabled for Postgres then please omit the SSL part from the connecting string.
   postgresql://<username>:<password>@<yugabytedb>:5433
4. Cassandra queries have a timeout of 60 seconds by default. We don't expect any query to take longer than this. If you are seeing CQL timeouts with 60 seconds, the problem is likely on the server side.
5. Please ensure that YugabyteDB does not suffer from resource exhaustions. If enough CPU and Memory are not available, then you will see a degraded performance as well as frequent timeouts. Extra care must be taken when deploying with other services on a single machine.
6. We don't recommend deploying with docker containers for production use. But, if you insist on deploying YugabyteDB as a container then please make sure that Directory/Filesystem used for YugabyteDB is not shared by any other process. This can result in lower IOPS for YugabyteDB and will cause performance degradation.
7. If YugabyteDB is deployed on a Virtual Machine (VM), then please ensure that the backing storage is NVME or SSD. Please do not use any other kind of storages.
8. The filesystem used to store the YugabyteDB data should be XFS.

[LexLuthr]

FilCollins (Boost team's test miner)

Deployment Platform: Docker
Storage Type: NVME
Machine details: 64 CPUs, 512 GiB memory
OS: "Ubuntu 20.04.5 LTS"
Docker command: sudo docker run -d --name yugabyte -p7000:7000 -p9000:9000 -p5433:5433 -p9042:9042 -v /home/nonsense/yb-home:/home/yugabyte yugabytedb/yugabyte:latest bin/yugabyted start --base_dir=/home/yugabyte/yb_data --daemon=false
Boostd-data command: boostd-data run yugabyte --hosts 127.0.0.1 --connect-string="postgresql://postgres:[email protected]:5433?sslmode=disable" --addr 0.0.0.0:8044

[SBudo]

I originally started my yb-tserver with no parameters (all defaults) expect the data directory and binding addresses as follow:
./bin/yb-tserver --tserver_master_addrs 10.10.10.1:7100,10.10.10.2.:7100,10.10.10.3:7100 --rpc_bind_addresses 10.10.10.1:9100 --enable_ysql --pgsql_proxy_bind_address 10.10.10.1:5433 --cql_proxy_bind_address 10.10.10.1:9042 --fs_data_dirs "/mnt/data1/tserver-data"

...but that gave me an SSL error message:

Setting up connection and creating data structures... this might take a minute...
Error: starting yugabyte store: running postgres migrations: getting number of rows in PieceTracker table: pq: SSL is not enabled on the server

To try to fix this, I thought it was necessary to enable SSL, so I went down the path of enabling Client encryption (as per the Yugabyte documentation link: https://docs.yugabyte.com/preview/secure/tls-encryption/client-to-server/) and then ran the yb-tserver with these additional parameters:

--use_client_to_server_encryption=true
--allow_insecure_connections=false
--ysql_enable_auth=true
--use_cassandra_authentication=true

This generated a new error message:

Setting up connection and creating data structures... this might take a minute...
Error: starting yugabyte store: creating cassandra keyspace idx: creating yugabyte cluster: gocql: unable to create session: unable to discover protocol version: EOF

Only by starting the yb-tserver with only the two paramaters below would it start to work:

--use_client_to_server_encryption=true
--ysql_enable_auth=true

(leaving the other parameters to their default values)

The final yb-tserver startup command then looked like this:
./bin/yb-tserver --tserver_master_addrs 10.10.10.1:7100,10.10.10.2.:7100,10.10.10.3:7100 --rpc_bind_addresses 10.10.10.1:9100 --enable_ysql --pgsql_proxy_bind_address 10.10.10.1:5433 --cql_proxy_bind_address 10.10.10.1:9042 --fs_data_dirs "/mnt/data1/tserver-data --use_client_to_server_encryption=true --ysql_enable_auth=true"

I used the "default" migration command on the boost server as per the boost documentation:
./migrate-lid yugabyte --hosts 10.10.10.1,10.10.10.2,10.10.10.3 --connect-string="postgresql://yugabyte:[email protected]:5433" dagstore

[Shekelme]

Our setup is pretty simple. Three instances of the YB-Master+YB-TServer per cluster, with one set directly on the miner's server.

Deployment Platform: manual deployment (+systemd units)
Storage Type: PCI-e Gen4 NVME
Machine details: AMD Ryzen Threadripper PRO 3955WX (16C/32T) or 3945WX (12C/24T), 256 or 512 GB DDR4-3200 RAM
OS: Ubuntu 22.04.3 LTS
boostd: version 2.1.0-rc1+mainnet+git.ba2b53c

master.conf ( /home/<user>/yugabyte-2.18.2.1/master.conf):

--master_addresses=192.168.11.91:7100,192.168.11.14:7100,192.168.11.101:7100
--rpc_bind_addresses=192.168.11.91:7100 #IP of current server here
--fs_data_dirs=/media/yugabyte
--placement_cloud=fcmsk
--placement_region=fcmsk-len91 #miner's hostname here
--placement_zone=len91 #hostname

tserver.conf (/home/<user>/yugabyte-2.18.2.1/tserver.conf)

--tserver_master_addrs=192.168.11.91:7100,192.168.11.14:7100,192.168.11.101:7100
--rpc_bind_addresses=192.168.11.91:9100 #IP of current server here
--enable_ysql
--pgsql_proxy_bind_address=192.168.11.91:5433 #IP of current server here
--cql_proxy_bind_address=192.168.11.91:9042 #IP of current server here
--fs_data_dirs=/media/yugabyte
--placement_cloud=fcmsk
--placement_region=fcmsk-len91 #miner's hostname here
--placement_zone=len91 #hostname

systemd units files:

YB-Master:

sudo nano /etc/systemd/system/yb-master.service

Description=YB-Master
After=network-online.target
Requires=network-online.target

[Service]
ExecStart=/home/<user>/yugabyte-2.18.2.1/bin/yb-master --flagfile /home/<user>/yugabyte-2.18.2.1/master.conf
Restart=always
RestartSec=5
TimeoutStartSec=20
TimeoutStopSec=20
User=<user>
Group=<user's group>
Type=simple

LimitCPU=infinity
LimitFSIZE=infinity
LimitDATA=infinity
LimitSTACK=8388608
LimitCORE=infinity
LimitRSS=infinity
LimitNOFILE=1048576
LimitAS=infinity
LimitNPROC=12000
LimitMEMLOCK=64
LimitLOCKS=infinity
LimitSIGPENDING=119934
LimitMSGQUEUE=819200
LimitNICE=0
LimitRTPRIO=0

[Install]
WantedBy=multi-user.target

systemctl daemon-reload
sudo systemctl restart yb-master
sudo systemctl enable yb-master

YB-TServer:

sudo nano /etc/systemd/system/yb-tserver.service

Description=YB-TServer
After=network-online.target
After=yb-master.service
Requires=network-online.target
Requires=yb-master.service

[Service]
ExecStart=/home/<user>/yugabyte-2.18.2.1/bin/yb-tserver --flagfile /home/<user>/yugabyte-2.18.2.1/tserver.conf
Restart=always
RestartSec=5
TimeoutStartSec=20
TimeoutStopSec=20
User=<user>
Group=<user's group>
Type=simple

LimitCPU=infinity
LimitFSIZE=infinity
LimitDATA=infinity
LimitSTACK=8388608
LimitCORE=infinity
LimitRSS=infinity
LimitNOFILE=1048576
LimitAS=infinity
LimitNPROC=12000
LimitMEMLOCK=64
LimitLOCKS=infinity
LimitSIGPENDING=119934
LimitMSGQUEUE=819200
LimitNICE=0
LimitRTPRIO=0

[Install]
WantedBy=multi-user.target

systemctl daemon-reload
sudo systemctl start yb-tserver
sudo systemctl enable yb-tserver

At last boostd-data unit:

sudo nano /etc/systemd/system/boostd-data.service

[Unit]
Description=boostd-data
After=network-online.target
After=yb-tserver.service
Requires=network-online.target
Wants=yb-tserver.service

[Service]
ExecStart=/usr/local/bin/boostd-data run yugabyte --hosts 192.168.11.91,192.168.11.14,192.168.11.101 --connect-string="postgresql://yugabyte:[email protected]:5433?sslmode=disable" --addr 0.0.0.0:8044
Restart=always
RestartSec=5
TimeoutStartSec=20
TimeoutStopSec=20
User=<user>
Group=<user's group>
Type=simple

[Angelo-gh3990]

Quick and easy deployment for an single instance, easy upgrade and management if comfortable with docker:

I'm testing the new YugabyteDB install and testing it. I'm running as a local docker image on 1 server:

$ docker pull yugabytedb/yugabyte:latest

And then after :

$ docker run -d --name yugabyte -p7000:7000 -p9000:9000 -p9042:9042 -p5433:5433 yugabytedb/yugabyte:latest bin/yugabyted start --daemon=false

Seems to run fine and is up in less then 10sec, admin page seems to load fine.
Just dropping it here if anybody can use it to test.
define BOOST_PATH

./migrate-lid yugabyte \
  --hosts localhost \
  --connect-string="postgresql://yugabyte:yugabyte@localhost:5433" \
  dagstore

[piknikSteven2021]

I notice that a lot of the PostgreSQL connect strings here are connecting to a singular tserver address of a Yugabyte cluster. Keep in mind that if that one tserver you are using in your connect string goes offline, your boost node has now lost access to the cluster. In PostgreSQL HA clusters (which YugabyteDB is, with a bunch of features on top) - you would typically have a pacemaker virtual IP address to reference in your connect string. We can't use pacemaker like most HA PostgreSQL solutions (or it would be non-trivial to do so, and why would you? YugabyteDB already handles syncing). It would be highly recommended that you put a load balancer in front of your tablet servers to keep your connect string from being non-redundant. This can be achieved with either keepalived or haproxy.

Note that this only applies to those who are setting up a YugabyteDB cluster.

[Shekelme]

to keep your connect string from being non-redundant

Can it be implemented in boostd-data itself?

[piknikSteven2021]

***@***.***:3333,198.51.100.33:5555 You can connect to multiple postgres nodes in this format.

…

On Sat, Nov 4, 2023, 12:02 AM Shekelme ***@***.***> wrote: to keep your connect string from being non-redundant Can it be implemented in boostd-data itself? — Reply to this email directly, view it on GitHub <#1797 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AWQ3UTVOBSCOA4ORJZWPL6DYCXR7ZAVCNFSM6AAAAAA62PPILCVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM3TINZSHAZDE> . You are receiving this because you commented.Message ID: ***@***.***>

[LexLuthr]

I am checking with Yugabyte about this. There is some bug there I think. It is not recognising the parameter.

[LexLuthr]

@Shekelme Can you test with #1809? The branch should have the fix.

[Shekelme]

It has started and seems to be working.

[LexLuthr]

Great! Fix is already merged and will be part of v2.1.0 stable release

[Shekelme]

I hope this is not a very serious issue:

Nov 08 18:05:34 lenovo92 boostd-data[3784458]: 2023-11-08T18:05:34.753+0300        INFO        boostd-data        cmd/run.go:179        Started boostd-data yugabyte service on address 0.0.0.0:8044
Nov 08 20:06:34 lenovo92 boostd-data[3784458]: 2023/11/08 20:06:34 connection count for 192.168.11.92 going negative!
Nov 08 21:06:03 lenovo92 boostd-data[3784458]: 2023/11/08 21:06:03 connection count for 192.168.11.92 going negative!
Nov 08 21:06:42 lenovo92 boostd-data[3784458]: 2023/11/08 21:06:42 connection count for 192.168.11.92 going negative!

[piknikSteven2021]

It is possible that yugabyte's version of postgres doesn't support it, in which case you need to setup a load balancer instead.

…

On Sun, Nov 5, 2023, 12:18 AM Shekelme ***@***.***> wrote: I'm afraid I can't. Nov 05 10:15:53 lenovo92 boostd-data[1681018]: Error: starting yugabyte store: starting local index directory service: running postgres migrations: getting number of rows in PieceTracker table: dial tcp: lookup 192.168.11.92:5433,192.168.11.11:5433,192.168.11.20:5433: no such host My boostd-data systemd unit looks like [Unit] Description=boostd-data After=network-online.target After=yb-tserver.service Requires=network-online.target Wants=yb-tserver.service [Service] ExecStart=/tmp/boostv2/boostd-data run yugabyte --hosts 192.168.11.92,192.168.11.11,192.168.11.20 ***@***.***:5433,192.168.11.11:5433,192.168.11.20:5433" --addr 0.0.0.0:8044 Restart=always RestartSec=5 TimeoutStartSec=20 TimeoutStopSec=20 User=admfc Group=admfc Type=simple [Install] WantedBy=multi-user.target Or did I misunderstand something? — Reply to this email directly, view it on GitHub <#1797 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AWQ3UTUV4S6EUYC7VDBQIBDYC44VVAVCNFSM6AAAAAA62PPILCVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM3TINZXG42DA> . You are receiving this because you commented.Message ID: ***@***.***>