Node Bootstrap

Step 0 - Cleaning Up your Previous State

You can skip this step if it is your first time running a node on Flow.

Stop your Flow node
Clear the contents of your data directory that you have previously created. The default location is /var/flow/data. The data directory contains the Flow chain state.

If you had a previous node running on mainnet 14, you’ll need to turn it off just before joining the updated network and re-start your node with the updated configuration.

Step 1 - Run Genesis Bootstrap

You will need to run this process for each node that you are operating

Download the Bootstrapping Kit

If you have downloaded the bootstrapping kit previously, ensure that you do this step again to get the latest copy of the bootstrapping kit since we have made some significant changes to it.

curl -sL -O storage.googleapis.com/flow-genesis-bootstrap/boot-tools.tar
tar -xvf boot-tools.tar

CheckSHA256

sha256sum ./boot-tools/bootstrap
03ef039014a7a598e5083b052bc7fd118ad1f4cbbd7812d3f82cb795f1c43281  ./boot-tools/bootstrap

sha256sum ./boot-tools/transit
9b75fc00f83126561a191c0c04e892dd7bfe935391ed7c69b3fdbe9b4f216043  ./boot-tools/transit

Generating your Node ID

A node ID is a 32 byte string that is unique to every node in Flow. When you register your node to stake with Flow, you will generate a unique ID by hashing your staking public key.

Generate Your Node Keys

Addresses

You will need to have a real address, for example, a CNAME pointed at the node or just the IP address e.g. example.com:3569 or 189.23.42.12:3569.

Do not include in http:// format.

If you are running multiple nodes, please ensure you have different addresses for each node.

All your current keys and Flow genesis files should be in the bootstrap folder created earlier. Please take a back up of the entire folder.

## Skip this section if this is your first time ##
# If you joined our network previously, make sure to take a backup of your previously generated keys!
cp -r /path/to/bootstrap /path/to/bootstrap.bak
#########################################################
# Generate Keys
$ mkdir ./bootstrap
# YOUR_NODE_ADDRESS: FQDN or IP address associated to your instance
# YOUR_NODE_ROLE: The Flow nodes that you wish to run, it should be ONE of the following - [ access, collection, consensus, execution, verification ]
$ ./boot-tools/bootstrap key --address \"${YOUR_NODE_ADDRESS}:3569\" --role ${YOUR_NODE_ROLE} -o ./bootstrap

Example

$./boot-tools/bootstrap key --address "198.123.42.10:3569" --role consensus  -o ./bootstrap
<nil> DBG will generate networking key
<nil> INF generated networking key
<nil> DBG will generate staking key
<nil> INF generated staking key
<nil> DBG will generate db encryption key
<nil> INF generated db encryption key
<nil> DBG assembling node information address=198.123.42.10:3569
<nil> DBG encoded public staking and network keys networkPubKey=7f31ae79017a2a58a5e59af9184f440d08885a16614b2c4e361019fa72a9a1a42bf85b4e3f9674782f12ca06afd9782e9ccf19496baed069139385b82f8f40f6 stakingPubKey=829d086b292d84de8e7938fd2fafa8f51a6e025f429291835c20e59d9e25665febf24fa59de12a4df08be7e82c5413180cc7b1c73e01f26e05344506aaca4fa9cc009dc1c33f8ba3d7c7509e86d3d3e7341b43b9bf80bb9fba56ae0b3135dd72
<nil> INF wrote file bootstrap/public-root-information/node-id
<nil> INF wrote file bootstrap/private-root-information/private-node-info_ab6e0b15837de7e5261777cb65665b318cf3f94492dde27c1ea13830e989bbf9/node-info.priv.json
<nil> INF wrote file bootstrap/private-root-information/private-node-info_5e44ad5598bb0acb44784f629e84000ffea34d5552427247d9008ccf147fb87f/secretsdb-key
<nil> INF wrote file bootstrap/public-root-information/node-info.pub.ab6e0b15837de7e5261777cb65665b318cf3f94492dde27c1ea13830e989bbf9.json
<nil> DBG will generate machine account key
<nil> INF generated machine account key
<nil> DBG assembling machine account information address=198.123.42.10:3569
<nil> INF encoded machine account public key for entry to Flow Port machineAccountPubKey=f847b8406e8969b869014cd1684770a8db02d01621dd1846cdf42fc2bca3444d2d55fe7abf740c548639cc8451bcae0cd6a489e6ff59bb6b38c2cfb83e095e81035e507b02038203e8
<nil> INF wrote file bootstrap/private-root-information/private-node-info_ab6e0b15837de7e5261777cb65665b318cf3f94492dde27c1ea13830e989bbf9/node-machine-account-key.priv.json

$tree ./bootstrap/
./bootstrap
├── private-root-information
│   └── private-node-info_ab6e0b15837de7e5261777cb65665b318cf3f94492dde27c1ea13830e989bbf9
│       ├── node-info.priv.json
│       └── node-machine-account-key.priv.json
│       └── secretsdb-key
└── public-root-information
    ├── node-id
    └── node-info.pub.ab6e0b15837de7e5261777cb65665b318cf3f94492dde27c1ea13830e989bbf9.json

3 directories, 4 files

For consensus and collection node types an additional key will be created for the Machine Account. For all other node types this will not be needed.

Machine Account Creation

If you are running a collection and consensus node, you will have an additional private key file (node-machine-account-key.priv.json) which contains the private key for your node's machine account. You can learn more about machine accounts here.

In Step 2 of this guide, when you submit a transaction to stake your node, you will need to provide the machine account public key, which can be found in the output of the previous bootstrap key command.

MachineAccountPublicKey

$./boot-tools/bootstrap key --address ${YOUR_NODE_ADDRESS} --role ${YOUR_NODE_ROLE}  -o ./bootstrap
...
<nil> DBG encoded public machine account key machineAccountPubKey=1b9c00e6f0930792c5738d3397169f8a592416f334cf11e84e6327b98691f2b72158b40886a4c3663696f96cd15bfb5a08730e529f62a00c78e2405013a6016d
<nil> INF wrote file bootstrap/private-root-information/private-node-info_ab6e0b15837de7e5261777cb65665b318cf3f94492dde27c1ea13830e989bbf9/node-machine-account-key.priv.json

Copy the machine account public key somewhere safe. You will need it in a later step.

Upload your Public Keys

If you're running multiple nodes, you only need one token - it can be used to generate multiple keys.

Please use the following pattern when generating your token mainnet-15-yourname (you can pick any name, it just helps us keep track of the most recent keys you've sent and would like us to include in the bootstrapping process)

$ ./boot-tools/transit push -b ./bootstrap -t ${TOKEN} -r ${YOUR_NODE_ROLE}

Running push
Generating keypair
Uploading ...
Uploaded 400 bytes

Step 2 - Stake Your Node

Stake your node via Flow Port

The node details (Node ID, Network Address, Networking Key and Staking Key) that need to be submitted when staking the node on Flow Port, can be found in the file: ./bootstrap/public-root-information/node-info.pub.<node-id>.json.

Example

$cat ./bootstrap/public-root-information/node-info.pub.39fa54984b8eaa463e129919464f61c8cec3a4389478df79c44eb9bfbf30799a.json
{
  "Role": "consensus",
  "Address": "198.123.42.10:3569",
  "NodeID": "39fa54984b8eaa463e129919464f61c8cec3a4389478df79c44eb9bfbf30799a",
  "Stake": 1000,
  "NetworkPubKey": "d92e3d5880abe233cf9fe9104db34bbb31251468a541454722b3870c04156a1b0504aef443bcaad124b997384b8fe7052847ce1e6189af1392d865e6be69835b",
  "StakingPubKey": "917826e018f056a00b778a58ae83054906957ffd4b6f1b7da083551f7a9f35e02b76ace50424ed7d2c9fc69207a59f0f08a031048f5641db94e77d0648b24d150dedd54bab7cd44b4aa60cfd54be418647b0b3965f8ae54c0bcb48ae9d705162"
}

If you are running a collection or consensus node, you will need to provide an additional field Machine Account Public Key. This value is found in the output of the bootstrap key command from Step 1.

Finalize Machine Account Setup

If you are not running a collection or consensus node, you can skip this step.

You will now need to use the bootstrap utility to run machine-account with the created address to finalize the set up of your Machine account.

$ ./boot-tools/bootstrap machine-account --address ${YOUR_MACHINE_ACCOUNT_ADDRESS} -o ./bootstrap

Example

$ ./boot-tools/bootstrap machine-account --address 0x1de23de44985c7e7 -o ./bootstrap
<nil> INF read machine account private key json
<nil> DBG encoded public machine account key machineAccountPubKey=2743786d1ff1bf7d7026d693a774210eaa54728343859baab62e2df7f71a370651f4c7fd239d07af170e484eedd4f3c2df47103f6c39baf2eb2a50f67bbcba6a
<nil> INF wrote file bootstrap/private-root-information/private-node-info_6f6e98c983dbd9aa69320452949b81abeab2ac591a247f55f19f4dbf0b477d26/node-machine-account-info.priv.json

$tree ./bootstrap/
./bootstrap
├── private-root-information
│   └── private-node-info_d60bd55ee616c5c297cae1d5cfb7f65e7e04014d9c4abe595af2fd83f3cfe160
│       ├── node-info.priv.json
│       ├── node-machine-account-info.priv.json
│       └── node-machine-account-key.priv.json
│       └── secretsdb-key
└── public-root-information
    ├── node-id
    └── node-info.pub.d60bd55ee616c5c297cae1d5cfb7f65e7e04014d9c4abe595af2fd83f3cfe160.json

3 directories, 5 files

After running this step, you should see the node-machine-account-info.priv.json file in your bootstrap directory as shown above.

Verify Machine Account Setup

After finalizing your machine account setup, you should verify its correctness with the check-machine-account command:

CheckMachineAccount

$ ./boot-tools/bootstrap check-machine-account --access-address access.mainnet.nodes.onflow.org:9000 -o ./bootstrap
<nil> DBG read machine account info from disk hash_algo=SHA3_256 key_index=0 machine_account_address=0x284463aa6e25877c machine_account_pub_key=f847b84051bad4512101640772bf5e05e8a49868d92eaf9ebed41030881d95485769afd28653c5c53216cdcda4554384bb3ff6396a2ac04842422d55f0562496ad8d952802038203e8 signing_algo=ECDSA_P256
<nil> DBG checking machine account configuration... machine_account_address=0x284463aa6e25877c role=consensus
<nil> DBG machine account balance: 0.10000000
<nil> INF 🤖 machine account is configured correctly

This command will detect and provide information about common misconfigurations, or confirm that the machine account is configured correctly.

Step 3 - Getting Ready to Start your Flow Node

Before proceeding to Step 3, ensure that Step 1 was completed successfully with the bootstrap directory handy (default: /var/flow/bootstrap)

Expose TCP port 3569 on your firewall for both ingress and egress traffic

If you are running Flow Access node, Collection node or an Execution node you will also need to expose TCP port 9000.

Optionally, for any node type, if you would like to expose the node metrics you can open port 8000.

Create the data directory if you have not previously created it and/or removed a previous version of it (default: /var/flow/data)

You can choose to run your Flow Node either using docker or have systemd to manage Flow node. Please choose only one of these two options

Systemd

If you are running your Flow Node via Docker, you can skip this step and go to the Docker section below.

Ensure that you pulled the latest changes from flow-go repository via git

## Clone the repo if you haven't already done so
git clone https://github.com/onflow/flow-go

## Get latest changes
cd flow-go
git pull origin master

Copy your respective systemd unit file to: /etc/systemd/system
Create directory sudo mkdir /etc/flow
Copy the runtime-conf.env file to: /etc/flow/
Enable your service sudo systemctl enable YOUR-FLOW-NODE.service

Wait for the Flow team announcement before you proceed to Step 4

Step 4 - Start Your Flow Node

Once you receive an announcement from the Flow team that the network has been booted (via Discord), you will need to fetch the genesis info, update your runtime configuration and then boot your Flow node up!

The Flow team will provide you a new token PULL_TOKEN to pull the genesis info from. (Please do NOT use the original Token that you used in Step 1) The YOUR_NODE_TYPE must be the same ones that you used on step 1 - generating your node keys

Run the transit script to fetch the new genesis info: ./boot-tools/transit pull -b ./bootstrap -t ${PULL_TOKEN} -r ${YOUR_NODE_TYPE}

Example

$ ./boot-tools/transit pull -b ./bootstrap -t mainnet-15  -r consensus
Transit script Commit: a9f6522855e119ad832a97f8b7bce555a163e490
2020/11/25 01:02:53 Running pull
2020/11/25 01:02:53 Downloading bootstrap/public-root-information/node-infos.pub.json
2020/11/25 01:02:54 Downloading bootstrap/public-root-information/root-protocol-snapshot.json
2020/11/25 01:02:54 Downloading bootstrap/random-beacon.priv.json.39fa54984b8eaa463e129919464f61c8cec3a4389478df79c44eb9bfbf30799a.enc
2020/11/25 01:02:54 SHA256 of the root block is: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855

$ tree ./bootstrap/
  ./bootstrap/
  ├── private-root-information
  │   └── private-node-info_39fa54984b8eaa463e129919464f61c8cec3a4389478df79c44eb9bfbf30799a
  │       └── node-info.priv.json
  ├── public-root-information
  │   ├── node-id
  │   ├── node-info.pub.39fa54984b8eaa463e129919464f61c8cec3a4389478df79c44eb9bfbf30799a.json
  │   ├── node-infos.pub.json
  │   └── root-protocol-snapshot.json
  └── random-beacon.priv.json.39fa54984b8eaa463e129919464f61c8cec3a4389478df79c44eb9bfbf30799a

Pull the latest changes from flow-go repository
Get your node-id, you can find it at /path/to/bootstrap/public-genesis-information/node-id
Update the FLOW_GO_NODE_ID inside runtime-conf.env to the node-id that you got from the previous step
Start your Flow node via docker or systemd

Docker

Please make sure you are running Docker version 19 (`docker --version`). There is currently an issue with Docker version 20.

You can skip this section if you are running via systemd

Update your environment variables: source /path/to/runtime-conf.env
Run the following docker command based on the Flow node that you are running:

Remember: update your docker volume mounting path

You'll need to update the path for data and bootstrap volumes in the following commands. Those paths are from step 2.2 and 2.3a.

Access

docker run --rm \
	-v /path/to/bootstrap:/bootstrap:ro \
	-v /path/to/data:/data:rw \
	--name flow-go \
	--network host \
	gcr.io/flow-container-registry/access:v0.23.3 \
	--nodeid=${FLOW_GO_NODE_ID} \
	--bootstrapdir=/bootstrap \
	--datadir=/data/protocol \
	--secretsdir=/data/secrets \
	--rpc-addr=0.0.0.0:9000 \
	--http-addr=0.0.0.0:8000 \
	--collection-ingress-port=9000 \
	--script-addr=${FLOW_NETWORK_EXECUTION_NODE} \
	--bind 0.0.0.0:3569 \
	--loglevel=error

Collection

docker run --rm \
	-v /path/to/bootstrap:/bootstrap:ro \
	-v /path/to/data:/data:rw \
	--name flow-go \
	--network host \
	gcr.io/flow-container-registry/collection:v0.23.3 \
	--nodeid=${FLOW_GO_NODE_ID} \
	--bootstrapdir=/bootstrap \
	--datadir=/data/protocol \
	--secretsdir=/data/secrets \
	--ingress-addr=0.0.0.0:9000 \
	--bind 0.0.0.0:3569 \
	--loglevel=error

Consensus

docker run --rm \
	-v /path/to/bootstrap:/bootstrap:ro \
	-v /path/to/data:/data:rw \
	--name flow-go \
	--network host \
	gcr.io/flow-container-registry/consensus:v0.23.3 \
	--nodeid=${FLOW_GO_NODE_ID} \
	--bootstrapdir=/bootstrap \
	--datadir=/data/protocol \
	--secretsdir=/data/secrets \
	--bind 0.0.0.0:3569 \
	--loglevel=error

Execution

docker run --rm \
	-v /path/to/bootstrap:/bootstrap:ro \
	-v /path/to/data:/data:rw \
	--name flow-go \
	--network host \
	gcr.io/flow-container-registry/execution:v0.23.3 \
	--nodeid=${FLOW_GO_NODE_ID} \
	--bootstrapdir=/bootstrap \
	--datadir=/data/protocol \
	--secretsdir=/data/secrets \
	--triedir=/data/execution \
	--rpc-addr=0.0.0.0:9000 \
	--bind 0.0.0.0:3569 \
	--loglevel=error

Verification

docker run --rm \
	-v /path/to/bootstrap:/bootstrap:ro \
	-v /path/to/data:/data:rw \
	--name flow-go \
	--network host \
	gcr.io/flow-container-registry/verification:v0.23.3 \
	--nodeid=${FLOW_GO_NODE_ID} \
	--bootstrapdir=/bootstrap \
	--datadir=/data/protocol \
	--secretsdir=/data/secrets \
	--bind 0.0.0.0:3569 \
	--loglevel=error

Systemd

You can skip this section if you are running via docker

Check that your runtime-conf.env is at /etc/flow/runtime-conf.env
Update your environment variables: source /etc/flow/runtime-conf.env
Start your service: sudo systemctl start flow

Step 5 - Verify if your Node is Running

Here are a few handy commands that you can use to check if your Flow node is up and running

Docker

To get Flow logs: sudo journalctl -u flow-YOUR_ROLE
To get the status: sudo docker ps

$ sudo docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
1dc5d43385b6 gcr.io/flow-container-registry/verification:candidate8 \"/bin/app --nodeid=4…\" 30 hours ago Up 30 hours flow-go

Systemd

To get Flow logs: sudo journalctl -u flow-YOUR_ROLE
To get your Flow node status: sudo systemctl status flow

● flow-verification.service - Flow Access Node running with Docker
Loaded: loaded (/etc/systemd/system/flow-verification.service; enabled; vendor preset: enabled)
Active: active (running) since Wed 2020-05-20 18:18:13 UTC; 1 day 6h ago
Process: 3207 ExecStartPre=/usr/bin/docker pull gcr.io/flow-container-registry/verification:${FLOW_GO_NODE_VERSION} (code=exited, status=0/SUCCESS)
Main PID: 3228 (docker)
Tasks: 10 (limit: 4915)
Memory: 33.0M
CGroup: /system.slice/flow-verification.service
        └─3228 /usr/bin/docker run --rm -v /var/flow/bootstrap:/bootstrap:ro -v /var/flow/data:/data:rw --rm --name flow-go --network host gcr.io/flow-container-registry/verification:candidate8 --nodeid=489f8a4513d5bd8b8b093108fec00327b683db545b37b4ea9153f61b2c0c49dc --bootstrapdir=/bootstrap --datadir=/data/protocol --alpha=1 --bind 0.0.0.0:3569 --loglevel=error

Step 6 (optional) - Monitoring and Metrics

This is intended for operators who would like to see what their Flow nodes are currently doing. Head over to Monitoring Node Health to get setup.

Node Status

The metrics for the node should be able to provide a good overview of the status of the node. If we want to get a quick snapshot of the status of the node, and if it's properly participating in the network, you can check the consensus_compliance_finalized_height or consensus_compliance_sealed_height metric, and ensure that it is not zero and strictly increasing.

curl localhost:8080/metrics | grep consensus_compliance_sealed_height

# HELP consensus_compliance_sealed_height the last sealed height
# TYPE consensus_compliance_sealed_height gauge
consensus_compliance_sealed_height 1.132054e+06

Common Issues

Error: cannot create connection

20T18:34:21Z","message":"could not create connection"}
{"level":"error","node_role":"consensus","node_id":"6d3fac8675a1df96f4bb7a27305ae531b6f4d0d2bc13a233e37bb07ab6b852dc","target":"QmVcSQaCdhmk1CMeMN7HTgGiUY1i2KqgVE2vvEmQXK4gAA","error":"failed to dial : all dials failed
  * [/ip4/155.138.151.101/tcp/3569] dial tcp4 155.138.151.101:3569: connect: connection refused","retry_attempt":2,"time":"2020-05-20T18:34:21Z","message":"could not create connection"}

This error is OK. Your fellow node operators have not turned on/joined the network yet. So no need to worry about it!

Flow Node Not Booting Up

If your Flow node is not able to boot up, or it exits right after it boots up. You will need to do a clean up of your state.

After cleaning up the state try booting it up again. If the problem persists, message a member from the Flow team on Discord.

MAINNET:
Loading...

TESTNET:
Loading...

Node Bootstrap

Running your own flow Node

Step 0 - Cleaning Up your Previous State

Step 1 - Run Genesis Bootstrap

Download the Bootstrapping Kit

Generating your Node ID

Generate Your Node Keys

Addresses

Machine Account Creation

Upload your Public Keys

Step 2 - Stake Your Node

Finalize Machine Account Setup

Verify Machine Account Setup

Step 3 - Getting Ready to Start your Flow Node

Systemd

Step 4 - Start Your Flow Node

Docker

Access

Collection

Consensus

Execution

Verification

Systemd

Step 5 - Verify if your Node is Running

Docker

Systemd

Step 6 (optional) - Monitoring and Metrics

Node Status

Common Issues

Error: cannot create connection

Flow Node Not Booting Up