Multi-Node Cluster Deployment

Deploy Garage as a distributed cluster with data replication

This guide explains how to deploy Garage as a multi-node cluster with data replication. This is the recommended setup for production environments, providing high availability and fault tolerance.

Overview

A production Garage cluster requires at least 3 nodes to enable three-way replication, the safest and most available mode. This ensures your data remains accessible even if one node fails.

Prerequisites

Before deploying a multi-node cluster:

• At least 3 machines with sufficient storage space
• Direct IP connectivity between all nodes (IPv4 or IPv6)
• For nodes behind NAT, consider:
  • IPv6 for end-to-end connectivity
  • Mesh VPN like Nebula or Yggdrasil

Architecture Planning

Sample Infrastructure

Consider this example cluster layout:

Location	Name	IP Address	Disk Space	Zone	Capacity
Paris	Mercury	fc00:1::1	1 TB	par1	1T
Paris	Venus	fc00:1::2	2 TB	par1	2T
London	Earth	fc00:B::1	2 TB	lon1	2T
Brussels	Mars	fc00:F::1	1.5 TB	bru1	1.5T

Garage stores three copies of data on nodes at different locations. In this example, the usable capacity is 1.5 TB (limited by Brussels).

Best Practices

Storage Configuration

• Metadata directory: Store on SSD for fast access to frequently-used metadata
• Data directory: Store on HDD for large, slower storage
• Multiple HDDs: Supported natively without RAID

Filesystem Recommendations

• Data directory: Use XFS for best performance (EXT4 has inode limitations)
• Metadata directory: Use BTRFS or ZFS for checksumming and integrity
• Database engine: LMDB (default) for performance, or SQLite for robustness

Performance Tuning

• For fast networking and large files: Set block_size = "10M"
• Enable automatic snapshots: metadata_auto_snapshot_interval = "6h"
• Use compression: compression_level = 2

Capacity Planning

• Balance capacity across zones for optimal utilization
• Garage replicates across zones, not just nodes
• Ensure similar capacity in each zone to avoid under-utilization

Installation

Get Docker Image

Pull the Garage Docker image on each node:

docker pull dxflrs/garage:v2.2.0

Always use a fixed version tag (e.g., v2.2.0) instead of latest for production deployments.

Directory Structure

Create the following directories on each node:

sudo mkdir -p /etc/garage
sudo mkdir -p /var/lib/garage/meta
sudo mkdir -p /var/lib/garage/data

Configuration

1. Generate RPC Secret

   Generate a shared secret for secure node communication:

   openssl rand -hex 32
   

   Use the same value for all nodes in the cluster.

2. Create Configuration File

   Create /etc/garage.toml on each node with the following template:

   metadata_dir = "/var/lib/garage/meta"
   data_dir = "/var/lib/garage/data"
   db_engine = "lmdb"
   metadata_auto_snapshot_interval = "6h"
   
   replication_factor = 3
   
   compression_level = 2
   
   rpc_bind_addr = "[::]:3901"
   rpc_public_addr = "<this node's public IP>:3901"
   rpc_secret = "<RPC secret>"
   
   [s3_api]
   s3_region = "garage"
   api_bind_addr = "[::]:3900"
   root_domain = ".s3.garage"
   
   [s3_web]
   bind_addr = "[::]:3902"
   root_domain = ".web.garage"
   index = "index.html"
   

   • Replace <this node's public IP> with each node’s actual IP address
   • Use the same rpc_secret on all nodes
   • Adjust replication_factor based on your cluster size (3 is recommended)

Starting Nodes

Using Docker

Start Garage on each node:

docker run \
  -d \
  --name garaged \
  --restart always \
  --network host \
  -v /etc/garage.toml:/etc/garage.toml \
  -v /var/lib/garage/meta:/var/lib/garage/meta \
  -v /var/lib/garage/data:/var/lib/garage/data \
  dxflrs/garage:v2.2.0

Using --network host allows direct node-to-node communication, especially important for IPv6.

Using Docker Compose

Alternatively, create a docker-compose.yml file:

version: "3"
services:
  garage:
    image: dxflrs/garage:v2.2.0
    network_mode: "host"
    restart: unless-stopped
    volumes:
      - /etc/garage.toml:/etc/garage.toml
      - /var/lib/garage/meta:/var/lib/garage/meta
      - /var/lib/garage/data:/var/lib/garage/data

Start with:

docker-compose up -d

Cluster Formation

1. Verify Nodes are Running

   On each node, check that Garage is running:

   docker exec garaged /garage status
   

   Or create an alias:

   alias garage="docker exec -ti garaged /garage"
   garage status
   

   Initially, each node will show NO ROLE ASSIGNED.

2. Get Node Identifiers

   On each node, get its unique identifier:

   # Mercury
   garage node id
   # Output: 563e1ac825ee3323aa441e72c26d1030d6d4414aeb3dd25287c531e7fc2bc95d@[fc00:1::1]:3901
   
   # Venus
   garage node id
   # Output: 86f0f26ae4afbd59aaf9cfb059eefac844951efd5b8caeec0d53f4ed6c85f332@[fc00:1::2]:3901
   
   # Earth
   garage node id
   # Output: 68143d720f20c89d...
   
   # Mars
   garage node id
   # Output: 212f7572f0c89da9...
   
3. Connect Nodes Together

   From any node, connect to the other nodes:

   # From Venus, connect to Mercury
   garage node connect 563e1ac825ee3323aa441e72c26d1030d6d4414aeb3dd25287c531e7fc2bc95d@[fc00:1::1]:3901
   
   # Connect to other nodes
   garage node connect 68143d720f20c89d...@[fc00:B::1]:3901
   garage node connect 212f7572f0c89da9...@[fc00:F::1]:3901
   

   You don’t need to connect every node to every other node. Nodes discover each other transitively.

4. Verify Cluster Connectivity

   Run garage status on any node:

   garage status
   

   You should see all nodes listed:

   ==== HEALTHY NODES ====
   ID                  Hostname  Address           Tag                   Zone  Capacity
   563e1ac825ee3323…   Mercury   [fc00:1::1]:3901  NO ROLE ASSIGNED
   86f0f26ae4afbd59…   Venus     [fc00:1::2]:3901  NO ROLE ASSIGNED
   68143d720f20c89d…   Earth     [fc00:B::1]:3901  NO ROLE ASSIGNED
   212f7572f0c89da9…   Mars      [fc00:F::1]:3901  NO ROLE ASSIGNED
   

Configuring Cluster Layout

1. Assign Node Roles

   Assign capacity, zone, and tags to each node:

   garage layout assign 563e -z par1 -c 1T -t mercury
   garage layout assign 86f0 -z par1 -c 2T -t venus
   garage layout assign 6814 -z lon1 -c 2T -t earth
   garage layout assign 212f -z bru1 -c 1.5T -t mars
   

   Parameters:

   • -z: Zone identifier (e.g., datacenter location)
   • -c: Storage capacity available to Garage
   • -t: Optional tags for node identification
2. Review Layout

   Review the proposed layout:

   garage layout show
   

   Carefully check the output before applying.

3. Apply Layout

   Apply the cluster layout:

   garage layout apply
   

   Garage will start rebalancing data across the cluster.

Managing the Cluster

Creating Buckets and Keys

# Create a bucket
garage bucket create my-bucket

# Create an API key
garage key create my-app-key

# Grant permissions
garage bucket allow --read --write --owner my-bucket --key my-app-key

Monitoring

# Check cluster status
garage status

# View cluster layout
garage layout show

# List buckets
garage bucket list

# List keys
garage key list

Upgrading

Always read the upgrade documentation for your specific version before upgrading. Major version upgrades may require additional steps.

To upgrade a node:

# Stop the container
docker stop garaged
docker rm garaged

# Pull new version
docker pull dxflrs/garage:v2.3.0

# Start with new version
docker run \
  -d \
  --name garaged \
  --restart always \
  --network host \
  -v /etc/garage.toml:/etc/garage.toml \
  -v /var/lib/garage/meta:/var/lib/garage/meta \
  -v /var/lib/garage/data:/var/lib/garage/data \
  dxflrs/garage:v2.3.0

Upgrade nodes one at a time, waiting for the cluster to stabilize between upgrades.

Troubleshooting

Nodes Cannot Connect

• Verify firewall rules allow port 3901 (RPC)
• Check rpc_public_addr is set correctly on all nodes
• Ensure all nodes use the same rpc_secret

Layout Changes Not Taking Effect

# Check if layout changes are staged
garage layout show

# Apply staged changes
garage layout apply

Checking Logs

# View container logs
docker logs garaged

# Follow logs
docker logs -f garaged

# With increased verbosity
docker exec garaged /garage server # with RUST_LOG=garage=debug

Next Steps

• Configure systemd service for non-Docker deployments
• Set up monitoring for your cluster
• Configure reverse proxy for external access
• Learn about Kubernetes deployment