Hermes IBC relayer setup is well documented for the happy path: install the binary, run hermes config auto, add your keys, start relaying. What the documentation does not emphasize is everything that happens after that, the gas configuration that the auto-generated config leaves at unsafe defaults, the monitoring that tells you a packet is stuck before users complain, the fee management that determines whether your relayer operates at a loss, and the light client expiry that freezes a channel if you are not watching it.
Quick answer: A production Hermes IBC relayer setup requires four things beyond the basic install: a manually corrected config.toml (the auto-generated default_gas and max_gas values are placeholders that must be reset), a funded and monitored relayer wallet per chain, Prometheus telemetry with tx_confirmation = true enabled, and alerting on packet backlog and light client expiry. The install takes 20 minutes. The production hardening is where the operational work lives, and it is what this guide covers.
What changed: Hermes is the reference Rust-based IBC relayer maintained by Informal Systems. With the move to ibc-go v10 and IBC v2 (covered in our Cosmos IBC tutorial), Hermes 1.10+ supports relaying for both IBC Classic and IBC v2 protocol versions simultaneously. The dynamic gas fee configuration introduced in recent versions adds parameters that are delicate to configure correctly, and misconfiguring them is the most common cause of failed relayer transactions.
At a Glance: Hermes Production Requirements
| Component | Requirement |
|---|---|
| Binary version | Hermes 1.10+ (IBC v2 support) |
| RPC/gRPC endpoints | Dedicated full nodes preferred over public endpoints |
| WebSocket | Required for push-mode event source |
| Relayer wallet | Funded per chain, monitored for low balance |
| Gas config | default_gas and max_gas manually set (not auto defaults) |
| Telemetry | Prometheus with tx_confirmation = true |
| Critical monitoring | Packet backlog, light client expiry, wallet balance |
| Trusting period | ~2/3 of unbonding period (typically 14 days) |
Why Hermes IBC Relayer Setup Needs More Than the Auto Config
The hermes config auto command generates a working configuration by pulling chain data from the Cosmos chain registry. It is the right starting point and the wrong ending point. The Hermes documentation itself warns that the auto-generated default_gas and max_gas parameters are set to default values that should be manually reset, and the gas_price is set to the chain registry’s average, which may not reflect current network conditions.
For a hermes ibc relayer setup that runs in production without silent failures, three categories of configuration need manual attention:
Gas and fees. The auto config’s gas defaults will cause transaction failures during network congestion or succeed at a cost that makes your relayer unprofitable. Gas configuration is the single most common source of production relayer problems.
Endpoints. The auto config uses public RPC and gRPC endpoints from the chain registry. Public endpoints have rate limits that cause missed events during high-activity periods. Production relayers should use dedicated full nodes.
Monitoring. The auto config does not enable telemetry. Without tx_confirmation = true and a Prometheus scrape, you have no visibility into whether packets are being relayed successfully or silently backing up.
Step 1: Install Hermes and Generate the Base Config
# Install Hermes from the official release (Rust binary)
curl -L https://github.com/informalsystems/hermes/releases/latest/download/hermes-v1.10.0-x86_64-unknown-linux-gnu.tar.gz | tar xz
sudo mv hermes /usr/local/bin/
# Verify version (must be 1.10+ for IBC v2 support)
hermes version
# Create the Hermes directory
mkdir -p $HOME/.hermes
# Generate the base config from the chain registry
hermes config auto \
--output $HOME/.hermes/config.toml \
--chain cosmoshub:relayer-hub \
--chain osmosis:relayer-osmosisThis produces a working starting config with packet filters scoped to the canonical channels (for example, channel-141 for cosmoshub-4 to osmosis-1). The next steps correct the parts the auto config leaves unsafe.
Step 2: Hermes IBC Relayer Setup: Harden the config.toml
The generated config.toml needs manual correction in the global section and per-chain. Here is the production-hardened structure:
[global]
log_level = 'info'
[mode.clients]
enabled = true
refresh = true # Critical: auto-refresh light clients before expiry
misbehaviour = true
[mode.connections]
enabled = false # No new connections needed for existing channels
[mode.channels]
enabled = false # No new channels needed
[mode.packets]
enabled = true
clear_interval = 100 # Clear stuck packets every 100 blocks
clear_on_start = true
tx_confirmation = true # REQUIRED for telemetry metrics to populate
[telemetry]
enabled = true
host = '127.0.0.1'
port = 3001
[[chains]]
id = 'cosmoshub-4'
type = 'CosmosSdk'
rpc_addr = 'https://your-dedicated-hub-rpc:26657' # Dedicated node, not public
grpc_addr = 'https://your-dedicated-hub-grpc:9090'
event_source = { mode = 'push', url = 'wss://your-dedicated-hub-rpc:26657/websocket', batch_delay = '500ms' }
rpc_timeout = '10s'
account_prefix = 'cosmos'
key_name = 'relayer-hub'
store_prefix = 'ibc'
gas_price = { price = 0.025, denom = 'uatom' }
default_gas = 100000 # Manually set - not auto default
max_gas = 4000000 # Manually set based on observed usage
gas_multiplier = 1.2 # Buffer above estimated gas
max_msg_num = 30
max_tx_size = 2097152
clock_drift = '5s'
max_block_time = '30s'
trusting_period = '14days' # ~2/3 of the 21-day unbonding period
client_refresh_rate = '1/3'
trust_threshold = { numerator = '1', denominator = '3' }
[chains.packet_filter]
policy = 'allow'
list = [
['transfer', 'channel-141'], # Hub <> Osmosis canonical
]The critical corrections versus the auto config: tx_confirmation = true enables telemetry, event_source points to your dedicated WebSocket endpoint, default_gas and max_gas are set to real values, gas_multiplier provides a buffer above estimated gas, and client_refresh_rate ensures light clients are refreshed before they can expire.
Validate the config before starting:
hermes config validate
# SUCCESS: "validation passed successfully"Step 3: Key Management and Wallet Funding
# Add the relayer key for each chain
hermes keys add \
--chain cosmoshub-4 \
--key-file $HOME/.hermes/keys/hub-relayer.json \
--key-name relayer-hub
hermes keys add \
--chain osmosis-1 \
--key-file $HOME/.hermes/keys/osmosis-relayer.json \
--key-name relayer-osmosis
# Verify keys are loaded
hermes keys list --chain cosmoshub-4
# Run a health check across all configured chains
hermes health-checkWallet funding is an operational requirement, not a one-time setup. The relayer wallet on each chain pays gas for every packet relayed. An empty wallet means relaying stops silently. Fund each relayer address with enough native token for sustained operation, and monitor the balance continuously, covered in Step 5.
The relayer economics matter: on chains where you relay high volume, gas costs accumulate. Hermes does not earn fees by default unless the chains have ICS-29 fee middleware enabled (which was removed in ibc-go v10). For most operators, relaying is either a public good, a service to chains you have a stake in, or a cost you accept to support your own validator operation. Understand the economics before committing to relay a high-traffic path.
Step 4: Gas and Fee Management: The Production Gotcha
The introduction of dynamic gas fees adds configuration that the Hermes documentation explicitly describes as delicate to handle correctly. This is where most hermes ibc relayer setup problems originate.
How Hermes calculates fees: Hermes simulates how much gas a transaction will expend on the target chain, then multiplies the estimated gas by gas_multiplier and gas_price to compute the total fee. If gas_price is too low, transactions fail during congestion. If max_gas is too low, large transactions (high IBC message volume) fail. If gas_multiplier is too tight, transactions fail when actual gas exceeds the estimate.
The configuration that prevents failures:
# Conservative production gas settings
gas_price = { price = 0.025, denom = 'uatom' } # Check current network rates
default_gas = 100000 # Starting estimate when simulation unavailable
max_gas = 4000000 # Ceiling - set above your observed maximum
gas_multiplier = 1.2 # 20% buffer above estimated gasDynamic gas configuration (for chains with EIP-1559-style dynamic fees):
# Enable dynamic gas price querying
dynamic_gas_price = { enabled = true, multiplier = 1.1, max = 0.6 }The dynamic_gas_queried_fees metric (exposed in telemetry) shows the gas price used after the query but before filtering by the configured max, the key signal for tuning these parameters against real network conditions.
The practical gotcha: It is genuinely difficult to estimate gas spend in advance because it depends on transaction volume, transaction size, and network congestion. The Hermes documentation recommends observing other operators’ configurations and checking IBC transfers on a block explorer like Mintscan to see real gas spend. Start conservative (higher max_gas, healthy gas_multiplier), monitor actual usage, and tighten over time.
Step 5: Production Monitoring with Prometheus
The telemetry section enabled in Step 2 exposes Hermes metrics on port 3001. These metrics are the difference between knowing a packet is stuck and finding out when a user complains.
The metrics that matter for a hermes ibc relayer setup:
| Metric | What it reveals |
|---|---|
backlog_oldest_sequence | Oldest unrelayed packet – rising = relaying failing |
backlog_size | Number of pending packets per channel |
cleared_send_packet_count_total | Packets Hermes had to clear (only with tx_confirmation = true) |
tx_latency_confirmed | Time from event to confirmed transaction |
wallet_balance | Relayer wallet balance per chain |
client_updates_submitted_total | Light client updates – confirms refresh is working |
Critical Prometheus alerts:
groups:
- name: hermes-relayer
rules:
# Packet backlog growing - relaying is failing
- alert: HermesPacketBacklogGrowing
expr: |
increase(backlog_size[15m]) > 10
for: 15m
labels:
severity: warning
annotations:
summary: "Hermes packet backlog growing on {{ $labels.chain }}"
description: "Backlog increased by {{ $value }} packets in 15 minutes. Check gas config and wallet balance."
# Relayer wallet running low
- alert: HermesWalletLow
expr: |
wallet_balance < 5000000 # Adjust per chain and denom
labels:
severity: critical
annotations:
summary: "Hermes relayer wallet low on {{ $labels.chain }}"
description: "Wallet balance {{ $value }}. Relaying will stop when funds run out. Top up immediately."
# No client updates - light client expiry risk
- alert: HermesClientNotRefreshing
expr: |
increase(client_updates_submitted_total[6h]) == 0
for: 6h
labels:
severity: critical
annotations:
summary: "Hermes not refreshing light client on {{ $labels.chain }}"
description: "No client updates in 6 hours. If the light client expires, the channel freezes."For correlating relayer telemetry with broader infrastructure observability, the OpenTelemetry patterns in our OpenTelemetry tutorial bridge Hermes metrics into a unified monitoring view alongside your validator and node metrics.
Step 6: Run Hermes as a systemd Service
# /etc/systemd/system/hermes.service
[Unit]
Description=Hermes IBC Relayer
After=network-online.target
[Service]
User=hermes
ExecStart=/usr/local/bin/hermes start
Restart=on-failure
RestartSec=10s
LimitNOFILE=65535
Environment="RUST_LOG=info"
[Install]
WantedBy=multi-user.targetsystemctl enable hermes
systemctl start hermes
journalctl -u hermes -f
# Confirm relaying is active
hermes query packet pending --chain cosmoshub-4 --port transfer --channel channel-141The Operational Gotchas the Docs Underemphasize
These are the issues that surface in production hermes ibc relayer setup that the happy-path documentation does not foreground:
Light client expiry freezes channels. If Hermes stops refreshing a light client – because it crashed, lost its RPC connection, or ran out of gas, and the trusting period elapses, the light client expires and the channel freezes. Recovery requires a governance proposal on some chains. The client_refresh_rate config and the client update alert are what prevent this.
Public RPC endpoints cause missed events. A relayer on public RPC works in testing and silently misses events during high-activity periods when rate limits hit. The missed events become stuck packets. Use dedicated nodes.
Gas defaults from auto config fail under congestion. The single most common production failure. The auto-generated gas values are placeholders. They must be replaced with values tuned to real network conditions.
CosmWasm chains need pull mode. Chains that emit IBC events without the message attribute (some CosmWasm-enabled chains) require event_source = { mode = 'pull' } because Hermes misses their events over WebSocket. If you relay for a CosmWasm chain and see missed packets, this is likely the cause.
Wallet drain stops relaying silently. No error, no crash, relaying just stops when the wallet empties. The wallet balance alert is not optional for production.
The Hermes Production Setup Checklist
INSTALLATION
[ ] Hermes 1.10+ installed (IBC v2 support)
[ ] Config generated with hermes config auto
[ ] hermes config validate passes
CONFIG HARDENING
[ ] default_gas and max_gas manually set (not auto defaults)
[ ] gas_multiplier set to 1.2+ for buffer
[ ] gas_price checked against current network rates
[ ] event_source pointing to dedicated WebSocket endpoint
[ ] tx_confirmation = true enabled
[ ] client_refresh_rate configured (1/3 default)
[ ] trusting_period set to ~2/3 of unbonding period
[ ] packet_filter scoped to canonical channels only
KEYS AND FUNDING
[ ] Relayer key added per chain
[ ] hermes health-check passes
[ ] Relayer wallet funded per chain
[ ] Wallet balance monitoring configured
MONITORING
[ ] Telemetry enabled on port 3001
[ ] Prometheus scraping Hermes metrics
[ ] Alert on packet backlog growing
[ ] Alert on wallet balance low
[ ] Alert on client not refreshing (expiry risk)
OPERATIONS
[ ] Running as systemd service with auto-restart
[ ] Pull mode configured for any CosmWasm chains
[ ] Dedicated RPC/gRPC nodes (not public endpoints)
[ ] Redundant relayer on independent infrastructure (optional but recommended)FAQ: Hermes IBC Relayer Setup
What is the difference between push and pull mode in Hermes?
In a hermes ibc relayer setup, push mode (the default) receives IBC events over a WebSocket. Push mode (the default) receives IBC events over a WebSocket connection in real time. Pull mode polls the /block_results RPC endpoint at a configured interval. Use push mode for standard chains. Use pull mode only when Hermes misses events it should receive, specifically for CosmWasm-enabled chains that emit IBC events without the message attribute. Push mode has lower latency; pull mode is more reliable for the specific case of event-attribute-less chains.
Why are my Hermes transactions failing?
The most common cause is gas configuration. The auto-generated config sets default_gas and max_gas to placeholder values and gas_price to a registry average that may not reflect current conditions. Failed transactions during congestion almost always mean gas_price or gas_multiplier is too low. Check the dynamic_gas_queried_fees metric against your configured max, and increase gas_multiplier to provide more buffer.
How much does running a Hermes relayer cost?
The cost is gas paid by the relayer wallet on each chain for every packet relayed, plus the infrastructure to run dedicated RPC nodes (recommended). Since ICS-29 fee middleware was removed in ibc-go v10, relayers do not earn fees by default. Gas spend depends on volume, transaction size, and congestion, observe other operators on a block explorer like Mintscan to estimate, and budget for sustained wallet funding.
How do I prevent a light client from expiring?
Enable refresh = true in [mode.clients] and set client_refresh_rate (default 1/3 of the trusting period, meaning the client is refreshed three times per trusting period). Then alert on client_updates_submitted_total showing zero updates over a multi-hour window. If Hermes stops refreshing and the trusting period elapses, the channel freezes and recovery can require governance intervention.
Does Hermes support IBC v2 and Eureka?
Yes, Hermes 1.10+ supports relaying for both IBC Classic and IBC v2 simultaneously. For the full context on the ibc-go v10 migration and IBC v2, see our Cosmos IBC tutorial. A relayer running an older version will relay Classic packets correctly but will not relay v2 packets, upgrade to 1.10+ if you relay for any chain using IBC v2.
Conclusion
Hermes IBC relayer setup is straightforward to start and operationally demanding to run well. The install and auto config get you relaying in 20 minutes. The production hardening, corrected gas configuration, dedicated endpoints, telemetry, wallet monitoring, and light client refresh, is what keeps it relaying reliably without silent failures.
The gotchas that matter most: the auto config’s gas defaults will fail under congestion, public RPC endpoints cause missed events that become stuck packets, and an unmonitored light client can expire and freeze a channel. Each one is preventable with the configuration and monitoring in this guide.
At The Good Shell we operate Cosmos IBC relayer and validator infrastructure for teams that need reliable cross-chain operations. See our Web3 infrastructure services and case studies, and the related Cosmos validator setup guide for the broader validator context.
For the authoritative reference, the Hermes documentation at hermes.informal.systems and the source repository at github.com/informalsystems/hermes cover every configuration parameter with up-to-date detail.
Related Articles
- → Cosmos IBC Tutorial: 8 Proven Steps to Master IBC Eureka Migration
- → Cosmos Validator Setup: The Ultimate Step-by-Step Guide
- → Cosmos Validator Slashing: How to Prevent It and Recover Fast
- → Cosmos Consumer Chains: 8 Proven Steps to Evaluate PSS Chains
- → OpenTelemetry Tutorial for Validators: 7 Proven Production Patterns
