Cosmos Validator Slashing: 7 Ways to Protect Your Node and Never Get Slashed Again

In this guide

Running this in production?

Get a senior review of your infrastructure — in 7 days

We run validator and cloud infrastructure across 24 chains with 10M+ daily checks at 99.97% uptime. Fixed-price 7-day audit: written report, prioritised findings, 90-min debrief call. $4,500 fixed, no long engagement.

Get the 7-day audit → Book a free 30-min infra review — leave with 2-3 concrete findings
# Validator node config.toml
pex = false
persistent_peers = "sentry1_node_id@sentry1_private_ip:26656,sentry2_node_id@sentry2_private_ip:26656"
private_peer_ids = ""
addr_book_strict = false
# Sentry node config.toml
pex = true
persistent_peers = "validator_node_id@validator_private_ip:26656"
private_peer_ids = "validator_node_id"
unconditional_peer_ids = "validator_node_id"
# Install dependencies
sudo apt install build-essential pkg-config libusb-1.0-0-dev

# Install TMKMS
cargo install tmkms --features=softsign
tmkms init /etc/tmkms
# /etc/tmkms/tmkms.toml
[[chain]]
id = "cosmoshub-4"
key_format = { type = "bech32", account_key_prefix = "cosmospub", consensus_key_prefix = "cosmosvalconspub" }
state_file = "/etc/tmkms/state/cosmoshub-4-consensus.json"

[[providers.softsign]]
chain_ids = ["cosmoshub-4"]
path = "/etc/tmkms/secrets/cosmoshub-4-consensus.key"

[[validator]]
addr = "tcp://validator_private_ip:26659"
chain_id = "cosmoshub-4"
reconnect = true
priv_validator_laddr = "tcp://0.0.0.0:26659"
git clone https://github.com/strangelove-ventures/horcrux
cd horcrux
make install
# On each Horcrux node
horcrux config init \
  --node "tcp://validator_ip:1234" \
  --cosigner "tcp://horcrux1_ip:2222|1" \
  --cosigner "tcp://horcrux2_ip:2222|2" \
  --cosigner "tcp://horcrux3_ip:2222|3" \
  --threshold 2 \
  --grpc-timeout 1000ms \
  --raft-timeout 1000ms
# /etc/systemd/system/gaiad.service
[Unit]
Description=Cosmos Hub Node
After=network-online.target

[Service]
User=cosmos
ExecStart=/usr/local/bin/gaiad start --home /home/cosmos/.gaia
Restart=always
RestartSec=3
LimitNOFILE=65535

[Install]
WantedBy=multi-user.target
# Add to crontab - alert if disk usage above 80%
*/5 * * * * df -h / | awk 'NR==2{if(int($5)>80) system("curl -s -X POST https://hooks.slack.com/YOUR_WEBHOOK -d \"{\\\"text\\\":\\\"ALERT: Disk usage at "$5" on validator\\\"}\"")}'
# prometheus-rules.yaml
groups:
  - name: validator.rules
    rules:
      - alert: ValidatorMissedBlocks
        expr: |
          increase(cosmos_validator_missed_blocks_total[10m]) > 10
        for: 2m
        labels:
          severity: warning
        annotations:
          summary: "Validator missing blocks"
          description: "Validator has missed {{ $value }} blocks in the last 10 minutes."

      - alert: ValidatorJailRisk
        expr: |
          cosmos_validator_missed_blocks_total > 400
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "CRITICAL: Validator at risk of jailing"
          description: "Validator has missed {{ $value }} blocks - approaching jail threshold of 500."
kubectl apply -f prometheus-rules.yaml

# Install Cosmovisor
go install cosmossdk.io/tools/cosmovisor/cmd/cosmovisor@latest

# Set environment variables
export DAEMON_NAME=gaiad
export DAEMON_HOME=$HOME/.gaia
export DAEMON_ALLOW_DOWNLOAD_BINARIES=true
export DAEMON_RESTART_AFTER_UPGRADE=true

# Run with Cosmovisor instead of gaiad directly
cosmovisor run start
```

Cosmovisor watches for upgrade governance proposals, downloads the new binary when the upgrade height approaches, and swaps the binary automatically at the correct block height. This eliminates the most common source of upgrade-related downtime.

---

## Protection 7 - Incident Runbooks for Every Failure Mode

The final layer of cosmos validator slashing protection is operational: documented runbooks for every failure scenario. When an alert fires at 3am, you don't want to be figuring out the unjail command from memory.

**Minimum runbook set:**

**Runbook 1 - Validator jailed for downtime:**
```
1. SSH to validator node
2. Check gaiad process: systemctl status gaiad
3. If stopped: systemctl start gaiad
4. Wait for node to sync: gaiad status | jq .SyncInfo.catching_up
5. Once synced, unjail: gaiad tx slashing unjail --from validator-key --chain-id cosmoshub-4 --gas auto --fees 1000uatom
6. Verify back in active set: gaiad query tendermint-validator-set | grep YOUR_VALIDATOR_ADDRESS
```

**Runbook 2 - Disk space critical:**
```
1. SSH to validator node
2. Check disk usage: df -h
3. Find large files: du -sh /home/cosmos/.gaia/* | sort -rh | head -20
4. Prune old blocks if needed: gaiad tendermint unsafe-reset-all (WARNING: only if necessary)
5. Clear old logs: journalctl --vacuum-size=2G
```

**Runbook 3 - Sentry node offline:**
```
1. Check sentry node status from monitoring dashboard
2. SSH to sentry node
3. Check connectivity: gaiad status
4. If node not syncing, restart: systemctl restart gaiad
5. Verify validator still connected through remaining sentries

Slashing Under Partial Set Security: Your Exposure Across Consumer Chains

Everything above protects a single validator on a single chain. Since the Gaia v17 upgrade, the Cosmos Hub runs Partial Set Security (PSS), and validators can opt into securing consumer chains: Top-N chains require the top N% of the set, Opt-In chains are voluntary. The moment you validate a consumer chain, your slashing surface changes, because faults on that chain can reach back to your Hub stake.

How cross-chain slashing actually works:

Double signing (equivocation) on a consumer chain is the dangerous one. The equivocation evidence is relayed to the provider (the Hub), and the validator is slashed and permanently tombstoned on the Hub. The slash fraction is set by that consumer chain’s own parameters, since ADR-020 made slashing and jailing customizable per chain, but the tombstone is what ends you: it removes your main Hub validator permanently. A double-sign on any chain you validate can kill your entire Hub operation, not just your position on that one chain.

Downtime on a consumer chain is far less severe. It results in temporary jailing on that chain, so you stop earning there until you recover, but it does not slash your Hub stake. Consumer-chain downtime is not propagated as a provider-level token slash.

Why this changes your risk calculus:

Under Replicated Security, every validator secured every consumer chain by default. Under PSS you choose, and every chain you opt into adds another independent path to a tombstone on the Hub. Ten consumer chains means ten more places where a misconfigured failover, a duplicated key or a botched upgrade can equivocate and tombstone your main validator. Exposure is cumulative, and the worst case is always the same: permanent removal from the Hub.

How to choose chains to minimize cumulative exposure:

Assign a separate consensus key per consumer chain, using gaiad tx provider assign-consensus-key. A unique key per chain means a signing fault on one chain cannot produce equivocation evidence against your Hub key.

Read each chain’s slashing and jailing parameters before opting in. ADR-020 lets chains set their own slash fractions, and parameter changes carry a notice period of one unbonding period, so you can opt out before harsher terms take effect. Watch for those proposals.

Only opt into chains you can run to the same double-sign-safe standard as your Hub node: its own sentry or remote-signer setup, its own failover discipline, its own Cosmovisor and monitoring. If you cannot operate a chain safely, its rewards are not worth the tombstone risk.

Operational reality of validating multiple chains:

Each consumer chain is a full node with its own binary, upgrade schedule, key, monitoring and on-call surface. The failure modes do not add, they multiply, and a single equivocation anywhere is catastrophic for the whole operation. This is why deciding which consumer chains to validate, weighing rewards against accumulated slashing exposure, is a discipline of its own.

See the 7-day audit →

Leave a Reply

Your email address will not be published. Required fields are marked *