Module 4: Network Management and Maintenance

4.4 Network Management

After deploying a Hyperledger Fabric network, ongoing management and maintenance are essential to ensure its continued operation, security, and performance. This section covers key aspects of network management.

Channel Management

Channels in Hyperledger Fabric require ongoing management to maintain their functionality and adapt to changing requirements.

Channel Configuration Updates

Channel configurations can be updated to modify policies, add or remove organizations, or change other parameters:

# Export environment variables
export ORDERER_CA=/opt/gopath/src/github.com/hyperledger/fabric/peer/crypto/ordererOrganizations/example.com/orderers/orderer.example.com/msp/tlscacerts/tlsca.example.com-cert.pem
export CHANNEL_NAME=channel1

# Fetch the latest config block
peer channel fetch config config_block.pb -o orderer.example.com:7050 -c $CHANNEL_NAME --tls --cafile $ORDERER_CA

# Convert the config block to JSON
configtxlator proto_decode --input config_block.pb --type common.Block | jq .data.data[0].payload.data.config > config.json

# Create a modified JSON (e.g., to update batch timeout)
jq '.channel_group.groups.Orderer.values.BatchTimeout.value.timeout = "5s"' config.json > modified_config.json

# Convert the original and modified configs back to protobuf
configtxlator proto_encode --input config.json --type common.Config --output config.pb
configtxlator proto_encode --input modified_config.json --type common.Config --output modified_config.pb

# Calculate the config update
configtxlator compute_update --channel_id $CHANNEL_NAME --original config.pb --updated modified_config.pb --output config_update.pb

# Convert the config update to JSON
configtxlator proto_decode --input config_update.pb --type common.ConfigUpdate | jq . > config_update.json

# Wrap the config update in an envelope
echo '{"payload":{"header":{"channel_header":{"channel_id":"'$CHANNEL_NAME'", "type":2}},"data":{"config_update":'$(cat config_update.json)'}}}' | jq . > config_update_in_envelope.json

# Convert the envelope to protobuf
configtxlator proto_encode --input config_update_in_envelope.json --type common.Envelope --output config_update_in_envelope.pb

# Sign and submit the config update
peer channel update -f config_update_in_envelope.pb -c $CHANNEL_NAME -o orderer.example.com:7050 --tls --cafile $ORDERER_CA

Adding a New Organization

To add a new organization to an existing channel:

  1. Generate crypto material for the new organization
  2. Create a JSON file defining the new organization's MSP
  3. Fetch the current channel configuration
  4. Add the new organization to the channel configuration
  5. Update the channel with the new configuration
# Generate crypto material for the new organization (Org3)
cryptogen generate --config=./crypto-config-org3.yaml --output=./crypto-config

# Create configtx.yaml with Org3 definition
# Generate the Org3 configuration JSON
configtxgen -printOrg Org3MSP > org3.json

# Fetch the channel configuration
peer channel fetch config config_block.pb -o orderer.example.com:7050 -c $CHANNEL_NAME --tls --cafile $ORDERER_CA

# Convert the config block to JSON
configtxlator proto_decode --input config_block.pb --type common.Block | jq .data.data[0].payload.data.config > config.json

# Add Org3 to the channel configuration
jq -s '.[0] * {"channel_group":{"groups":{"Application":{"groups":{"Org3MSP":.[1]}}}}}' config.json org3.json > modified_config.json

# Convert configs to protobuf and calculate update
configtxlator proto_encode --input config.json --type common.Config --output config.pb
configtxlator proto_encode --input modified_config.json --type common.Config --output modified_config.pb
configtxlator compute_update --channel_id $CHANNEL_NAME --original config.pb --updated modified_config.pb --output config_update.pb

# Wrap and sign the config update
# ... (similar to the previous example)

# Submit the config update
peer channel update -f config_update_in_envelope.pb -c $CHANNEL_NAME -o orderer.example.com:7050 --tls --cafile $ORDERER_CA

Channel Policies Management

Update channel policies to reflect changing governance requirements:

# Fetch and decode the current config
peer channel fetch config config_block.pb -o orderer.example.com:7050 -c $CHANNEL_NAME --tls --cafile $ORDERER_CA
configtxlator proto_decode --input config_block.pb --type common.Block | jq .data.data[0].payload.data.config > config.json

# Modify the policy (e.g., change Endorsement policy from MAJORITY to ANY)
jq '.channel_group.groups.Application.policies.Endorsement.policy.value.rule = "ANY Writers"' config.json > modified_config.json

# Convert, calculate update, and submit
# ... (similar to previous examples)

Chaincode Management

Managing chaincode throughout its lifecycle is a critical aspect of network management.

Chaincode Upgrades

To upgrade chaincode to a new version:

# Package the new version
peer lifecycle chaincode package mycc_2.0.tar.gz --path /opt/gopath/src/github.com/chaincode/mycc/ --lang golang --label mycc_2.0

# Install on peers
peer lifecycle chaincode install mycc_2.0.tar.gz

# Get the package ID
peer lifecycle chaincode queryinstalled
export CC_PACKAGE_ID=mycc_2.0:hash

# Approve the new version (by each organization)
peer lifecycle chaincode approveformyorg -o orderer.example.com:7050 --tls --cafile $ORDERER_CA --channelID $CHANNEL_NAME --name mycc --version 2.0 --package-id $CC_PACKAGE_ID --sequence 2

# Commit the new version
peer lifecycle chaincode commit -o orderer.example.com:7050 --tls --cafile $ORDERER_CA --channelID $CHANNEL_NAME --name mycc --version 2.0 --sequence 2 --peerAddresses peer0.org1.example.com:7051 --tlsRootCertFiles /opt/gopath/src/github.com/hyperledger/fabric/peer/crypto/peerOrganizations/org1.example.com/peers/peer0.org1.example.com/tls/ca.crt --peerAddresses peer0.org2.example.com:9051 --tlsRootCertFiles /opt/gopath/src/github.com/hyperledger/fabric/peer/crypto/peerOrganizations/org2.example.com/peers/peer0.org2.example.com/tls/ca.crt

Chaincode Governance

Implement governance processes for chaincode management:

  1. Development Process:
  2. Code review requirements
  3. Testing standards

  4. Documentation requirements

  5. Approval Process:

  6. Multi-organization approval workflow
  7. Testing in development/staging environments

  8. Security review

  9. Deployment Process:

  10. Scheduled maintenance windows
  11. Rollback procedures

  12. Communication plan

  13. Monitoring Process:

  14. Performance metrics
  15. Error tracking
  16. Usage statistics

Certificate Management

Managing certificates is crucial for maintaining network security.

Certificate Renewal

X.509 certificates in Fabric have expiration dates and need to be renewed:

# Using Fabric CA client to renew certificates
export FABRIC_CA_CLIENT_HOME=/opt/fabric/organizations/peerOrganizations/org1.example.com
fabric-ca-client enroll -u https://admin:adminpw@ca.org1.example.com:7054 --csr.hosts peer0.org1.example.com --tls.certfiles /opt/fabric/organizations/fabric-ca/org1/tls-cert.pem

Certificate Revocation

Revoke certificates that have been compromised:

# Revoke a certificate
fabric-ca-client revoke -e admin -r keycompromise --gencrl

# Generate a CRL (Certificate Revocation List)
fabric-ca-client gencrl -M /opt/fabric/organizations/peerOrganizations/org1.example.com/msp

Certificate Rotation Strategy

Implement a certificate rotation strategy:

  1. Planning:
  2. Inventory all certificates and their expiration dates
  3. Create a rotation schedule

  4. Define roles and responsibilities

  5. Preparation:

  6. Generate new certificates
  7. Distribute new certificates to relevant nodes

  8. Update configuration files

  9. Execution:

  10. Perform rotation during maintenance windows
  11. Update one component at a time

  12. Verify connectivity after each update

  13. Verification:

  14. Confirm all components are using new certificates
  15. Verify network functionality
  16. Update certificate inventory

Monitoring and Logging

Effective monitoring and logging are essential for maintaining network health.

Metrics Collection

Hyperledger Fabric provides metrics that can be collected using Prometheus:

# prometheus.yml
global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'fabric'
    static_configs:
      - targets: ['peer0.org1.example.com:9443', 'peer0.org2.example.com:9444', 'orderer.example.com:9443']

Key metrics to monitor: - Transaction throughput - Endorsement time - Commit time - Resource usage (CPU, memory, disk) - Chaincode execution time

Log Management

Implement centralized logging with the ELK stack (Elasticsearch, Logstash, Kibana):

# docker-compose-elk.yaml
version: '3.7'

services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.10.0
    environment:
      - discovery.type=single-node
    ports:
      - 9200:9200
    volumes:
      - esdata:/usr/share/elasticsearch/data

  logstash:
    image: docker.elastic.co/logstash/logstash:7.10.0
    volumes:
      - ./logstash/pipeline:/usr/share/logstash/pipeline
    depends_on:
      - elasticsearch

  kibana:
    image: docker.elastic.co/kibana/kibana:7.10.0
    ports:
      - 5601:5601
    depends_on:
      - elasticsearch

volumes:
  esdata:

Configure Fabric components to send logs to Logstash:

# Update docker-compose.yaml for peers and orderers
services:
  peer0.org1.example.com:
    logging:
      driver: "json-file"
      options:
        max-size: "100m"
        max-file: "5"

Use Filebeat to ship logs to Logstash:

# filebeat.yml
filebeat.inputs:
- type: docker
  containers.ids:
    - '*'
  processors:
    - add_docker_metadata: ~

output.logstash:
  hosts: ["logstash:5044"]

Alerting

Set up alerts for critical issues:

# alertmanager.yml
global:
  resolve_timeout: 5m

route:
  group_by: ['alertname']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 1h
  receiver: 'email-notifications'

receivers:
- name: 'email-notifications'
  email_configs:
  - to: 'admin@example.com'
    from: 'alertmanager@example.com'
    smarthost: 'smtp.example.com:587'
    auth_username: 'alertmanager'
    auth_password: 'password'

Create alert rules:

# alert_rules.yml
groups:
- name: fabric
  rules:
  - alert: HighEndorsementLatency
    expr: fabric_endorser_proposal_duration_seconds > 1
    for: 5m
    labels:
      severity: warning
    annotations:
      summary: "High endorsement latency"
      description: "Endorsement latency is high for {{ $labels.instance }}"

  - alert: PeerDown
    expr: up{job="fabric"} == 0
    for: 5m
    labels:
      severity: critical
    annotations:
      summary: "Peer is down"
      description: "{{ $labels.instance }} has been down for more than 5 minutes"

Backup and Recovery

Implementing backup and recovery procedures is crucial for disaster recovery.

Backup Strategy

Components to back up: - Cryptographic material (MSP directories) - Ledger data - State databases - Configuration files - Chaincode source code

Example backup script:

#!/bin/bash
# backup.sh - Script to backup Hyperledger Fabric components

BACKUP_DIR="/opt/fabric/backups/$(date +%Y%m%d_%H%M%S)"
FABRIC_DIR="/opt/fabric"

# Create backup directory
mkdir -p $BACKUP_DIR

# Backup crypto material
echo "Backing up crypto material..."
cp -r $FABRIC_DIR/crypto-config $BACKUP_DIR/

# Backup configuration files
echo "Backing up configuration files..."
cp $FABRIC_DIR/configtx.yaml $BACKUP_DIR/
cp $FABRIC_DIR/crypto-config.yaml $BACKUP_DIR/
cp $FABRIC_DIR/docker-compose.yaml $BACKUP_DIR/

# Backup chaincode source
echo "Backing up chaincode source..."
cp -r $FABRIC_DIR/chaincode $BACKUP_DIR/

# Backup ledger data (requires stopping the network or using volume snapshots)
echo "Backing up ledger data..."
# Option 1: Stop the network and copy data
# docker-compose -f $FABRIC_DIR/docker-compose.yaml down
# cp -r /var/lib/docker/volumes/fabric_* $BACKUP_DIR/
# docker-compose -f $FABRIC_DIR/docker-compose.yaml up -d

# Option 2: Use volume snapshots (if supported by your storage system)
# Example for LVM snapshots
# lvcreate --size 1G --snapshot --name fabric_snapshot /dev/vg0/fabric_volume

echo "Backup completed: $BACKUP_DIR"

Recovery Procedures

Document recovery procedures for different scenarios:

  1. Node Failure Recovery: bash # Restore peer node cp -r /opt/fabric/backups/latest/crypto-config/peerOrganizations/org1.example.com/peers/peer0.org1.example.com /opt/fabric/crypto-config/peerOrganizations/org1.example.com/peers/ docker-compose -f docker-compose.yaml up -d peer0.org1.example.com

  2. Full Network Recovery: ```bash # Restore from backup cp -r /opt/fabric/backups/latest/* /opt/fabric/

# Start the network docker-compose -f docker-compose.yaml up -d ```

  1. Ledger Corruption Recovery: ```bash # Stop the affected peer docker stop peer0.org1.example.com

# Remove corrupted ledger rm -rf /var/lib/docker/volumes/fabric_peer0.org1.example.com

# Restore from backup cp -r /opt/fabric/backups/latest/fabric_peer0.org1.example.com /var/lib/docker/volumes/

# Restart the peer docker start peer0.org1.example.com ```

Performance Tuning

Optimizing network performance is an ongoing process.

Orderer Tuning

Tune orderer parameters for better performance:

# configtx.yaml
Orderer:
  BatchTimeout: 1s  # Reduce for faster transactions, increase for better batching
  BatchSize:
    MaxMessageCount: 500  # Increase for higher throughput
    AbsoluteMaxBytes: 10 MB  # Adjust based on transaction size
    PreferredMaxBytes: 2 MB  # Adjust based on transaction size

Peer Tuning

Optimize peer configuration:

# core.yaml
peer:
  gossip:
    dialTimeout: 3s
    aliveTimeInterval: 5s
    aliveExpirationTimeout: 25s
    reconnectInterval: 25s
    maxBlockCountToStore: 100
    maxPropagationBurstLatency: 10ms
    maxPropagationBurstSize: 10
    propagateIterations: 1
    propagatePeerNum: 3
    pullInterval: 4s
    pullPeerNum: 3
    requestStateInfoInterval: 4s
    publishStateInfoInterval: 4s
    stateInfoRetentionInterval: 0s
    publishCertPeriod: 10s
    skipBlockVerification: false
    timeForMembershipTracker: 5s
    timeForPullMsgStore: 10s
    timeForPurgeMsgStore: 100s
    timeForPurgePrivateDataStore: 100s

  couchdb:
    maxRetries: 5
    maxRetriesOnStartup: 10
    requestTimeout: 35s
    queryLimit: 10000
    maxBatchUpdateSize: 1000
    warmIndexesAfterNBlocks: 1

State Database Optimization

Optimize CouchDB for better performance:

# couchdb.yaml
couchdb:
  image: couchdb:3.1.1
  environment:
    - COUCHDB_USER=admin
    - COUCHDB_PASSWORD=adminpw
    - COUCHDB_MAX_DOCUMENT_SIZE=4194304  # 4MB
    - COUCHDB_MAX_ATTACHMENT_SIZE=1073741824  # 1GB
    - COUCHDB_QUERY_SERVER_CONFIG="{\"reduce_limit\": false}"
    - COUCHDB_LOG_LEVEL=error
  ports:
    - 5984:5984
  volumes:
    - couchdb_data:/opt/couchdb/data

Create indexes for frequently queried fields:

// index.json
{
  "index": {
    "fields": ["docType", "owner"]
  },
  "ddoc": "indexOwnerDoc",
  "name": "indexOwner",
  "type": "json"
}

Install the index:

curl -X POST -H "Content-Type: application/json" -d @index.json http://admin:adminpw@localhost:5984/mychannel_mycc/_index

Network Upgrades

Periodically, the Hyperledger Fabric network will need to be upgraded to newer versions.

Upgrade Planning

  1. Assessment:
  2. Review release notes for new features and breaking changes
  3. Test upgrades in a development environment

  4. Create an upgrade plan with rollback procedures

  5. Preparation:

  6. Back up all components
  7. Update configuration files for the new version

  8. Prepare new Docker images

  9. Execution:

  10. Upgrade components in a specific order (typically: orderers, then peers)
  11. Verify functionality after each component upgrade
  12. Monitor logs for errors

Orderer Upgrade

Upgrade orderer nodes:

# Stop the orderer
docker stop orderer.example.com

# Update the Docker image in docker-compose.yaml
# From: image: hyperledger/fabric-orderer:2.3.0
# To:   image: hyperledger/fabric-orderer:2.5.0

# Start the orderer with the new image
docker-compose -f docker-compose.yaml up -d orderer.example.com

# Check logs for errors
docker logs orderer.example.com

Peer Upgrade

Upgrade peer nodes:

# Stop the peer
docker stop peer0.org1.example.com

# Update the Docker image in docker-compose.yaml
# From: image: hyperledger/fabric-peer:2.3.0
# To:   image: hyperledger/fabric-peer:2.5.0

# Start the peer with the new image
docker-compose -f docker-compose.yaml up -d peer0.org1.example.com

# Check logs for errors
docker logs peer0.org1.example.com

Capability Updates

Update channel capabilities to enable new features:

# Fetch the current config
peer channel fetch config config_block.pb -o orderer.example.com:7050 -c $CHANNEL_NAME --tls --cafile $ORDERER_CA

# Convert to JSON
configtxlator proto_decode --input config_block.pb --type common.Block | jq .data.data[0].payload.data.config > config.json

# Update capabilities
jq '.channel_group.groups.Application.values.Capabilities.value.capabilities = {"V2_5": {}}' config.json > modified_config.json

# Convert, calculate update, and submit
# ... (similar to previous examples)

By implementing these network management practices, you can ensure the ongoing health, security, and performance of your Hyperledger Fabric network. Regular maintenance, monitoring, and updates are essential for a successful blockchain deployment.

Back to Course Home