Troubleshooting - Nokia BNG Lab

This guide covers common operational issues and provides step-by-step troubleshooting procedures for the Nokia BNG lab.

Container Status Issues

Check Overall Lab Status

Verify Lab Deployment

sudo containerlab inspect -t lab.yml

This shows all containers, their status, and IP addresses.

Check Container Health

docker ps -a | grep clab-lab

Look for:

Status: Should be “Up” for all containers
Restart count: Should be 0 or low

Identify Failed Containers

# Show only stopped containers
docker ps -a --filter "status=exited" | grep clab-lab

# Show containers that are restarting
docker ps -a --filter "status=restarting" | grep clab-lab

Container Won't Start

Symptoms: Container immediately exits or restarts continuouslyDiagnosis:

# View container logs
docker logs clab-lab-<container-name>

# Check last 50 lines with timestamps
docker logs --tail 50 -t clab-lab-<container-name>

# Follow logs in real-time
docker logs -f clab-lab-<container-name>

Common Causes:

Missing or invalid license file (Nokia devices)
Configuration file errors
Port conflicts
Insufficient system resources

Solutions:

# Restart specific container
docker restart clab-lab-<container-name>

# Force recreate container
sudo containerlab destroy -t lab.yml --cleanup
sudo containerlab deploy -t lab.yml

Check Container Resource Usage

# View resource consumption
docker stats clab-lab-bng1

# Check all containers
docker stats $(docker ps --format '{{.Names}}' | grep clab-lab)

Nokia SR OS containers typically require:

CPU: 2+ cores
Memory: 4-8 GB per BNG instance
Storage: 10+ GB available

Connectivity Issues

Management Network Connectivity

Verify Management Network

docker network inspect lab

Confirm:

Subnet: 10.77.1.0/24
All containers are attached

Test Inter-Container Connectivity

# Ping from BNG1 to Grafana
docker exec clab-lab-bng1 ping -c 3 10.77.1.14

# Ping from Grafana to Prometheus
docker exec clab-lab-grafana ping -c 3 10.77.1.13

# Ping from gNMIc to BNG devices
docker exec clab-lab-gnmic ping -c 3 10.77.1.2
docker exec clab-lab-gnmic ping -c 3 10.77.1.3

Verify DNS Resolution

# Check if container names resolve
docker exec clab-lab-grafana ping -c 2 prometheus
docker exec clab-lab-prometheus ping -c 2 gnmic

Cannot Access Services from Host

Symptoms: Cannot reach Grafana (localhost:3030) or SSH (localhost:56661)Diagnosis:

# Check if ports are listening
netstat -tuln | grep -E '(3030|9090|56661|56664)'

# Or using ss
ss -tuln | grep -E '(3030|9090|56661|56664)'

# Verify Docker port mappings
docker port clab-lab-grafana
docker port clab-lab-bng1

Solutions:

Check firewall rules: sudo iptables -L -n | grep <port>
Verify no other service is using the port: sudo lsof -i :<port>
Restart container: docker restart clab-lab-<container>
Check Docker iptables: sudo iptables -t nat -L -n

Data Plane Connectivity Issues

Between BNG and ONT:

# Verify link status on BNG1
docker exec -it clab-lab-bng1 sr_cli "show port"

# Check OLT interfaces
docker exec -it clab-lab-olt sr_cli "show port"

# Verify ONT1 interface
docker exec clab-lab-ont1 ip link show

Check VLAN configuration:

# ONT1 should have VLAN 150
docker exec clab-lab-ont1 ip -d link show eth1

# Verify service configuration on BNG
docker exec -it clab-lab-bng1 sr_cli "show service id 150"

License Problems

Nokia devices will not function without valid licenses. This is the most common deployment issue.

Verify License Status

Check License File Exists

# Verify license file in configs directory
ls -lh configs/license/SR_SIM_license.txt

# Check file is not empty
wc -l configs/license/SR_SIM_license.txt

Verify License in Container

# BNG1 license status
docker exec -it clab-lab-bng1 sr_cli "show system license"

# Check license details
docker exec -it clab-lab-bng1 sr_cli "show system license detail"

Check License Expiry

docker exec -it clab-lab-bng1 sr_cli \
  "show system license | match \"Expiry Date\""

License Not Found Error

Symptoms: Device starts but shows “License not installed” or similar errorDiagnosis:

# Check if license was mounted correctly
docker exec clab-lab-bng1 ls -l /configs/license/

# Verify lab.yml has correct license path
grep -A 2 "license:" lab.yml

Solution:

# Ensure license file exists
mkdir -p configs/license
cp /path/to/your/SR_SIM_license.txt configs/license/

# Verify permissions
chmod 644 configs/license/SR_SIM_license.txt

# Redeploy lab
sudo containerlab destroy -t lab.yml
sudo containerlab deploy -t lab.yml

License Expired

Symptoms: Device starts but features are disabledSolution:

Obtain a new license from Nokia

Replace the old license file:

cp /path/to/new_license.txt configs/license/SR_SIM_license.txt

Restart affected containers:

docker restart clab-lab-bng1 clab-lab-bng2 clab-lab-switch clab-lab-olt

Verify new license:

docker exec -it clab-lab-bng1 sr_cli "show system license"

Wrong License Type

Symptoms: License is valid but features are missingCheck Required License Features:

SR-7 platform support (for BNG1/BNG2)
IXR-ec platform support (for Switch/OLT)
ISA (Integrated Service Adapter) support for BNG
Required capacity limits

# View licensed features
docker exec -it clab-lab-bng1 sr_cli "show system license detail"

Authentication Failures

RADIUS Authentication Issues

Verify RADIUS Service

# Check RADIUS container is running
docker ps | grep radius

# Test RADIUS locally
docker exec clab-lab-radius radiusd -X

Test User Authentication

# Test PPPoE user credentials
docker exec clab-lab-radius radtest \
  test@test.com testlab123 \
  localhost 0 testing123

# Should return "Access-Accept"

Check RADIUS Logs

# View authentication attempts
docker exec clab-lab-radius tail -100 /var/log/radius/radius.log

# Follow logs in real-time
docker exec clab-lab-radius tail -f /var/log/radius/radius.log

RADIUS Access-Reject

Symptoms: Subscriber authentication fails, Access-Reject in logsCommon Causes:

Incorrect username/password
User not in authorize file
Wrong shared secret between BNG and RADIUS

Diagnosis:

# Verify user exists in database
docker exec clab-lab-radius cat /etc/raddb/mods-config/files/authorize | grep test

# Check BNG RADIUS configuration
docker exec -it clab-lab-bng1 sr_cli "show system security radius"

# Verify shared secret in clients.conf
docker exec clab-lab-radius cat /etc/raddb/clients.conf

Solution:

# Edit authorize file to add/fix user
docker exec -it clab-lab-radius vi /etc/raddb/mods-config/files/authorize

# Restart RADIUS to apply changes
docker restart clab-lab-radius

RADIUS Server Not Responding

Symptoms: RADIUS timeout, no response from serverDiagnosis:

# Check connectivity from BNG to RADIUS
docker exec clab-lab-bng1 ping -c 3 10.77.1.10

# Check RADIUS process is running
docker exec clab-lab-radius ps aux | grep radiusd

# Check RADIUS is listening on port 1812
docker exec clab-lab-radius netstat -uln | grep 1812

Solution:

# Restart RADIUS service
docker restart clab-lab-radius

# Check startup script executed
docker logs clab-lab-radius | grep "client.sh"

SSH Authentication Issues

Cannot SSH to Device

For Nokia Devices (admin/lab123):

# Verify SSH port mapping
docker port clab-lab-bng1 | grep 22

# Try with verbose output
ssh -v admin@localhost -p 56661

# Check if SSH is running in container
docker exec clab-lab-bng1 netstat -tln | grep :22

For Linux Containers (user/test for ONTs):

# Verify ONT1 SSH access
ssh -o StrictHostKeyChecking=no user@localhost -p 56673

# Or use direct shell access
docker exec -it clab-lab-ont1 /bin/bash

Telemetry Issues

gNMIc Not Collecting Metrics

Check gNMIc Status

# Verify container is running
docker ps | grep gnmic

# Check logs for errors
docker logs --tail 100 clab-lab-gnmic

Verify Device Discovery

# Check Docker socket is accessible
docker exec clab-lab-gnmic ls -l /var/run/docker.sock

# Test container discovery
docker exec clab-lab-gnmic docker ps | grep clab-lab

Test gRPC Connectivity

# Test connection to BNG1
docker exec clab-lab-gnmic gnmic -a 10.77.1.2:57400 \
  -u admin -p lab123 --insecure capabilities

# Test subscription
docker exec clab-lab-gnmic gnmic -a 10.77.1.2:57400 \
  -u admin -p lab123 --insecure \
  subscribe --path "/state/system/cpu"

gNMIc Connection Timeout

Symptoms: “context deadline exceeded” or timeout errorsDiagnosis:

# Check gRPC port is accessible from gNMIc container
docker exec clab-lab-gnmic nc -zv 10.77.1.2 57400

# Verify device is listening on gRPC port
docker exec clab-lab-bng1 netstat -tln | grep 57400

Solution:

Verify credentials are correct: admin/lab123
Check skip-verify: true or insecure: true in config
Ensure device has finished booting
Restart gNMIc: docker restart clab-lab-gnmic

Metrics Not Appearing in Prometheus

Diagnosis Chain:

# 1. Check gNMIc metrics endpoint
curl http://localhost:9273/metrics | grep -i system_cpu

# 2. Verify Prometheus can reach gNMIc
docker exec clab-lab-prometheus wget -qO- http://gnmic:9273/metrics | head

# 3. Check Prometheus targets
curl http://localhost:9090/api/v1/targets | jq

# 4. Query Prometheus for metrics
curl -G http://localhost:9090/api/v1/query \
  --data-urlencode 'query=system_cpu_total' | jq

Common Issues:

gNMIc not exporting metrics (check logs)
Prometheus not scraping (check targets page)
Metric name mismatch (check processors in gNMIc config)
Time range issue (metrics may be too old or in future)

Grafana Dashboard Issues

Dashboard Shows 'No Data'

Checklist:

# 1. Verify Grafana is running
docker ps | grep grafana

# 2. Check Prometheus data source
curl -u admin:admin http://localhost:3030/api/datasources | jq

# 3. Test data source from Grafana
curl -X POST -u admin:admin \
  http://localhost:3030/api/datasources/proxy/1/api/v1/query \
  -H "Content-Type: application/json" \
  -d '{"query":"up"}'

Solutions:

Verify time range is appropriate (last 6h, 24h)
Check query in Prometheus UI first
Verify dashboard variables are set correctly
Refresh dashboard or clear browser cache

Common Containerlab Issues

Lab Won’t Deploy

Error: Address Already in Use

Symptoms: Port conflict during deployment

# Find process using the port
sudo lsof -i :56661
sudo lsof -i :3030

# Kill the process or change port in lab.yml
sudo kill <PID>

Error: Image Not Found

Symptoms: Cannot pull or find specified image

# List available images
docker images | grep nokia
docker images | grep ont-ds

# Pull missing images
docker pull ghcr.io/abelperezr/ont-ds:0.2
docker pull ghcr.io/openconfig/gnmic:latest
docker pull prom/prometheus
docker pull grafana/grafana:10.3.5

Nokia SR-SIM images must be built locally. They are not publicly available.

Error: Network Already Exists

Symptoms: Stale network from previous deployment

# Clean up old lab
sudo containerlab destroy -t lab.yml --cleanup

# Remove stale network
docker network rm lab

# Redeploy
sudo containerlab deploy -t lab.yml

Lab Won’t Destroy

Containers Won't Stop

# Force stop all lab containers
docker stop $(docker ps -q --filter "name=clab-lab")

# Force remove
docker rm -f $(docker ps -aq --filter "name=clab-lab")

# Clean up networks
docker network prune -f

Log Locations

Containerlab Logs

# Containerlab deployment logs
sudo journalctl -u docker | grep containerlab

# Or check syslog
sudo tail -f /var/log/syslog | grep containerlab

Device Logs

Nokia SR OS Logs

# BNG1 system logs
docker exec -it clab-lab-bng1 sr_cli "show log log-id 99"

# Event logs
docker exec -it clab-lab-bng1 sr_cli "show log event-triggered log-id 99"

# Debug logs (if enabled)
docker exec -it clab-lab-bng1 sr_cli "show debug"

Application Logs

# gNMIc
docker logs clab-lab-gnmic 2>&1 | tee gnmic.log

# Prometheus
docker logs clab-lab-prometheus 2>&1 | tee prometheus.log

# Grafana
docker logs clab-lab-grafana 2>&1 | tee grafana.log

# RADIUS
docker exec clab-lab-radius cat /var/log/radius/radius.log

Debug Commands

Network Debugging

# Capture packets on container interface
docker exec clab-lab-bng1 tcpdump -i eth1 -w /tmp/capture.pcap

# Copy capture file to host
docker cp clab-lab-bng1:/tmp/capture.pcap ./

# View interface details
docker exec clab-lab-ont1 ip addr show
docker exec clab-lab-ont1 ip route show

# Check ARP tables
docker exec clab-lab-bng1 sr_cli "show router arp"

# Trace route
docker exec clab-lab-ont1 traceroute 172.19.1.1

Service Debugging

BNG Subscriber Debugging

# View active subscribers
docker exec -it clab-lab-bng1 sr_cli \
  "show service active-subscribers"

# Check DHCP bindings
docker exec -it clab-lab-bng1 sr_cli \
  "show router dhcp bindings"

# PPPoE sessions
docker exec -it clab-lab-bng1 sr_cli \
  "show service id 150 pppoe session"

# Debug RADIUS
docker exec -it clab-lab-bng1 sr_cli \
  "debug router radius all"

Enable Debug Mode

# Enable gNMIc debug
docker exec -it clab-lab-gnmic sh -c \
  'gnmic --config /gnmic-config.yml --log --debug subscribe'

# Enable Prometheus debug
# Edit prometheus.yml and add: --log.level=debug to cmd

# RADIUS debug mode
docker exec -it clab-lab-radius radiusd -X

Getting Help

When reporting issues, include:

Output of sudo containerlab inspect -t lab.yml
Container logs: docker logs clab-lab-<device>
Error messages (full text)
Steps to reproduce
System information: uname -a, docker version
Lab topology file (lab.yml)

Before seeking help:

Check license is valid and not expired
Verify all containers are running: docker ps
Review recent changes to configuration files
Try redeploying: sudo containerlab destroy && sudo containerlab deploy
Check system resources: free -h, df -h

Recovery Procedures

Complete Lab Reset

# 1. Destroy existing lab
sudo containerlab destroy -t lab.yml --cleanup

# 2. Remove all containers and networks
docker stop $(docker ps -q --filter "name=clab-lab")
docker rm $(docker ps -aq --filter "name=clab-lab")
docker network rm lab

# 3. Clean up resources
docker system prune -f

# 4. Redeploy from scratch
sudo containerlab deploy -t lab.yml

# 5. Wait for devices to boot (2-3 minutes)
sleep 180

# 6. Verify deployment
sudo containerlab inspect -t lab.yml

Backup Configuration

# Backup BNG configuration
docker exec -it clab-lab-bng1 sr_cli \
  "admin save" > backup-bng1-$(date +%Y%m%d).cfg

# Backup telemetry data (if needed)
docker exec clab-lab-prometheus tar czf /tmp/prometheus-data.tar.gz /prometheus
docker cp clab-lab-prometheus:/tmp/prometheus-data.tar.gz ./

​Container Status Issues

​Check Overall Lab Status

​Connectivity Issues

​Management Network Connectivity

​License Problems

​Verify License Status

​Authentication Failures

​RADIUS Authentication Issues

​SSH Authentication Issues

​Telemetry Issues

​gNMIc Not Collecting Metrics

​Grafana Dashboard Issues

​Common Containerlab Issues

​Lab Won’t Deploy

​Lab Won’t Destroy

​Log Locations

​Containerlab Logs

​Device Logs

​Debug Commands

​Network Debugging

​Service Debugging

​Getting Help

​Recovery Procedures

​Complete Lab Reset

​Backup Configuration

Container Status Issues

Check Overall Lab Status

Connectivity Issues

Management Network Connectivity

License Problems

Verify License Status

Authentication Failures

RADIUS Authentication Issues

SSH Authentication Issues

Telemetry Issues

gNMIc Not Collecting Metrics

Grafana Dashboard Issues

Common Containerlab Issues

Lab Won’t Deploy

Lab Won’t Destroy

Log Locations

Containerlab Logs

Device Logs

Debug Commands

Network Debugging

Service Debugging

Getting Help

Recovery Procedures

Complete Lab Reset

Backup Configuration