Grafana Setup

Overview¶

The focus of this experiment was to let Claude do what I am not good at, building dashboard queries. A local GrafanaPodman container created by solti-containers project acts as a quick test/development environment. Grafana can be managed programmatically via HTTP API. This enables debugging and creating dashboard panels without manual UI interaction.

I took broken sample dashboards and made them work in my environment. By adding variables for database name and hostname these can be reused quickly. Just tell the AI to upload them.

This work was quick and very vibey in nature with a high rate of success. Static dashboards may become a thing of the past. A phrase I like is "disposable pixels". Real-time dashboard/report creation based upon the needs of the moment.

Note: This documentation focuses on concepts and patterns, not specific IP addresses. Examples use localhost:3000 for consistency.

Environment Setup¶

Local Grafana Container¶

Container Details:

Container name: grafana-infra or grafana-svc
Internal endpoint: http://127.0.0.1:3000
Optional proxy: HTTPS access via Traefik reverse proxy
Admin password: ~/.secrets/grafana.admin.pass

API Authentication:

# Read admin password from file
admin:$(cat ~/.secrets/grafana.admin.pass)

Data Source Endpoints¶

InfluxDB:

Internal: http://localhost:8086
Remote: Use WireGuard tunnel or direct hostname

Loki:

Internal: http://localhost:3100
Remote: Use WireGuard tunnel or direct hostname

Useful Grafana API Endpoints¶

List All Dashboards¶

curl -s -u admin:$(cat ~/.secrets/grafana.admin.pass) \
  http://localhost:3000/api/search?type=dash-db | python3 -m json.tool

Output:

[
  {
    "id": 1,
    "uid": "fail2ban",
    "title": "Fail2ban Activity",
    "url": "/d/fail2ban/fail2ban-activity",
    "type": "dash-db"
  }
]

List Data Sources¶

curl -s -u admin:$(cat ~/.secrets/grafana.admin.pass) \
  http://localhost:3000/api/datasources | python3 -m json.tool

Output:

[
  {
    "id": 1,
    "uid": "influxdb-uid",
    "name": "InfluxDB",
    "type": "influxdb",
    "url": "http://localhost:8086",
    "isDefault": false
  },
  {
    "id": 2,
    "uid": "loki-uid",
    "name": "Loki",
    "type": "loki",
    "url": "http://localhost:3100",
    "isDefault": true
  }
]

Get Dashboard by UID¶

curl -s -u admin:$(cat ~/.secrets/grafana.admin.pass) \
  http://localhost:3000/api/dashboards/uid/fail2ban | jq '.dashboard.title'

Output:

"Fail2ban Activity"

Get Organization Info¶

curl -s -u admin:$(cat ~/.secrets/grafana.admin.pass) \
  http://localhost:3000/api/org

Output:

{
  "id": 1,
  "name": "Main Org.",
  "address": {
    "address1": "",
    "address2": "",
    "city": "",
    "zipCode": "",
    "state": "",
    "country": ""
  }
}

Create Dashboard¶

curl -s -X POST \
  -u admin:$(cat ~/.secrets/grafana.admin.pass) \
  -H "Content-Type: application/json" \
  -d @dashboard.json \
  http://localhost:3000/api/dashboards/db

Response:

{
  "id": 5,
  "uid": "new-dashboard-uid",
  "url": "/d/new-dashboard-uid/dashboard-name",
  "status": "success",
  "version": 1,
  "slug": "dashboard-name"
}

Update Dashboard¶

curl -s -X POST \
  -u admin:$(cat ~/.secrets/grafana.admin.pass) \
  -H "Content-Type: application/json" \
  -d @dashboard-updated.json \
  http://localhost:3000/api/dashboards/db

Payload format:

{
  "dashboard": { ... },
  "message": "Updated fail2ban pie chart",
  "overwrite": true
}

Delete Dashboard¶

curl -s -X DELETE \
  -u admin:$(cat ~/.secrets/grafana.admin.pass) \
  http://localhost:3000/api/dashboards/uid/DASHBOARD_UID

Connection Verification¶

Test that Grafana and data sources are accessible before creating or modifying dashboards.

Verify Grafana API¶

# Health check
curl -I http://localhost:3000/api/health

# Expected output
HTTP/1.1 200 OK

Verify InfluxDB Connection¶

# From Grafana host
curl -I http://localhost:8086/health

# Expected output
HTTP/1.1 200 OK

Verify Loki Connection¶

# From Grafana host
curl -s http://localhost:3100/ready

# Expected output
ready

Test Data Source Query¶

InfluxDB:

curl -s -X POST http://localhost:8086/api/v2/query \
  -H "Authorization: Token YOUR_TOKEN" \
  -H "Content-Type: application/vnd.flux" \
  -d 'from(bucket: "telegraf") |> range(start: -5m) |> limit(n: 1)'

Loki:

curl -s -G http://localhost:3100/loki/api/v1/query \
  --data-urlencode 'query={hostname="ispconfig3"}' \
  --data-urlencode 'limit=1'

Python Helper Functions¶

Grafana API Client¶

#!/usr/bin/env python3
from pathlib import Path
import subprocess
import json

class GrafanaAPI:
    def __init__(self, url="http://localhost:3000"):
        self.url = url
        self.password = Path.home() / '.secrets' / 'grafana.admin.pass'
        self.auth = f"admin:{self.password.read_text().strip()}"

    def get_dashboards(self):
        """List all dashboards"""
        cmd = ['curl', '-s', '-u', self.auth,
               f'{self.url}/api/search?type=dash-db']
        result = subprocess.run(cmd, capture_output=True, text=True)
        return json.loads(result.stdout)

    def get_dashboard(self, uid):
        """Get dashboard by UID"""
        cmd = ['curl', '-s', '-u', self.auth,
               f'{self.url}/api/dashboards/uid/{uid}']
        result = subprocess.run(cmd, capture_output=True, text=True)
        return json.loads(result.stdout)

    def create_dashboard(self, dashboard_json, message="Created via API"):
        """Create or update dashboard"""
        payload = {
            "dashboard": dashboard_json,
            "message": message,
            "overwrite": True
        }

        # Write payload to temp file
        payload_file = Path('/tmp/grafana-payload.json')
        payload_file.write_text(json.dumps(payload, indent=2))

        cmd = ['curl', '-s', '-X', 'POST',
               '-u', self.auth,
               '-H', 'Content-Type: application/json',
               '-d', f'@{payload_file}',
               f'{self.url}/api/dashboards/db']

        result = subprocess.run(cmd, capture_output=True, text=True)
        return json.loads(result.stdout)

# Usage
api = GrafanaAPI()
dashboards = api.get_dashboards()
for d in dashboards:
    print(f"{d['uid']}: {d['title']}")

Data Source Tester¶

#!/usr/bin/env python3
import subprocess
import json

def test_loki_query(query, limit=5):
    """Test Loki query and return results"""
    cmd = [
        'curl', '-s', '-G', 'http://localhost:3100/loki/api/v1/query',
        '--data-urlencode', f'query={query}',
        '--data-urlencode', f'limit={limit}'
    ]

    result = subprocess.run(cmd, capture_output=True, text=True)
    data = json.loads(result.stdout)

    if data['status'] == 'success' and data['data']['result']:
        print(f"✅ Query works! {len(data['data']['result'])} results")
        return data['data']['result']
    else:
        print(f"❌ Query failed: {data.get('error', 'no results')}")
        return None

# Usage
test_loki_query('{service_type="fail2ban"}')

Deployment Considerations¶

Container vs. Production¶

Development/Test (Podman container):

Grafana runs as local container
Data sources on localhost or via WireGuard
Admin password in ~/.secrets/
HTTP only (no TLS)

Production (Optional proxy):

Same Grafana container
Optional Traefik reverse proxy for HTTPS
DNS name points to proxy
TLS certificates managed by Traefik

Key Point: The API access pattern is the same - always use http://localhost:3000 for API calls, regardless of external access method.

Data Source Configuration¶

Local Development:

# All services on localhost
influxdb_url: "http://localhost:8086"
loki_url: "http://localhost:3100"

Distributed Deployment:

# Remote data sources via WireGuard
influxdb_url: "http://10.10.0.11:8086"  # monitor11 via VPN
loki_url: "http://10.10.0.11:3100"

Concepts Apply to Both:

Token-based authentication
HTTP API access
Dashboard JSON structure
Query syntax (Flux, LogQL)

Reference¶

For programmatic dashboard creation:

See CLAUDE.md "Creating Grafana Dashboards Programmatically"
See CLAUDE.md "Grafana Dashboard Development Workflow"

For troubleshooting blank graphs:

See CLAUDE.md "Troubleshooting Blank Grafana Graphs"