Go Observability Stack – Production-Ready Monitoring

Comprehensive observability solution for Go applications and microservices

OpenTelemetry • Grafana Tempo • Loki • Prometheus • Grafana • DragonflyDB

📐 ARCHITECTURE OVERVIEW

┌─────────────────────────────────────────────────────────────────────┐
│                     Your Go Proxy Service (port 8089)               │
│                    /var/log/proxy-services/*.log                    │
└────────────┬────────────────────────────────────┬───────────────────┘
             │                                     │
             │ OTLP (traces/metrics)               │ Log files
             ▼                                     ▼
    ┌────────────────────┐              ┌─────────────────┐
    │ OTel Collector     │              │   Promtail      │
    │ :4317 (gRPC)       │              │   :9080         │
    │ :4318 (HTTP)       │              └────────┬────────┘
    │ :8889 (Prom)       │                       │
    └──┬──────┬──────┬───┘                       │
       │      │      │                            │
       │      │      └──────────┐                 │
       │      │                 │                 │
       ▼      ▼                 ▼                 ▼
    ┌─────┐ ┌──────────┐   ┌────────┐      ┌─────────┐
    │Tempo│ │Prometheus│   │Metrics │      │  Loki   │
    │:3200│ │  :9090   │   │Generator      │  :3100  │
    └──┬──┘ └────┬─────┘   └───┬────┘      └────┬────┘
       │         │             │                 │
       │         │             └────────┐        │
       │         │                      │        │
       └─────────┴──────────────────────┴────────┘
                          │
                          ▼
                   ┌──────────────┐
                   │   Grafana    │
                   │   :3000      │
                   └──────────────┘

    ┌─────────────────────────────────┐
    │  DragonflyDB (Redis Cache)      │
    │  :6382 (host) → :6379 (container)
    └─────────────────────────────────┘

Data Flow:

Go Proxy sends traces/metrics via OTLP → OTel Collector
Go Proxy writes logs to /var/log/proxy-services/ → Promtail reads them
OTel Collector forwards traces → Tempo
OTel Collector exposes metrics → Prometheus scrapes them
Promtail ships logs → Loki
Tempo generates span metrics → Prometheus
Grafana queries all three backends for unified observability

Separate Service:

DragonflyDB provides Redis-compatible caching for the Go application

=============================================================================

🚀 QUICK START GUIDE

📦 What's in the Package

observability/
├── 📄 README.md                  ✅ Updated with new name
├── 📝 CONTRIBUTING.md           ✅ Updated with new name
├── 📜 LICENSE                   ✅ MIT License
├── 📋 CODE_OF_CONDUCT.md        ✅ Community standards
├── 🔒 .env.example              ✅ Safe template
├── 🚫 .gitignore                ✅ Protects secrets
├── 📘 OPENSOURCE_GUIDE.md       ✅ Publishing steps
├── 🎯 GITHUB_SETUP.md           ✅ Repository setup guide (NEW!)
├── 🐳 docker-compose.yaml       ✅ Service definitions
└── ⚙️  Configuration files       ✅ All YAML configs

STEP 1: Create .env File

Copy .env.example to .env and set your secure passwords:

cp .env.example .env
# Edit .env and set secure passwords

Or create .env manually with:

DFLY_PASSWORD=your_secure_dragonfly_password
GRAFANA_ADMIN_PASSWORD=your_secure_grafana_password

STEP 2: Start the Stack

docker-compose up -d

STEP 3: Verify Services

docker-compose ps

All services should show "Up (healthy)" status: ✓ dragonfly ✓ otel-collector ✓ tempo ✓ loki ✓ promtail ✓ prometheus ✓ grafana

STEP 4: Configure Your Go Application

Update your main .env file with:

# DragonflyDB (Redis-compatible cache)
REDIS_HOST=localhost
REDIS_PORT=6382
REDIS_PASSWORD=<copy from observability/.env DFLY_PASSWORD>
REDIS_DB=0

# OpenTelemetry Configuration
OTEL_ENABLED=true
OTEL_ENDPOINT=localhost:4318
OTEL_SERVICE_NAME=proxy-services
OTEL_ENVIRONMENT=production

STEP 5: Ensure Log Directory Exists

Promtail reads logs from /var/log/proxy-services/

sudo mkdir -p /var/log/proxy-services
sudo chown $USER:staff /var/log/proxy-services

Configure your Go app to write logs to: /var/log/proxy-services/app.log

STEP 6: Start Your Application

cd proxy-services
go run main.go

STEP 7: Go Application Integration Examples

Connecting to DragonflyDB (Redis)

Add the following to your Go application to connect to DragonflyDB:

package main

import (
    "context"
    "log"
    "os"

    "github.com/redis/go-redis/v9"
)

func initRedis() *redis.Client {
    rdb := redis.NewClient(&redis.Options{
        Addr:     os.Getenv("REDIS_HOST") + ":" + os.Getenv("REDIS_PORT"),
        Password: os.Getenv("REDIS_PASSWORD"),
        DB:       0, // use default DB
    })

    // Test connection
    pong, err := rdb.Ping(context.Background()).Result()
    if err != nil {
        log.Fatal("Failed to connect to Redis:", err)
    }
    log.Println("Redis connected:", pong)

    return rdb
}

Initializing OpenTelemetry with OTLP Exporter

Add OpenTelemetry initialization to your main function:

package main

import (
    "context"
    "log"
    "os"

    "go.opentelemetry.io/otel"
    "go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp"
    "go.opentelemetry.io/otel/sdk/resource"
    semconv "go.opentelemetry.io/otel/semconv/v1.17.0"
    "go.opentelemetry.io/otel/sdk/trace"
)

func initOpenTelemetry(ctx context.Context) func() {
    // Create OTLP HTTP exporter
    exporter, err := otlptracehttp.New(ctx,
        otlptracehttp.WithEndpoint(os.Getenv("OTEL_ENDPOINT")),
        otlptracehttp.WithInsecure(), // Use WithTLSConfig for production
    )
    if err != nil {
        log.Fatal("Failed to create OTLP exporter:", err)
    }

    // Create resource
    res, err := resource.New(ctx,
        resource.WithAttributes(
            semconv.ServiceNameKey.String(os.Getenv("OTEL_SERVICE_NAME")),
            semconv.ServiceVersionKey.String("1.0.0"),
            semconv.DeploymentEnvironmentKey.String(os.Getenv("OTEL_ENVIRONMENT")),
        ),
    )
    if err != nil {
        log.Fatal("Failed to create resource:", err)
    }

    // Create tracer provider
    tracerProvider := trace.NewTracerProvider(
        trace.WithBatcher(exporter),
        trace.WithResource(res),
    )

    // Set global tracer provider
    otel.SetTracerProvider(tracerProvider)

    // Return cleanup function
    return func() {
        if err := tracerProvider.Shutdown(ctx); err != nil {
            log.Printf("Error shutting down tracer provider: %v", err)
        }
    }
}

Use it in your main function:

func main() {
    ctx := context.Background()

    // Initialize OpenTelemetry
    shutdown := initOpenTelemetry(ctx)
    defer shutdown()

    // Initialize Redis
    rdb := initRedis()
    defer rdb.Close()

    // Your application logic here
    tracer := otel.Tracer("proxy-services")
    ctx, span := tracer.Start(ctx, "main")
    defer span.End()

    // Example Redis usage
    err := rdb.Set(ctx, "key", "value", 0).Err()
    if err != nil {
        span.RecordError(err)
    }

    log.Println("Application started successfully")
}

=============================================================================

🌐 ACCESS URLS

Service URL Credentials

Grafana http://localhost:3000 admin / <GRAFANA_ADMIN_PASSWORD> Prometheus http://localhost:9090 (no auth) Tempo http://localhost:3200 (no auth) Loki http://localhost:3100 (no auth) OTel Collector http://localhost:8888 (metrics endpoint) OTel Prometheus http://localhost:8889 (metrics exporter) Promtail http://localhost:9080 (no auth)

DragonflyDB localhost:6382 Password: <DFLY_PASSWORD>

All ports are bound to 127.0.0.1 (localhost only) for security.

Remote Access via SSH Tunnel

If running on a remote server, forward ports to your local machine:

ssh -L 3000:localhost:3000 -L 9090:localhost:9090 user@your-server-ip

This allows you to access:

Grafana at http://localhost:3000 on your local machine
Prometheus at http://localhost:9090 on your local machine

For additional services, add more -L flags:

ssh -L 3000:localhost:3000 \
    -L 9090:localhost:9090 \
    -L 3200:localhost:3200 \
    -L 3100:localhost:3100 \
    user@your-server-ip

=============================================================================

📊 GRAFANA CONFIGURATION

AUTO-PROVISIONED DATA SOURCES:

✅ Prometheus (default) - Metrics from OTel & scrape targets ✅ Tempo - Distributed traces with service map ✅ Loki - Application logs with trace correlation ✅ OTel Collector - Direct collector metrics

INTEGRATED FEATURES:

• Traces → Logs: Click trace ID to see related logs • Traces → Metrics: See service metrics for trace spans • Logs → Traces: Click traceID field to view full trace • Service Map: Visualize service dependencies • Node Graph: Interactive trace visualization

EXAMPLE QUERIES:

Logs (Loki): {job="proxy-services"} |= "error" {job="proxy-services", level="error"} {job="proxy-services"} | json | level="warn" {job="proxy-services"} |= "traceID" | json

Traces (Tempo): • Search by service name: proxy-services • Filter by duration: > 100ms • Filter by status: error • Use trace ID from logs

Metrics (Prometheus): rate(http_requests_total[5m]) histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m])) proxy_services_cache_hits_total up{job="otel-collector"}

=============================================================================

🔧 COMPONENT DETAILS

DRAGONFLY DB

• Redis-compatible in-memory cache • Port: 6382 (host) → 6379 (container) • Max memory: 2GB with eviction • Password protected • Persistent storage: dfly_data volume

OTEL COLLECTOR

• Receives: OTLP gRPC (4317) and HTTP (4318) • Exports traces to: Tempo via OTLP • Exports metrics to: Prometheus exporter (8889) • Scrapes: Optional Prometheus /metrics from your app • Batch processing: 1024 records, 10s timeout

TEMPO

• Distributed tracing backend • Receives: OTLP (gRPC 4317, HTTP 4318) from collector • HTTP API: Port 3200 • Storage: Local filesystem (/var/tempo) • Trace retention: 48 hours • Metrics generator: Sends span metrics to Prometheus

LOKI

• Log aggregation system • Push API: Port 3100 • Schema: TSDB (modern, v13) • Storage: Local filesystem (/loki) • Log retention: 30 days (720h) • Max line size: 256KB • Ingestion rate: 10MB/s (burst 20MB/s)

PROMTAIL

• Log collector and shipper • Watches: /var/log/proxy-services/*.log • Pipeline: JSON parsing (zerolog format) • Extracts: level, timestamp, message, caller • Ships to: Loki (port 3100) • Position tracking: /tmp/positions

PROMETHEUS

• Time-series metrics database • HTTP API: Port 9090 • Scrape interval: 15s • Remote write: Enabled (for Tempo metrics) • Scrape targets:

otel-collector:8888 (collector metrics)
otel-collector:8889 (application metrics)
prometheus:9090 (self-monitoring)
tempo:3200 (tempo metrics)
loki:3100 (loki metrics)

GRAFANA

• Visualization and dashboards • Web UI: Port 3000 • Anonymous access: Disabled • Sign-up: Disabled • Data sources: Auto-provisioned with correlation • Persistent storage: grafana_data volume

=============================================================================

🐛 TROUBLESHOOTING

CHECK SERVICE HEALTH:

docker-compose ps
docker-compose logs -f <service-name>

SERVICE-SPECIFIC DEBUGGING:

OTel Collector not receiving traces: docker-compose logs -f otel-collector # Check your app connects to localhost:4318 or :4317 curl http://localhost:8888/metrics

Tempo not showing traces: docker-compose logs -f tempo curl http://localhost:3200/api/search # Verify OTel collector can reach tempo:4317

Loki not receiving logs: docker-compose logs -f loki docker-compose logs -f promtail # Verify /var/log/proxy-services/*.log exists and has content curl http://localhost:3100/ready

Promtail not reading logs: docker-compose logs -f promtail # Check log file permissions ls -la /var/log/proxy-services/ # Verify JSON format in logs

Prometheus not scraping: docker-compose logs -f prometheus curl http://localhost:9090/api/v1/targets # Check if targets are UP

DragonflyDB connection issues: docker-compose logs -f dragonfly # Test connection docker exec -it observability-dragonfly-1 redis-cli -p 6379 -a "$DFLY_PASSWORD" ping # Should return: PONG

Grafana data source issues: docker-compose logs -f grafana # Check data sources in Grafana UI: Configuration → Data Sources # Test each data source

NETWORK DEBUGGING:

# Check if services can communicate
docker-compose exec otel-collector wget -O- http://tempo:3200/ready
docker-compose exec otel-collector wget -O- http://loki:3100/ready
docker-compose exec promtail wget -O- http://loki:3100/ready

RESET EVERYTHING:

cd observability
docker-compose down -v  # ⚠️  Removes all data!
docker-compose up -d

RESTART SINGLE SERVICE:

docker-compose restart <service-name>

=============================================================================

📝 CONFIGURATION FILES

docker-compose.yaml - Service definitions and networking otel-config.yaml - OpenTelemetry Collector pipeline tempo.yaml - Tempo tracing configuration loki.yaml - Loki log aggregation config promtail-config.yaml - Promtail log collection rules prometheus.yaml - Prometheus scrape configs grafana-datasources.yaml - Auto-provisioned data sources .env - Secrets (not in git)

=============================================================================

🔒 SECURITY NOTES

✅ IMPLEMENTED SECURITY:

• All ports bound to 127.0.0.1 (localhost only) • DragonflyDB password protected • Grafana password protected • Grafana sign-up disabled • Grafana anonymous access disabled • Docker network isolation (observability network) • No ports exposed to external network

⚠️ PRODUCTION RECOMMENDATIONS:

• Use Docker secrets instead of .env files • Add TLS/SSL certificates for all services • Implement authentication for all endpoints • Set up reverse proxy (nginx/traefik) with SSL • Configure firewall rules • Enable audit logging • Implement log retention policies • Set up automated backups • Use separate networks for production • Rotate passwords regularly • Monitor failed login attempts

=============================================================================

📦 DOCKER VOLUMES

dfly_data - DragonflyDB persistent cache tempo_data - Tempo trace storage loki_data - Loki log storage prom_data - Prometheus time-series data grafana_data - Grafana dashboards and settings promtail_positions - Promtail read positions

To back up data: docker run --rm -v observability_grafana_data:/data -v $(pwd):/backup
alpine tar czf /backup/grafana-backup.tar.gz -C /data .

=============================================================================

🎯 FEATURE HIGHLIGHTS

✅ Complete observability stack in one command ✅ Auto-configured with best practices ✅ Trace-to-log correlation out of the box ✅ Service dependency mapping ✅ High-performance Redis-compatible cache ✅ Modern TSDB-based log storage ✅ Metrics generation from traces ✅ Health checks for all services ✅ Proper service startup ordering ✅ Localhost-only binding for security ✅ Persistent storage for all data ✅ JSON log parsing (zerolog compatible) ✅ 30-day log retention ✅ 48-hour trace retention ✅ Distributed tracing support ✅ Prometheus remote write enabled

=============================================================================

📚 ADDITIONAL RESOURCES

OpenTelemetry: https://opentelemetry.io/docs/ Grafana Tempo: https://grafana.com/docs/tempo/ Grafana Loki: https://grafana.com/docs/loki/ Prometheus: https://prometheus.io/docs/ Promtail: https://grafana.com/docs/loki/latest/clients/promtail/ DragonflyDB: https://www.dragonflydb.io/docs Grafana: https://grafana.com/docs/grafana/

=============================================================================

🤝 CONTRIBUTING

We welcome contributions! Please see CONTRIBUTING.md for guidelines.

By participating in this project, you agree to abide by our Code of Conduct.

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

=============================================================================

Created: November 2025 by Abdorizak Abdalla Hassan ⚙️
Version: 2.0
Repository: masjidAbuhureyra/go-observability-stack

=============================================================================

Name		Name	Last commit message	Last commit date
Latest commit History 13 Commits
.env.example		.env.example
CODE_OF_CONDUCT.md		CODE_OF_CONDUCT.md
CONTRIBUTING.md		CONTRIBUTING.md
LICENSE		LICENSE
README.md		README.md
docker-compose.yaml		docker-compose.yaml
grafana-datasources.yaml		grafana-datasources.yaml
loki.yaml		loki.yaml
otel-config.yaml		otel-config.yaml
prometheus.yaml		prometheus.yaml
promtail-config.yaml		promtail-config.yaml
tempo.yaml		tempo.yaml

Folders and files

Latest commit

History

Repository files navigation

Go Observability Stack – Production-Ready Monitoring

📐 ARCHITECTURE OVERVIEW

🚀 QUICK START GUIDE

📦 What's in the Package

STEP 1: Create .env File

STEP 2: Start the Stack

STEP 3: Verify Services

STEP 4: Configure Your Go Application

STEP 5: Ensure Log Directory Exists

STEP 6: Start Your Application

STEP 7: Go Application Integration Examples

Connecting to DragonflyDB (Redis)

Initializing OpenTelemetry with OTLP Exporter

🌐 ACCESS URLS

Service URL Credentials

Remote Access via SSH Tunnel

📊 GRAFANA CONFIGURATION

AUTO-PROVISIONED DATA SOURCES:

INTEGRATED FEATURES:

EXAMPLE QUERIES:

🔧 COMPONENT DETAILS

DRAGONFLY DB

OTEL COLLECTOR

TEMPO

LOKI

PROMTAIL

PROMETHEUS

GRAFANA

🐛 TROUBLESHOOTING

CHECK SERVICE HEALTH:

SERVICE-SPECIFIC DEBUGGING:

NETWORK DEBUGGING:

RESET EVERYTHING:

RESTART SINGLE SERVICE:

📝 CONFIGURATION FILES

🔒 SECURITY NOTES

✅ IMPLEMENTED SECURITY:

⚠️ PRODUCTION RECOMMENDATIONS:

📦 DOCKER VOLUMES

🎯 FEATURE HIGHLIGHTS

📚 ADDITIONAL RESOURCES

🤝 CONTRIBUTING

📄 License

About

Resources

License

Code of conduct

Contributing

Uh oh!

Stars

Watchers

Forks

Releases

Packages 0

Uh oh!

Contributors

Uh oh!

Packages