Prometheus Support for Darkroom

It was my pre-bootcamp project in Gojek, to add Prometheus support to Darkroom for scraping metrics and displaying it on a dashboard using Grafana.

It was my first experience working on an open source project, it was interesting as well as daunting to work on such a big code base. We were a team of two, me and Chaitanya, assigned as our pre-bootcamp project.

Let me quickly introduce Darkroom and Prometheus.

Darkroom logo

Darkroom is an image processor tool and acts as an image proxy on your image source.
It is stateless, can use multiple storage backends and customisable where you can implement your own Storage and Processor interfaces to gain custom functionality while still keeping other Darkroom Server functionality.

Prometheus logo

Prometheus is an open-source systems monitoring and alerting toolkit.
It has inbuilt support for Grafana (a multi-platform open source analytics and interactive visualisation web application) which will be used in our project to display metrics data for visualisation.

There were two types of metrics to be scraped from Darkroom,

• Duration metrics for all image processing operations
• Error count metrics while handling requests for image processing

There were following image processing operations in darkroom,

• decode
• encode
• crop
• scale
• resize
• grayscale
• blur
• fix orientation
• flip
• rotate

First I followed the setup guide given in the official documentation of Darkroom to install and run the project on my local system using docker containers.
Setup and usage documentation can be found here.

First step was to add Prometheus image to docker container which would listen at port 9090 and map it to docker container’s port 9090, so we added below snippet to docker-compose.yml.

prometheus:    
 image: quay.io/prometheus/prometheus:latest    
 ports:      
   - 9090:9090    
 volumes:      
   - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro

Then we added “/metrics” to httpHandler to add endpoint from where the metrics of Darkroom will be scraped by Prometheus server.

r.Handle("/metrics", promhttp.HandlerFor(registry, promhttp.HandlerOpts{}))

Added configurations for Prometheus in prometheus.yml file which defined configs such as scrape_intervals, job_name, targets from where metrics had to be scraped etc.

Since there was also support of StatsD client for metrics, we decided to implement a metrics interface, which looked like this :

type MetricService interface { 
    TrackDuration(imageProcess string, start time.Time, ImageData []byte)  
    CountImageHandlerErrors(kind string)
}

We wanted to implement two types of metrics,

• ImageProcessDuration Metrics : for time taken to process image per operation
• ImageHandlerErrorCounter Metrics : to count error while handling image process requests

It was decided to use custom registry to avoid panic due to duplicate registration. So, a struct type of PrometheusService was created with two types of metrics.

type prometheusService struct {
    imageProcessDuration *prometheus.HistogramVec
    imageHandlerErrorCounter *prometheus.CounterVec
    reg *prometheus.Registry
}

Added metricService method to return NewPrometheusService instance.

func NewPrometheus(reg *prometheus.Registry) MetricService { 
    p := &prometheusService{  
           imageProcessDuration: prometheus.NewHistogramVec(
                prometheus.HistogramOpts{
                   Name: "image_process_duration",
                   Help: "Time taken by each stage to process requested image",
                }, []string{"process", "image_type"}),
           imageHandlerErrorCounter: prometheus.NewCounterVec(
                prometheus.CounterOpts{
                   Name: "image_handler_errors",
                   Help: "The total number of errors for each storage and processor",
                }, []string{"error_type"}),
           reg: reg,
    }
    p.registerMetrics()  
    return p
}

Methods to track duration for image processing and to count errors while processing were added.

func (p prometheusService) TrackDuration(imageProcess string, start time.Time, ImageData []byte) { 
    imageType := p.getImageType(ImageData)
    p.imageProcessDuration.WithLabelValues(imageProcess, imageType).Observe(time.Since(start).Seconds())
}
func (p prometheusService) CountImageHandlerErrors(kind string) {
    p.imageHandlerErrorCounter.WithLabelValues(kind).Inc()
}

Added Grafana provisioning for metrics in docker-compose.yml.

grafana:    
    image: grafana/grafana:5.1.0    
    ports: 
        - 8080:3000    
    environment: 
        - GF_SECURITY_ADMIN_USER=${ADMIN_USER:-admin} 
        - GF_SECURITY_ADMIN_PASSWORD=${ADMIN_PASSWORD:-changeme}   
        - GF_USERS_ALLOW_SIGN_UP=false    
    volumes:     
        - ./grafana/provisioning:/etc/grafana/provisioning

Integration of Prometheus with Darkroom was finished, metrics from “/metrics” endpoint were getting scraped and were displayed beautifully on Grafana dashboard for visualisation.

Grafana dashboard example

It was a very good experience working on an interesting open source project. I learnt a lot how metrics are pulled, how docker containers work, their uses and benefits. It taught me how to work with big codebases.