Monitoring for the private Docker cloud project

A few days ago I blogged about my holiday project (Building a private cloud with Docker [1]) which is just a proof of concept as it lacks important features. One of them is monitoring which is a tough topic in cluster environments.

There are a few open source, free to use monitoring solutions for Docker out there, the most interesting one (imho) being cadvisor [2] for monitoring Docker containers and heapster [3] for aggregating cadvisor nodes in a cluster. Unfortunately I wasn’t able to get heapster running (I suspect a go-lang version mismatch), so once again I scripted my way through the jungle.

The result are two new scripts in the project repository [4]: node-stats, which reports statistics for a node and cloud-stats, which runs node-stats on every known node and aggregates the results.

node-stats connects to a node via SSH and collects filesystem and memory statistics from the kernel, and container statistics from Docker. It combines the output of those statistics into a comprehensive report; the fun part is the (for my league) elaborate parsing of intermediate outputs.

cloud-stats iterates over all known nodes (the list-nodes script knows them) and sequentially calls node-stats for each node and prints the results. The generated report is far from the quality what other tools like cadvisor or nagios generate and it is far from real time, but it’s a start and it doesn’t require any special software running on either the manager or the worker nodes.