Seeing what apps are doing

You can't improve what you can't see. A lot of my work on monitoring was about turning a flood of logs into a few screens that people actually looked at.

Two kinds of questions

Monitoring answers two different questions, and it helps to keep them apart:

• Is it working right now? Sensu handled this with checks and alerts.
• How is it behaving over time? Splunk handled this with queries and dashboards for performance and capacity.

Building dashboards people use

• Start from a question, not from the data. "Are response times getting worse after releases?" makes a far better dashboard than "show all the logs".
• A few panels, not fifty. If a dashboard needs scrolling, nobody reads the bottom half.
• Show trends, not just numbers. Capacity problems creep in slowly. A line over three months tells you more than today's value.

Alerts that don't cry wolf

The fastest way to make people ignore alerts is to send too many. Every alert should mean someone needs to do something. If it doesn't, it belongs on a dashboard instead.

What I learned

Good monitoring is a conversation with the team: what do you worry about, and how would you know? The tools come second.