Grafana And Prometheus

Description

Prometheus is a monitoring solution for storing time series data like metrics. Grafana allows to visualize the data stored in Prometheus (and other sources). This sample demonstrates how to capture NServiceBus metrics, storing these in Prometheus and visualizing these metrics using Grafana.

Course Content 

1-Introduction to system monitoring and Grafana

• Concepts of telemetry
• Push- and pull-based telemetry
• Sampling, data retention and down sampling
• Grafana and data sources

2-Details of Grafana

• Grafana installation
• Accessing Grafana
• Creating first dashboard
• Dashboards, rows and panels
• Timerange selector
• Relative time and Time shift

3-Data sources

• Prometheus architecture
• Installing Prometheus
• Accessing Prometheus web interface
• Installing node_exporter
• Getting metrics
• Querying Prometheus
• Scraping metrics to Prometheus
• Graphite architecture
• Installing Graphite
• Feeding Graphite
• Adding data sources to Grafana

4-Deep dive into Grafana panels

• Graph
• Singlestat
• Gauge
• Bargauge
• Heatmap
• Textpanel
• Tablepanel
• Dashboard list
• Plugin panels
• Manipulating panels

5-Annotations and alerting

• Annotations
• Alerts
• Alert list panel
• E-mail and other ways of alert notifications

6-Explore and variables

• Explore functionality
• Templating with variables
• Dynamic panels based on variables

7-Loki – Grafana log aggregation system

• Loki overview
• Installation
• Exploring logs

8-Grafana administration

• Organizing Grafana
• Migrating to MySQL
• High Availability in Grafana
• Running Grafana behind reverse proxy
• Securing Grafana with SSL certificate
• Troubleshooting Grafana

Prometheus

9- Introduction to Prometheus

• What is Prometheus
•  Basic Terminologies in Prometheus

10- Architecture

•  Prometheus Architecture
•  How does it work?

11- Installation and Walk through

• Prometheus Installation
• Prometheus UI walks through
• Understanding Prometheus Configuration File
•  Read application endpoints from Prometheus First Run

12- Exporters

• What are Exporters?
• Node Exporter – Monitoring Linux Systems

13- PromQL

• Data Types in PromQL
• Selectors & Matchers
• Binary Operators
• ‘ignoring’ and ‘on’ keywords
• Aggregation Operators
• Functions – ‘rate’ & ‘irate’
• Functions – changes, deriv, predict_linear
• Functions continued

14-Client Libraries – Adding Instrumentation to Java/Spring Boot Application

• What are Client Libraries and Metric Types
• Exposing Metrics from Python app using Prometheus Client
• Counter Metrics Exposition
• Adding Labels to Exposed Metrics
• Gauge Metrics Exposition
• Summary Metrics Exposition
• Histogram Metrics Exposition

15-Quantification of Instrument

• Why do you need to instrument?
• What to Instrument?
• How much to Instrument?

16-Re-coding Rules

• What are recording rules
• Reload Prometheus Configurations on-the-fly
• Writing Recording Rules
• Writing Recording Rules continued
• Add Multiple Rules
• Best Practices for Recording Rules

17-Alerting

• What is Alerting?
• Writing & Firing the first Alert
• ‘for’ clause
• Adding Labels to Alerts
• Installing Alertmanager
• Adding Alert Notifier – Gmail
• Sending Alert Notifications – Gmail
• Templating the Alerts

18-Create Routing Tree for Alerts

•  Why use routing tree?
•  Write Alerting Rules for the Use Case
• Coding the Routing Tree
• Run the Routing Tree
• Grouping the Alerts
• Throttling the Alerts
• Inhibiting the Alerts
• Silencing the Alerts
•  ‘continue’ clause

19-Blackbox Exporter and Relabeling

• What is Blackbox exporter
• Download Blackbox exporter
• ‘http’ probe module
• tcp’ & ‘icmp’ probe module
• ‘dns’ probe module
• Scraping targets via Blackbox
• Relabeling

20- Push-gateway

• Introduction to Push-gateway
• Getting Started with Push-gateway
• Push metrics to Push-gateway
• Automate Pushing metrics using Cron job
• App pushing metrics to Push-gateway
• Push-gateway Pitfalls

21- Service Discovery

• Introduction to Service Discovery
• Static Service Discovery
•  File based service discovery

22-Create custom exporters

• Introduction
• Creating Target application
• Writing Custom Exporter

23-Prometheus HTTP API

• Running Prometheus with HTTP API

24- Prometheus on a Docker Container