New monitoring pipeline architecture as an experimental option in the installer

Proposers

• Gerardo García (Telefónica)
• Francisco Javier Ramón (Telefónica)
• David Aguilar (Indra)

Description

A new architecture for monitoring based on pipelines has been introduced in previous OSM meetings. This new architecture relies on the use of Apache Airflow and Prometheus (and its exporters):

• Apache Airflow is a platform to create and manage pipelines as DAGs (Directed Acyclic Graphs) consisting of a set of tasks organized in a way that reflects their relationships and dependencies.
• Prometheus is the system used today in OSM to store monitoring metrics. Prometheus is not only a TSDB, but there is a large amount of exporters that could be scraped by Prometheus to get their metrics, thus making easy the integration with existing platforms. Those exporters include VIM exporters like Azure Metric Exporter, Google Cloud Operations Suite (formerly StackDriver) or CloudWatch Exporter, and VNF metric exporters like the SNMP Exporter used today by OSM. In addition, Prometheus community also provides a Pushgateway for those scenarios where an exporter does not exist, acting as an intermediary service allowing other systems to push their metrics and making them available to Prometheus.

This feature will cover the installation of Apache Airflow and Prometheus Pushgateway as an experimental option in the installer. When combined with features 10964, 10965 and 10966, they will allow VIM status, VNF status and NS status to be populated in Prometheus.

Demo or definition of done

• A demo will be showcased, where OSM will be installed with the new experimental option. Apache Airflow and Prometheus Pushgateway will be installed. When combined with the mentioned, VIM status, VNF status and NS status will be automatically populated in Prometheus.
• A daily job will be replayed replacing the installation command to add the new option. All Robot tests should work flawlessly.