Author: Jose Antonio Martinez (jamartinezv@minsait.com)
-->
-# Monitoring in Kubernetes based OSM
+# OSM Monitoring
## Introduction
-This implementation deploys a PM stack based on Prometheus Operator plus a series of exporters for monitoring the OSM nodes and third party software modules (Kafka, mongodb and mysql)
+This is an utility to monitor the OSM nodes and pods in the Kubernetes deployment. Metrics are stored in Prometheus and accessible in Grafana. Note that those "Prometheus" instance is not the same in the OSM core, but different one, aimed at the monitoring of the platform itself.
-In a high level, it consists of two scripts that deploy/undeploy the required objects in a previously existing Kubernetes based OSM installation.
-Those scripts use already existing and freely available software: Helm, Kubernetes Operator and a set of exporters and dashboards pretty much standard. Helm server part (tiller) and charts deployed depends on Kubernetes version 1.15.x. Charts versions are pre-configured in an installation script and can be easily changed.
+## Requirements
-As a result, there will be 3 folders in Grafana:
+OSM must be/have been deployed using the Kubernetes installer (that is, with the -c k8s option).
-- Summary: with a quick view of the platform global status.
-- OSM Third Party Modules: dashboards for MongoDB, MyslqDB and Kafka.
-- Kubernetes cluster: dashboards for pods, namespaces, nodes, etc.
+## Versions
-## Requirements
+For reference, the versions for the external components used are as follows:
-- Kubernetes 1.15.X
-- OSM Kubernetes version Release 7
+* PROMETHEUS_OPERATOR=6.18.0
+* PROMETHEUS_MONGODB_EXPORTER=2.3.0
+* PROMETHEUS_MYSQL_EXPORTER=0.5.1
+* HELM_CLIENT=2.15.2
-## Components
+## Functionality
-- Installs the helm client on the host where the script is run (if not already installed)
-- Creates a service account in the k8s cluster to be used by tiller, with sufficient permissions to be able to deploy kubernetes objects.
-- Installs the helm server part (tiller) and assigns to tiller the previously created service account (if not already installed)
-- Creates a namespace (monitoring) where all the components that are part of the OSM deployment monitoring `pack` will be installed.
-- Installs prometheus-operator using the `stable/prometheus-operator` chart which is located at the default helm repository (<https://kubernetes-charts.storage.googleapis.com/>). This installs a set of basic metrics for CPU, memory, etc. of hosts and pods. It also includes grafana and dashboards.
-- Installs an exporter for mongodb using the `stable/prometheus-mongodb-exporter` chart, which is located at the default helm repository (<https://kubernetes-charts.storage.googleapis.com/>).
-- Adds a dashboard for mongodb to grafana through a local yaml file.
-- Installs an exporter for mysql using the `stable/prometheus-mysql-exporter` chart which is located at the default helm repository (<https://kubernetes-charts.storage.googleapis.com/>).
-- Adds a dashboard for mysql to grafana through a local yaml file.
-- Installs an exporter for kafka using a custom-build helm chart with a deployment and its corresponding service and service monitor with local yaml files. We take the kafka exporter from <https://hub.docker.com/r/danielqsj/kafka-exporter>.
-- Add a dashboard for kafka to grafana through a local yaml file.
+Kubernetes cluster metrics (for nodes, pods, deployments, etc.) are stored in the dedicated Prometheus instance and accessible using Grafana.
-## Versions
+"Prometheus-operator" (<https://github.com/helm/charts/tree/master/stable/prometheus-operator>) provides the basic components and the monitoring of the basic Kubernetes resources. Additional "exporters" are used to gather metrics from Kafka, Mysql and Mongodb.
+It is important to note that Grafana is not installed with this chart because we are using Grafana installed with OSM core.
-We use the following versions:
+## Install procedure
-- PROMETHEUS_OPERATOR=6.18.0
-- PROMETHEUS_MONGODB_EXPORTER=2.3.0
-- PROMETHEUS_MYSQL_EXPORTER=0.5.1
-- HELM_CLIENT=2.15.2
+There are two ways to install the monitoring component based on the OSM global installer (<https://osm-download.etsi.org/ftp/osm-7.0-seven/install_osm.sh>)
-## Install
+* Using the --k8s_monitor switch in the OSM installation:
-Note: This implementation is dependent on the Kubernetes OSM deployment, and the installation script must be executed AFTER the Kubernetes deployment has been completed. Notice that it is not applicable to the basic docker deployment.
+```bash
+./install_osm.sh -c k8s --k8s_monitor
+```
+
+* As a separated component (K8s based OSM only):
```bash
+./install_osm.sh -o k8s_monitor
+```
+
+All the components will be installed in the "monitoring" namespace. In addition, for debugging purposes, there is a standalone script is available in `devops/installers/k8s/install_osm_k8s_monitoring.sh`. To see the available options, type --help.
+
+```sh
usage: ./install_osm_k8s_monitoring.sh [OPTIONS]
Install OSM Monitoring
OPTIONS
-h / --help : print this help
```
-## Uninstall
+## Access to Grafana
+
+The Grafana console can be accessed on the IP address of any node using port 3000, since a NodePort service is used: `http://<ip_your_osm_host>:3000`
+
+The initial credentials are:
+
+* Username: admin
+* Password: admin
+
+## Uninstall procedure
-To uninstall the utility you must use the installation script.
+Use the uninstall script
```sh
-./uninstall_osm_k8s_monitoring.sh
+./install_osm.sh -o k8s_monitor --uninstall
```
-It will uninstall all components of this utility. To see the options type --help.
+In addition, for debugging purposes, there is a standalone script is available in `devops/installers/k8s/uninstall_osm_k8s_monitoring.sh`. To see the available options type --help.
-```sh
+```bash
usage: ./uninstall_osm_k8s_monitoring.sh [OPTIONS]
Uninstall OSM Monitoring
OPTIONS
-h / --help : print this help
```
-## Access to Grafana Web Monitoring
+## Grafana Dashboards
-To view the WEB with the different dashboards it is necessary to connect to the service "grafana" installed with this utility
-and view the NodePort that uses. If the utility is installed with the default namespace "monitoring" you must type this:
+Dashboard are organized in two folders:
-```sh
-kubectl get all --namespace monitoring
-```
+* The folder "Kubernetes cluster" contains the dashboards available upstream as part of the standard prometheus operator helm installation:
-You must see the NodePort (greater than 30000) that uses the grafana service and type in your WEB browser:
+ * Kubernetes components (api server, kubelet, pods, etc)
+ * Nodes of the cluster.
+ * Prometheus operator components.
-```sh
- http://<ip_your_osm_host>:<nodeport>
+* The folder "Open Source MANO" contains additional dashboards customized for OSM:
+ * Summary with a quick view of the overall status.
+ * Host information
+ * Third party components: Kafka, MongoDB, MySQL.
+
+## Adding new dashboards
+
+New dashboards for OSM components should be included in "Open Source MANO" folder. Once we have the dashboard json file, please follow the instructions below to incorporate it into Grafana.
+
+```bash
+kubectl -n monitoring create configmap <configmap-name> --from-file=<dashboard-json-file>
+kubectl -n monitoring patch configmap <configmap-name> --patch '{"metadata": {"labels": {"grafana_dashboard": "1"},{"annotations": {k8s-sidecar-target-directory: "/tmp/dashboards/Open Source MANO"}}}}'
+```
+where <configmap-name> and <dashboard-json-file> needs to be replaced with desired values. A proposal is that <configmap-name> begins with "osm-monitoring-osm-"
+
+Once configmap is created and patched, we can download the manifest file for future use with next command:
```
-
-- Username: admin
-- Password: prom-operator
+kubectl -n monitoring get configmap <configmap-name> -o yaml > <confimap-file>
+```
+
+Grafana Sidecar will read the label `grafana_dashboard: "1"` in the configmap and upload the dashboard information to Grafana.
+
+The current dashboards can also be updated. It is only needed to modify/update the required yaml file available in `devops/installers/k8s` and apply them via kubectl. As an example `kubectl -n monitoring apply -f summary-dashboard.yaml` will update the changes made in the summary dashboard.
projects / osm / devops.git / blobdiff