OSM Performance Management: Difference between revisions

From OSM Public Wiki
Jump to: navigation, search
(Updating PM basic functionality for Release FIVE)
Line 1: Line 1:
This is new feature available since the Release FOUR Lightweight build, which allows to continuously monitor VNF metrics and visualize them.
This documentation corresponds now to Release FIVE, previous documentation is deprecated.


==Basic functionality==
==Basic functionality==
By default, OSM MON allows to grab any metric, from any VDU and put it on the Kafka Bus, specifically in the 'metrics_response' topic.
OSM MON features a "mon-collector" module which will collect metrics whenever specified at the descriptor level.


Starting with OSM R4, an OSM CLI command is available to export a metric to from the VIM to the bus (in the example: NS name: "vnf01", VNF index: 1, VDU name: "ubuntuvnf_vnfd-VM", metric type: "cpu_utilization")
Please note that:
 
* As of Release 5.0.0, metric export has been tested with OpenStack VIM with Keystone v3 authentication and legacy or Gnocchi-based telemetry services.  Other VIM types will soon be added during the Release FIVE cycle.
osm ns-metric-export --ns vnf01 --vnf 1 --vdu ubuntuvnf_vnfd-VM --metric cpu_utilization
* For metrics to be exported, they have to exist at the VIM (Telemetry system) or at the VNF (Juju Metrics), so for recently created VDUs, it might take some time after they start being successfully collected by MON.
 
Possible metric names are: cpu_utilization, average_memory_utilization, disk_read_ops,disk_write_ops, disk_read_bytes, disk_write_bytes, packets_dropped_<nic number>, packets_received, packets_sent


The specific result can be read directly from the Kafka bus topic using a external tool, or can be seen in the MON logs by running a docker logs <MON Container ID>
=== Describing VIM Metrics ===


You can also add the '--interval' option to leave it running continuously, for example, every 10 seconds:
Every metric made available by the VIM for each VDU, has to be described both at the VDU level, and then at the VNF level.  For example:


  osm ns-metric-export --ns vnf01 --vnf 1 --vdu ubuntuvnf_vnfd-VM --metric cpu_utilization --interval 10
  vdu:
id: vdu1
  ... 
  monitoring-param:
        - id: metric_vdu1_cpu
          nfvi-metric: cpu_utilization
        - id: metric_vdu1_memory
          nfvi-metric: average_memory_utilization
...
monitoring-param:
-   id: metric_vim_vnf1_cpu
    name: metric_vim_vnf1_cpu
    aggregation-type: AVERAGE
    vdu-monitoring-param:
      vdu-ref: vdu1
      vdu-monitoring-param-ref: metric_vdu1_cpu
-  id: metric_vim_vnf1_memory
    name: metric_vim_vnf1_memory
    aggregation-type: AVERAGE
    vdu-monitoring-param:
      vdu-ref: vdu1
      vdu-monitoring-param-ref: metric_vdu1_memory


Finally, you can leave it running in the background using Linux default tools:
As you can see, a list of "NFVI metrics" is defined first at the VDU level, which contain an ID and the corresponding normalized metric name (in this case, "cpu_utilization" and "average_memory_utilization")
Then, at the VNF level, a list of monitoring-params is referred, with an ID, name, aggregation-type and their source, which is a 'vdu-monitoring-param' in this case.


osm ns-metric-export --ns vnf01 --vnf 1 --vdu ubuntuvnf_vnfd-VM --metric cpu_utilization --interval 10 > /dev/null 2>&1 &
A complete VNFD example can be downloaded from [https://osm-download.etsi.org/ftp/osm-4.0-four/4th-hackfest/packages/webserver_vimmetric_autoscale_vnfd.tar.gz here].


Please note that:
=== Describing VNF Metrics ===
* As of Release 4.0.0, metric export has been tested with OpenStack VIM with Keystone v3 authentication and legacy or Gnocchi-based telemetry services.  VNF metrics and other VIM types will soon be added during the Release Four cycle.
* For metrics to be exported, they have to exist at the VIM, so for recently created VDUs, it might take some time after they start appearing at the bus.


==Experimental functionality==
Metrics can also be collected directly from VNFs using VCA, through the [https://docs.jujucharms.com/2.4/en/developer-metrics Juju Metrics] framework. A simple charm containing a metrics.yaml file at its root folder specifies the metrics to be collected and the associated command.
Some extensions have been added to the OSM installer to include an optional 'OSM Performance Management' stack, consisting of a 'Kafka Exporter' component that reads the metrics from the bus, exposes them in Prometheus so it can store them in its TSDB, and presents them in Grafana.


Basic architecture is as follows:
For example, the following metrics.yaml file collects three metrics from the VNF, called 'users', 'load' and 'load_pct'


[[File:OSM Metrics Architecture 4.png|800px|Diagram of OSM Metrics Experimental add-ons]]
metrics:
    users:
      type: gauge
      description: "# of users"
      command: who|wc -l
    load:
      type: gauge
      description: "5 minute load average"
      command: cat /proc/loadavg |awk '{print $1}'
    load_pct:
      type: gauge
      description: "1 minute load average percent"
      command: cat /proc/loadavg  | awk '{load_pct=$1*100.00} END {print load_pct}'         


=== Enabling the OSM Performance Management Stack ===
Please note that the granularity of this metric collection method is fixed to 5 minutes.
If you want to install OSM along with the PM stack, run the installer as follows:


./install_osm.sh --pm_stack
After metrics.yaml is available, you have two options for describing the metric collection in the VNFD:


If you just want to add the PM stack to an existing OSM R4 Lightweight build, run the installer as follows:
1) VNF-level VNF metrics


   ./install_osm.sh -o pm_stack
mgmt-interface:
   cp: vdu_mgmt # is important to set the mgmt VDU or CP for metrics collection
vnf-configuration:
  initial-config-primitive:
  ...
  juju:
    charm: testmetrics
  metrics:
    - name: load
    - name: load_pct
    - name: users 
...             
monitoring-param:
-  id: metric_vim_vnf1_load
    name: metric_vim_vnf1_load
    aggregation-type: AVERAGE
    vnf-metric:
      vnf-metric-name-ref: load
-  id: metric_vim_vnf1_loadpct
    name: metric_vim_vnf1_loadpct
    aggregation-type: AVERAGE
    vnf-metric:
      vnf-metric-name-ref: load_pct


This will install three additional docker containers (Kafka Exporter, Prometheus and Grafana)
A complete VNFD example can be downloaded from [https://osm-download.etsi.org/ftp/osm-4.0-four/4th-hackfest/packages/ubuntuvm_vnfvdumetric_autoscale_vnfd.tar.gz here].


If you need to remove it at some point in time, just run the following command:
2) VDU-level VNF metrics
docker stack rm osm_metrics


If you need to deploy the stack again after being removed:
vdu:
  docker stack deploy -c /etc/osm/docker/osm_metrics/docker-compose.yml osm_metrics
- id: vdu1
  ...
  interface:
  - ...
    mgmt-interface: true ! is important to set the mgmt interface for metrics collection
    ...
  vdu-configuration:
    initial-config-primitive:
    ...
    juju:
      charm: testmetrics
    metrics:
      - name: load
      - name: load_pct
      - name: users 
...             
monitoring-param:
-  id: metric_vim_vnf1_load
    name: metric_vim_vnf1_load
    aggregation-type: AVERAGE
    vdu-metric:
      vdu-ref: vdu1
      vdu-metric-name-ref: load
  -   id: metric_vim_vnf1_loadpct
    name: metric_vim_vnf1_loadpct
    aggregation-type: AVERAGE
    vdu-metric:
      vdu-ref: vdu1
      vdu-metric-name-ref: load_pct


=== Testing the OSM PM Stack ===
A complete VNFD example can be downloaded from [https://osm-download.etsi.org/ftp/osm-4.0-four/4th-hackfest/packages/ubuntuvm_vnfmetric_autoscale_vnfd.tar.gz here].  
1. Create a continuous metric, that runs in the background as indicated in the first section.


2. Check if the Kafka Exporter is serving the metric to Prometheus, by visiting http://1.2.3.4:12340/metrics, replacing 1.2.3.4 with the IP address of your host. Metrics should appear in the following text format:
As you can see, a list of "metrics" is defined first either at the VNF or VDU "configuration" level, which contain a name that comes from the metrics.yaml file.
Then, at the VNF level, a list of monitoring-params is referred, with an ID, name, aggregation-type and their source, which can be a "vdu-metric" or a "vnf-metric" in this case.


# HELP kafka_exporter_topic_average_memory_utilization
==Interacting with Prometheus TSDB==
# TYPE kafka_exporter_topic_average_memory_utilization gauge
kafka_exporter_topic_average_memory_utilization{resource_uuid="5599ce48-a830-4c51-995e-a663e590952f",} 200.0
# HELP kafka_exporter_topic_cpu_utilization
# TYPE kafka_exporter_topic_cpu_utilization gauge
kafka_exporter_topic_cpu_utilization{resource_uuid="5599ce48-a830-4c51-995e-a663e590952f",} 0.7950777152296741


Note: if metrics appear at MON logs but not at this web service, we may have hit a rare issue (under investigation), where Kafka Exporter loses connection to the bus. 
[WIP]
* To confirm this issue, access the kafka-exporter container and check the log for a message like 'dead coordinator' (tail kafka-topic-exporter.log)
* To recover, just reload the service using 'docker service update --force osm_metrics_kafka-exporter'.


3. Visit Grafana at http://1.2.3.4:3000, replacing 1.2.3.4 with the IP address of your host.  Login with admin/admin credentials and visit the OSM Sample Dashboard.  It should already show graphs for CPU and Memory.  You can clone them and customize as desired.
==Experimental functionality==


[[File:OSM_Grafana_Example_4.png|800px|Diagram of OSM Grafana Example]]
[WIP]




{{Feedback}}
{{Feedback}}

Revision as of 15:11, 3 December 2018

This documentation corresponds now to Release FIVE, previous documentation is deprecated.

Basic functionality

OSM MON features a "mon-collector" module which will collect metrics whenever specified at the descriptor level.

Please note that:

  • As of Release 5.0.0, metric export has been tested with OpenStack VIM with Keystone v3 authentication and legacy or Gnocchi-based telemetry services. Other VIM types will soon be added during the Release FIVE cycle.
  • For metrics to be exported, they have to exist at the VIM (Telemetry system) or at the VNF (Juju Metrics), so for recently created VDUs, it might take some time after they start being successfully collected by MON.

Describing VIM Metrics

Every metric made available by the VIM for each VDU, has to be described both at the VDU level, and then at the VNF level. For example:

vdu:
id: vdu1
  ...  
  monitoring-param:
        - id: metric_vdu1_cpu
          nfvi-metric: cpu_utilization
        - id: metric_vdu1_memory
          nfvi-metric: average_memory_utilization
...
monitoring-param:
-   id: metric_vim_vnf1_cpu
    name: metric_vim_vnf1_cpu
    aggregation-type: AVERAGE
    vdu-monitoring-param:
      vdu-ref: vdu1
      vdu-monitoring-param-ref: metric_vdu1_cpu
-   id: metric_vim_vnf1_memory
    name: metric_vim_vnf1_memory
    aggregation-type: AVERAGE
    vdu-monitoring-param:
      vdu-ref: vdu1
      vdu-monitoring-param-ref: metric_vdu1_memory

As you can see, a list of "NFVI metrics" is defined first at the VDU level, which contain an ID and the corresponding normalized metric name (in this case, "cpu_utilization" and "average_memory_utilization") Then, at the VNF level, a list of monitoring-params is referred, with an ID, name, aggregation-type and their source, which is a 'vdu-monitoring-param' in this case.

A complete VNFD example can be downloaded from here.

Describing VNF Metrics

Metrics can also be collected directly from VNFs using VCA, through the Juju Metrics framework. A simple charm containing a metrics.yaml file at its root folder specifies the metrics to be collected and the associated command.

For example, the following metrics.yaml file collects three metrics from the VNF, called 'users', 'load' and 'load_pct'

metrics:
   users:
     type: gauge
     description: "# of users"
     command: who|wc -l
   load:
     type: gauge
     description: "5 minute load average"
     command: cat /proc/loadavg |awk '{print $1}'
   load_pct:
     type: gauge
     description: "1 minute load average percent"
     command: cat /proc/loadavg  | awk '{load_pct=$1*100.00} END {print load_pct}'          

Please note that the granularity of this metric collection method is fixed to 5 minutes.

After metrics.yaml is available, you have two options for describing the metric collection in the VNFD:

1) VNF-level VNF metrics

mgmt-interface:
 cp: vdu_mgmt # is important to set the mgmt VDU or CP for metrics collection
vnf-configuration:
  initial-config-primitive:
  ...
  juju:
    charm: testmetrics
  metrics:
    - name: load 
    - name: load_pct
    - name: users  
...              
monitoring-param:
-   id: metric_vim_vnf1_load
    name: metric_vim_vnf1_load
    aggregation-type: AVERAGE
    vnf-metric:
      vnf-metric-name-ref: load
-   id: metric_vim_vnf1_loadpct
    name: metric_vim_vnf1_loadpct
    aggregation-type: AVERAGE
    vnf-metric:
      vnf-metric-name-ref: load_pct

A complete VNFD example can be downloaded from here.

2) VDU-level VNF metrics

vdu:
- id: vdu1
  ...
  interface:
  - ...
    mgmt-interface: true ! is important to set the mgmt interface for metrics collection
    ...
  vdu-configuration:
    initial-config-primitive:
    ...
    juju:
      charm: testmetrics
    metrics:
      - name: load 
      - name: load_pct
      - name: users  
...              
monitoring-param:
-   id: metric_vim_vnf1_load
    name: metric_vim_vnf1_load
    aggregation-type: AVERAGE
    vdu-metric:
      vdu-ref: vdu1
      vdu-metric-name-ref: load
-   id: metric_vim_vnf1_loadpct
    name: metric_vim_vnf1_loadpct
    aggregation-type: AVERAGE
    vdu-metric:
      vdu-ref: vdu1
      vdu-metric-name-ref: load_pct

A complete VNFD example can be downloaded from here.

As you can see, a list of "metrics" is defined first either at the VNF or VDU "configuration" level, which contain a name that comes from the metrics.yaml file. Then, at the VNF level, a list of monitoring-params is referred, with an ID, name, aggregation-type and their source, which can be a "vdu-metric" or a "vnf-metric" in this case.

Interacting with Prometheus TSDB

[WIP]

Experimental functionality

[WIP]


Your feedback is most welcome!
You can send us your comments and questions to OSM_TECH@list.etsi.org
Or join the OpenSourceMANO Slack Workplace
See hereafter some best practices to report issues on OSM