2024-04-24T22:33:18.131156465Z Created MongoClient to connect to MongoDB!
2024-04-24T22:33:18.131198291Z Generating prometheus config files
2024-04-24T22:33:28.665321342Z Jobs detected: 0
2024-04-24T22:33:28.672650171Z Writing new config file to /etc/prometheus/prometheus.yml
2024-04-24T22:33:28.677467250Z alerting:
2024-04-24T22:33:28.677479260Z   alertmanagers:
2024-04-24T22:33:28.677482857Z   - static_configs:
2024-04-24T22:33:28.677485939Z     - targets:
2024-04-24T22:33:28.677489885Z       - alertmanager:9093
2024-04-24T22:33:28.677492845Z global:
2024-04-24T22:33:28.677499700Z   evaluation_interval: 15s
2024-04-24T22:33:28.677503055Z   scrape_interval: 15s
2024-04-24T22:33:28.677506068Z rule_files:
2024-04-24T22:33:28.677509063Z - osm_metric_rules.yml
2024-04-24T22:33:28.677511968Z - osm_alert_rules.yml
2024-04-24T22:33:28.677515355Z scrape_configs:
2024-04-24T22:33:28.677518188Z - job_name: mon_exporter
2024-04-24T22:33:28.677521039Z   static_configs:
2024-04-24T22:33:28.677524193Z   - targets:
2024-04-24T22:33:28.677527044Z     - mon:8000
2024-04-24T22:33:28.677530070Z - honor_labels: true
2024-04-24T22:33:28.677532873Z   job_name: pushgateway
2024-04-24T22:33:28.677535745Z   scrape_interval: 30s
2024-04-24T22:33:28.677538623Z   static_configs:
2024-04-24T22:33:28.677541514Z   - targets:
2024-04-24T22:33:28.677545116Z     - pushgateway-prometheus-pushgateway:9091
2024-04-24T22:33:28.677548269Z 
2024-04-24T22:33:28.684804165Z Alerts read: 0
2024-04-24T22:33:28.687009750Z Writing new alerts file to /etc/prometheus/osm_alert_rules.yml
2024-04-24T22:33:28.689621813Z groups:
2024-04-24T22:33:28.689654197Z - name: osm_alert_rules
2024-04-24T22:33:28.689657525Z   rules:
2024-04-24T22:33:28.689660364Z   - alert: vdu_down
2024-04-24T22:33:28.689663093Z     annotations:
2024-04-24T22:33:28.689666427Z       description: VDU {{ $labels.vm_id }} in VIM {{ $labels.vim_id }} has been down
2024-04-24T22:33:28.689669547Z         for more than 3 minutes. NS instance id is {{ $labels.ns_id }}
2024-04-24T22:33:28.689672520Z       summary: VDU {{ $labels.vm_id }} in VIM {{ $labels.vim_id }} is down
2024-04-24T22:33:28.689676355Z     expr: vm_status_extended != 1
2024-04-24T22:33:28.689679125Z     for: 3m
2024-04-24T22:33:28.689681820Z 
2024-04-24T22:33:28.691319874Z New config written, updating prometheus
2024-04-24T22:33:33.760896533Z Prometheus config update successful
2024-04-24T22:33:38.766733172Z Listening to changes in prometheus jobs and alerts collections
2024-04-24T22:56:57.567215209Z Changes detected, updating prometheus config
2024-04-24T22:56:57.568864884Z Jobs detected: 0
2024-04-24T22:56:57.576562207Z Writing new config file to /etc/prometheus/prometheus.yml
2024-04-24T22:56:57.580399480Z alerting:
2024-04-24T22:56:57.580416631Z   alertmanagers:
2024-04-24T22:56:57.580420488Z   - static_configs:
2024-04-24T22:56:57.580423857Z     - targets:
2024-04-24T22:56:57.580427608Z       - alertmanager:9093
2024-04-24T22:56:57.580430552Z global:
2024-04-24T22:56:57.580433809Z   evaluation_interval: 15s
2024-04-24T22:56:57.580436713Z   scrape_interval: 15s
2024-04-24T22:56:57.580439676Z rule_files:
2024-04-24T22:56:57.580442674Z - osm_metric_rules.yml
2024-04-24T22:56:57.580445711Z - osm_alert_rules.yml
2024-04-24T22:56:57.580448619Z scrape_configs:
2024-04-24T22:56:57.580451833Z - job_name: mon_exporter
2024-04-24T22:56:57.580454677Z   static_configs:
2024-04-24T22:56:57.580457611Z   - targets:
2024-04-24T22:56:57.580460483Z     - mon:8000
2024-04-24T22:56:57.580463461Z - honor_labels: true
2024-04-24T22:56:57.580483081Z   job_name: pushgateway
2024-04-24T22:56:57.580486072Z   scrape_interval: 30s
2024-04-24T22:56:57.580489368Z   static_configs:
2024-04-24T22:56:57.580492710Z   - targets:
2024-04-24T22:56:57.580496163Z     - pushgateway-prometheus-pushgateway:9091
2024-04-24T22:56:57.580499102Z 
2024-04-24T22:56:57.585349026Z Alerts read: 2
2024-04-24T22:56:57.587952667Z Writing new alerts file to /etc/prometheus/osm_alert_rules.yml
2024-04-24T22:56:57.592091544Z groups:
2024-04-24T22:56:57.592133869Z - name: osm_alert_rules
2024-04-24T22:56:57.592147701Z   rules:
2024-04-24T22:56:57.592160564Z   - alert: vdu_down
2024-04-24T22:56:57.592172788Z     annotations:
2024-04-24T22:56:57.592186226Z       description: VDU {{ $labels.vm_id }} in VIM {{ $labels.vim_id }} has been down
2024-04-24T22:56:57.592199312Z         for more than 3 minutes. NS instance id is {{ $labels.ns_id }}
2024-04-24T22:56:57.592211205Z       summary: VDU {{ $labels.vm_id }} in VIM {{ $labels.vim_id }} is down
2024-04-24T22:56:57.592230503Z     expr: vm_status_extended != 1
2024-04-24T22:56:57.592242909Z     for: 3m
2024-04-24T22:56:57.592256581Z - name: _osm_alert_rules_20240424225657_
2024-04-24T22:56:57.592266152Z   rules:
2024-04-24T22:56:57.592275252Z   - alert: scalein_020a4461-85bf-43f1-a9a0-0d01637430c8
2024-04-24T22:56:57.592283562Z     expr: (count (osm_cpu_utilization{ns_id="c77aaa1c-9a42-43a8-ad41-87e9d004c6a5",
2024-04-24T22:56:57.592293056Z       vnf_member_index="vnf", vdu_id="hackfest_basic_metrics-VM"}) > 1) and (avg(osm_cpu_utilization{ns_id="c77aaa1c-9a42-43a8-ad41-87e9d004c6a5",
2024-04-24T22:56:57.592304054Z       vnf_member_index="vnf", vdu_id="hackfest_basic_metrics-VM"}) < 10.0000000000)
2024-04-24T22:56:57.592312474Z     for: 1m
2024-04-24T22:56:57.592320741Z     labels:
2024-04-24T22:56:57.592329040Z       ns_id: c77aaa1c-9a42-43a8-ad41-87e9d004c6a5
2024-04-24T22:56:57.592337501Z       vdu_id: hackfest_basic_metrics-VM
2024-04-24T22:56:57.592346513Z       vnf_member_index: vnf
2024-04-24T22:56:57.592354796Z   - alert: scaleout_0c6979a5-28b8-4070-8716-3869420fb33f
2024-04-24T22:56:57.592363504Z     expr: (count (osm_cpu_utilization{ns_id="c77aaa1c-9a42-43a8-ad41-87e9d004c6a5",
2024-04-24T22:56:57.592371900Z       vnf_member_index="vnf", vdu_id="hackfest_basic_metrics-VM"}) < 2) and (avg(osm_cpu_utilization{ns_id="c77aaa1c-9a42-43a8-ad41-87e9d004c6a5",
2024-04-24T22:56:57.592380388Z       vnf_member_index="vnf", vdu_id="hackfest_basic_metrics-VM"}) > 60.0000000000)
2024-04-24T22:56:57.592388696Z     for: 1m
2024-04-24T22:56:57.592396939Z     labels:
2024-04-24T22:56:57.592405115Z       ns_id: c77aaa1c-9a42-43a8-ad41-87e9d004c6a5
2024-04-24T22:56:57.592413531Z       vdu_id: hackfest_basic_metrics-VM
2024-04-24T22:56:57.592421845Z       vnf_member_index: vnf
2024-04-24T22:56:57.592429671Z 
2024-04-24T22:56:57.596172016Z New config written, updating prometheus
2024-04-24T22:57:02.633036391Z Prometheus config update successful
2024-04-24T22:57:02.633057281Z 
2024-04-24T22:57:02.633090773Z Changes detected, updating prometheus config
2024-04-24T22:57:02.639871846Z Jobs detected: 0
2024-04-24T22:57:02.644657151Z Writing new config file to /etc/prometheus/prometheus.yml
2024-04-24T22:57:02.647783150Z alerting:
2024-04-24T22:57:02.647799036Z   alertmanagers:
2024-04-24T22:57:02.647802958Z   - static_configs:
2024-04-24T22:57:02.647806136Z     - targets:
2024-04-24T22:57:02.647809796Z       - alertmanager:9093
2024-04-24T22:57:02.647813043Z global:
2024-04-24T22:57:02.647816770Z   evaluation_interval: 15s
2024-04-24T22:57:02.647819928Z   scrape_interval: 15s
2024-04-24T22:57:02.647840126Z rule_files:
2024-04-24T22:57:02.647843225Z - osm_metric_rules.yml
2024-04-24T22:57:02.647846215Z - osm_alert_rules.yml
2024-04-24T22:57:02.647849438Z scrape_configs:
2024-04-24T22:57:02.647852408Z - job_name: mon_exporter
2024-04-24T22:57:02.647855477Z   static_configs:
2024-04-24T22:57:02.647858461Z   - targets:
2024-04-24T22:57:02.647861474Z     - mon:8000
2024-04-24T22:57:02.647864591Z - honor_labels: true
2024-04-24T22:57:02.647867503Z   job_name: pushgateway
2024-04-24T22:57:02.647870397Z   scrape_interval: 30s
2024-04-24T22:57:02.647873365Z   static_configs:
2024-04-24T22:57:02.647876346Z   - targets:
2024-04-24T22:57:02.647893751Z     - pushgateway-prometheus-pushgateway:9091
2024-04-24T22:57:02.647898345Z 
2024-04-24T22:57:02.654480294Z Alerts read: 2
2024-04-24T22:57:02.656244579Z Writing new alerts file to /etc/prometheus/osm_alert_rules.yml
2024-04-24T22:57:02.664433496Z groups:
2024-04-24T22:57:02.664459254Z - name: osm_alert_rules
2024-04-24T22:57:02.664463038Z   rules:
2024-04-24T22:57:02.664466383Z   - alert: vdu_down
2024-04-24T22:57:02.664469485Z     annotations:
2024-04-24T22:57:02.664475861Z       description: VDU {{ $labels.vm_id }} in VIM {{ $labels.vim_id }} has been down
2024-04-24T22:57:02.664479575Z         for more than 3 minutes. NS instance id is {{ $labels.ns_id }}
2024-04-24T22:57:02.664482629Z       summary: VDU {{ $labels.vm_id }} in VIM {{ $labels.vim_id }} is down
2024-04-24T22:57:02.664486667Z     expr: vm_status_extended != 1
2024-04-24T22:57:02.664489960Z     for: 3m
2024-04-24T22:57:02.664493381Z - name: _osm_alert_rules_20240424225702_
2024-04-24T22:57:02.664496519Z   rules:
2024-04-24T22:57:02.664500420Z   - alert: scalein_020a4461-85bf-43f1-a9a0-0d01637430c8
2024-04-24T22:57:02.664503520Z     expr: (count (osm_cpu_utilization{ns_id="c77aaa1c-9a42-43a8-ad41-87e9d004c6a5",
2024-04-24T22:57:02.664508788Z       vnf_member_index="vnf", vdu_id="hackfest_basic_metrics-VM"}) > 1) and (avg(osm_cpu_utilization{ns_id="c77aaa1c-9a42-43a8-ad41-87e9d004c6a5",
2024-04-24T22:57:02.664512068Z       vnf_member_index="vnf", vdu_id="hackfest_basic_metrics-VM"}) < 10.0000000000)
2024-04-24T22:57:02.664515210Z     for: 1m
2024-04-24T22:57:02.664518258Z     labels:
2024-04-24T22:57:02.664521381Z       ns_id: c77aaa1c-9a42-43a8-ad41-87e9d004c6a5
2024-04-24T22:57:02.664524455Z       vdu_id: hackfest_basic_metrics-VM
2024-04-24T22:57:02.664528013Z       vnf_member_index: vnf
2024-04-24T22:57:02.664531316Z   - alert: scaleout_0c6979a5-28b8-4070-8716-3869420fb33f
2024-04-24T22:57:02.664534250Z     expr: (count (osm_cpu_utilization{ns_id="c77aaa1c-9a42-43a8-ad41-87e9d004c6a5",
2024-04-24T22:57:02.664537334Z       vnf_member_index="vnf", vdu_id="hackfest_basic_metrics-VM"}) < 2) and (avg(osm_cpu_utilization{ns_id="c77aaa1c-9a42-43a8-ad41-87e9d004c6a5",
2024-04-24T22:57:02.664540402Z       vnf_member_index="vnf", vdu_id="hackfest_basic_metrics-VM"}) > 60.0000000000)
2024-04-24T22:57:02.664543433Z     for: 1m
2024-04-24T22:57:02.664546423Z     labels:
2024-04-24T22:57:02.664549436Z       ns_id: c77aaa1c-9a42-43a8-ad41-87e9d004c6a5
2024-04-24T22:57:02.664552822Z       vdu_id: hackfest_basic_metrics-VM
2024-04-24T22:57:02.664555826Z       vnf_member_index: vnf
2024-04-24T22:57:02.664558638Z 
2024-04-24T22:57:02.667487124Z New config written, updating prometheus
2024-04-24T22:57:07.700995799Z Prometheus config update successful
2024-04-24T22:57:07.701135677Z 
2024-04-24T22:58:26.515591065Z Changes detected, updating prometheus config
2024-04-24T22:58:26.518057126Z Jobs detected: 0
2024-04-24T22:58:26.530682872Z Writing new config file to /etc/prometheus/prometheus.yml
2024-04-24T22:58:26.535938083Z alerting:
2024-04-24T22:58:26.535974288Z   alertmanagers:
2024-04-24T22:58:26.535983976Z   - static_configs:
2024-04-24T22:58:26.536026235Z     - targets:
2024-04-24T22:58:26.536037191Z       - alertmanager:9093
2024-04-24T22:58:26.536045081Z global:
2024-04-24T22:58:26.536052445Z   evaluation_interval: 15s
2024-04-24T22:58:26.536060223Z   scrape_interval: 15s
2024-04-24T22:58:26.536067794Z rule_files:
2024-04-24T22:58:26.536075598Z - osm_metric_rules.yml
2024-04-24T22:58:26.536119009Z - osm_alert_rules.yml
2024-04-24T22:58:26.536127439Z scrape_configs:
2024-04-24T22:58:26.536134753Z - job_name: mon_exporter
2024-04-24T22:58:26.536142019Z   static_configs:
2024-04-24T22:58:26.536149611Z   - targets:
2024-04-24T22:58:26.536157059Z     - mon:8000
2024-04-24T22:58:26.536164688Z - honor_labels: true
2024-04-24T22:58:26.536172076Z   job_name: pushgateway
2024-04-24T22:58:26.536204708Z   scrape_interval: 30s
2024-04-24T22:58:26.536212670Z   static_configs:
2024-04-24T22:58:26.536220075Z   - targets:
2024-04-24T22:58:26.536244441Z     - pushgateway-prometheus-pushgateway:9091
2024-04-24T22:58:26.536252024Z 
2024-04-24T22:58:26.540872204Z Alerts read: 0
2024-04-24T22:58:26.543411190Z Writing new alerts file to /etc/prometheus/osm_alert_rules.yml
2024-04-24T22:58:26.544824550Z groups:
2024-04-24T22:58:26.544840173Z - name: osm_alert_rules
2024-04-24T22:58:26.544845109Z   rules:
2024-04-24T22:58:26.544848695Z   - alert: vdu_down
2024-04-24T22:58:26.544852445Z     annotations:
2024-04-24T22:58:26.544856691Z       description: VDU {{ $labels.vm_id }} in VIM {{ $labels.vim_id }} has been down
2024-04-24T22:58:26.544860944Z         for more than 3 minutes. NS instance id is {{ $labels.ns_id }}
2024-04-24T22:58:26.544864463Z       summary: VDU {{ $labels.vm_id }} in VIM {{ $labels.vim_id }} is down
2024-04-24T22:58:26.544870156Z     expr: vm_status_extended != 1
2024-04-24T22:58:26.544876501Z     for: 3m
2024-04-24T22:58:26.544882259Z 
2024-04-24T22:58:26.546218239Z New config written, updating prometheus
2024-04-24T22:58:31.579693495Z Prometheus config update successful
2024-04-24T22:58:31.579739600Z 
2024-04-24T22:58:31.579745551Z Changes detected, updating prometheus config
2024-04-24T22:58:31.582954436Z Jobs detected: 0
2024-04-24T22:58:31.594624883Z Writing new config file to /etc/prometheus/prometheus.yml
2024-04-24T22:58:31.598309424Z alerting:
2024-04-24T22:58:31.598338855Z   alertmanagers:
2024-04-24T22:58:31.598349539Z   - static_configs:
2024-04-24T22:58:31.598358502Z     - targets:
2024-04-24T22:58:31.598369583Z       - alertmanager:9093
2024-04-24T22:58:31.598378957Z global:
2024-04-24T22:58:31.598387495Z   evaluation_interval: 15s
2024-04-24T22:58:31.598396013Z   scrape_interval: 15s
2024-04-24T22:58:31.598404550Z rule_files:
2024-04-24T22:58:31.598413416Z - osm_metric_rules.yml
2024-04-24T22:58:31.598421791Z - osm_alert_rules.yml
2024-04-24T22:58:31.598430356Z scrape_configs:
2024-04-24T22:58:31.598438819Z - job_name: mon_exporter
2024-04-24T22:58:31.598446943Z   static_configs:
2024-04-24T22:58:31.598455300Z   - targets:
2024-04-24T22:58:31.598464901Z     - mon:8000
2024-04-24T22:58:31.598473331Z - honor_labels: true
2024-04-24T22:58:31.598481863Z   job_name: pushgateway
2024-04-24T22:58:31.598490153Z   scrape_interval: 30s
2024-04-24T22:58:31.598498483Z   static_configs:
2024-04-24T22:58:31.598507174Z   - targets:
2024-04-24T22:58:31.598516908Z     - pushgateway-prometheus-pushgateway:9091
2024-04-24T22:58:31.598563605Z 
2024-04-24T22:58:31.603088753Z Alerts read: 0
2024-04-24T22:58:31.605864238Z Writing new alerts file to /etc/prometheus/osm_alert_rules.yml
2024-04-24T22:58:31.607457602Z groups:
2024-04-24T22:58:31.607538725Z - name: osm_alert_rules
2024-04-24T22:58:31.607553423Z   rules:
2024-04-24T22:58:31.607566762Z   - alert: vdu_down
2024-04-24T22:58:31.607580019Z     annotations:
2024-04-24T22:58:31.607594384Z       description: VDU {{ $labels.vm_id }} in VIM {{ $labels.vim_id }} has been down
2024-04-24T22:58:31.607607103Z         for more than 3 minutes. NS instance id is {{ $labels.ns_id }}
2024-04-24T22:58:31.607618441Z       summary: VDU {{ $labels.vm_id }} in VIM {{ $labels.vim_id }} is down
2024-04-24T22:58:31.607631288Z     expr: vm_status_extended != 1
2024-04-24T22:58:31.607643125Z     for: 3m
2024-04-24T22:58:31.607652520Z 
2024-04-24T22:58:31.608509266Z New config written, updating prometheus
2024-04-24T22:58:36.642879474Z Prometheus config update successful
2024-04-24T22:58:36.642919237Z 
2024-04-24T23:18:53.370530476Z Changes detected, updating prometheus config
2024-04-24T23:18:53.374210766Z Jobs detected: 0
2024-04-24T23:18:53.379221011Z Writing new config file to /etc/prometheus/prometheus.yml
2024-04-24T23:18:53.381099165Z alerting:
2024-04-24T23:18:53.381110508Z   alertmanagers:
2024-04-24T23:18:53.381114844Z   - static_configs:
2024-04-24T23:18:53.381117869Z     - targets:
2024-04-24T23:18:53.381123617Z       - alertmanager:9093
2024-04-24T23:18:53.381126624Z global:
2024-04-24T23:18:53.381129669Z   evaluation_interval: 15s
2024-04-24T23:18:53.381132630Z   scrape_interval: 15s
2024-04-24T23:18:53.381135767Z rule_files:
2024-04-24T23:18:53.381138956Z - osm_metric_rules.yml
2024-04-24T23:18:53.381141994Z - osm_alert_rules.yml
2024-04-24T23:18:53.381145031Z scrape_configs:
2024-04-24T23:18:53.381154638Z - job_name: mon_exporter
2024-04-24T23:18:53.381158015Z   static_configs:
2024-04-24T23:18:53.381161200Z   - targets:
2024-04-24T23:18:53.381164240Z     - mon:8000
2024-04-24T23:18:53.381167375Z - honor_labels: true
2024-04-24T23:18:53.381170229Z   job_name: pushgateway
2024-04-24T23:18:53.381173146Z   scrape_interval: 30s
2024-04-24T23:18:53.381176073Z   static_configs:
2024-04-24T23:18:53.381179054Z   - targets:
2024-04-24T23:18:53.381182720Z     - pushgateway-prometheus-pushgateway:9091
2024-04-24T23:18:53.381185657Z 
2024-04-24T23:18:53.387979393Z Alerts read: 6
2024-04-24T23:18:53.389923294Z Writing new alerts file to /etc/prometheus/osm_alert_rules.yml
2024-04-24T23:18:53.416095088Z groups:
2024-04-24T23:18:53.416131134Z - name: osm_alert_rules
2024-04-24T23:18:53.416134806Z   rules:
2024-04-24T23:18:53.416138149Z   - alert: vdu_down
2024-04-24T23:18:53.416141338Z     annotations:
2024-04-24T23:18:53.416149546Z       description: VDU {{ $labels.vm_id }} in VIM {{ $labels.vim_id }} has been down
2024-04-24T23:18:53.416154788Z         for more than 3 minutes. NS instance id is {{ $labels.ns_id }}
2024-04-24T23:18:53.416157981Z       summary: VDU {{ $labels.vm_id }} in VIM {{ $labels.vim_id }} is down
2024-04-24T23:18:53.416162123Z     expr: vm_status_extended != 1
2024-04-24T23:18:53.416165581Z     for: 3m
2024-04-24T23:18:53.416168717Z - name: _osm_alert_rules_20240424231853_
2024-04-24T23:18:53.416171844Z   rules:
2024-04-24T23:18:53.416175951Z   - alert: scalein_1df09c7d-1d3f-4725-9b53-b496a20b91a2
2024-04-24T23:18:53.416179146Z     expr: (count (osm_cpu_utilization{ns_id="81dfdac3-2fc7-405b-861d-667a23425157",
2024-04-24T23:18:53.416190598Z       vnf_member_index="vnf", vdu_id="hackfest_basic_metrics-VM"}) > 3) and (avg(osm_cpu_utilization{ns_id="81dfdac3-2fc7-405b-861d-667a23425157",
2024-04-24T23:18:53.416209466Z       vnf_member_index="vnf", vdu_id="hackfest_basic_metrics-VM"}) < 10.0000000000)
2024-04-24T23:18:53.416212538Z     for: 1m
2024-04-24T23:18:53.416215577Z     labels:
2024-04-24T23:18:53.416218556Z       ns_id: 81dfdac3-2fc7-405b-861d-667a23425157
2024-04-24T23:18:53.416222354Z       vdu_id: hackfest_basic_metrics-VM
2024-04-24T23:18:53.416226241Z       vnf_member_index: vnf
2024-04-24T23:18:53.416229413Z   - alert: scaleout_7c59814d-8f05-4d26-9fd2-66ec30823798
2024-04-24T23:18:53.416232650Z     expr: (count (osm_cpu_utilization{ns_id="81dfdac3-2fc7-405b-861d-667a23425157",
2024-04-24T23:18:53.416235853Z       vnf_member_index="vnf", vdu_id="hackfest_basic_metrics-VM"}) < 5) and (avg(osm_cpu_utilization{ns_id="81dfdac3-2fc7-405b-861d-667a23425157",
2024-04-24T23:18:53.416238869Z       vnf_member_index="vnf", vdu_id="hackfest_basic_metrics-VM"}) > 60.0000000000)
2024-04-24T23:18:53.416241888Z     for: 1m
2024-04-24T23:18:53.416244785Z     labels:
2024-04-24T23:18:53.416247796Z       ns_id: 81dfdac3-2fc7-405b-861d-667a23425157
2024-04-24T23:18:53.416250821Z       vdu_id: hackfest_basic_metrics-VM
2024-04-24T23:18:53.416253930Z       vnf_member_index: vnf
2024-04-24T23:18:53.416257237Z   - alert: scalein_d2b585ff-71d3-4ae5-86db-267143351d7d
2024-04-24T23:18:53.416260200Z     expr: (count (osm_cpu_utilization{ns_id="81dfdac3-2fc7-405b-861d-667a23425157",
2024-04-24T23:18:53.416263583Z       vnf_member_index="vnf", vdu_id="hackfest_basic_metrics-VM"}) > 3) and (avg(osm_cpu_utilization{ns_id="81dfdac3-2fc7-405b-861d-667a23425157",
2024-04-24T23:18:53.416266602Z       vnf_member_index="vnf", vdu_id="hackfest_basic_metrics-VM"}) < 10.0000000000)
2024-04-24T23:18:53.416269508Z     for: 1m
2024-04-24T23:18:53.416272513Z     labels:
2024-04-24T23:18:53.416275564Z       ns_id: 81dfdac3-2fc7-405b-861d-667a23425157
2024-04-24T23:18:53.416278702Z       vdu_id: hackfest_basic_metrics-VM
2024-04-24T23:18:53.416281668Z       vnf_member_index: vnf
2024-04-24T23:18:53.416284633Z   - alert: scaleout_5ff47137-da8b-432a-b231-acd130a1992f
2024-04-24T23:18:53.416287710Z     expr: (count (osm_cpu_utilization{ns_id="81dfdac3-2fc7-405b-861d-667a23425157",
2024-04-24T23:18:53.416290882Z       vnf_member_index="vnf", vdu_id="hackfest_basic_metrics-VM"}) < 5) and (avg(osm_cpu_utilization{ns_id="81dfdac3-2fc7-405b-861d-667a23425157",
2024-04-24T23:18:53.416293871Z       vnf_member_index="vnf", vdu_id="hackfest_basic_metrics-VM"}) > 60.0000000000)
2024-04-24T23:18:53.416296790Z     for: 1m
2024-04-24T23:18:53.416299678Z     labels:
2024-04-24T23:18:53.416302907Z       ns_id: 81dfdac3-2fc7-405b-861d-667a23425157
2024-04-24T23:18:53.416305722Z       vdu_id: hackfest_basic_metrics-VM
2024-04-24T23:18:53.416308581Z       vnf_member_index: vnf
2024-04-24T23:18:53.416311642Z   - alert: scalein_063ed24b-fdef-4dd6-9df4-7dcbe473b819
2024-04-24T23:18:53.416314520Z     expr: (count (osm_cpu_utilization{ns_id="81dfdac3-2fc7-405b-861d-667a23425157",
2024-04-24T23:18:53.416317525Z       vnf_member_index="vnf", vdu_id="hackfest_basic_metrics-VM"}) > 3) and (avg(osm_cpu_utilization{ns_id="81dfdac3-2fc7-405b-861d-667a23425157",
2024-04-24T23:18:53.416320402Z       vnf_member_index="vnf", vdu_id="hackfest_basic_metrics-VM"}) < 10.0000000000)
2024-04-24T23:18:53.416323256Z     for: 1m
2024-04-24T23:18:53.416326276Z     labels:
2024-04-24T23:18:53.416329160Z       ns_id: 81dfdac3-2fc7-405b-861d-667a23425157
2024-04-24T23:18:53.416332154Z       vdu_id: hackfest_basic_metrics-VM
2024-04-24T23:18:53.416335118Z       vnf_member_index: vnf
2024-04-24T23:18:53.416343390Z   - alert: scaleout_effb3995-4125-4a6f-8d93-f6231f86a25b
2024-04-24T23:18:53.416346412Z     expr: (count (osm_cpu_utilization{ns_id="81dfdac3-2fc7-405b-861d-667a23425157",
2024-04-24T23:18:53.416349295Z       vnf_member_index="vnf", vdu_id="hackfest_basic_metrics-VM"}) < 5) and (avg(osm_cpu_utilization{ns_id="81dfdac3-2fc7-405b-861d-667a23425157",
2024-04-24T23:18:53.416352251Z       vnf_member_index="vnf", vdu_id="hackfest_basic_metrics-VM"}) > 60.0000000000)
2024-04-24T23:18:53.416355287Z     for: 1m
2024-04-24T23:18:53.416358233Z     labels:
2024-04-24T23:18:53.416361070Z       ns_id: 81dfdac3-2fc7-405b-861d-667a23425157
2024-04-24T23:18:53.416363991Z       vdu_id: hackfest_basic_metrics-VM
2024-04-24T23:18:53.416366877Z       vnf_member_index: vnf
2024-04-24T23:18:53.416369687Z 
2024-04-24T23:18:53.431411129Z New config written, updating prometheus
2024-04-24T23:18:58.470914074Z Prometheus config update successful
2024-04-24T23:18:58.470943052Z 
2024-04-24T23:18:58.470946771Z Changes detected, updating prometheus config
2024-04-24T23:18:58.472860499Z Jobs detected: 0
2024-04-24T23:18:58.477440975Z Writing new config file to /etc/prometheus/prometheus.yml
2024-04-24T23:18:58.480320820Z alerting:
2024-04-24T23:18:58.480336528Z   alertmanagers:
2024-04-24T23:18:58.480343974Z   - static_configs:
2024-04-24T23:18:58.480348205Z     - targets:
2024-04-24T23:18:58.480354134Z       - alertmanager:9093
2024-04-24T23:18:58.480358150Z global:
2024-04-24T23:18:58.480362387Z   evaluation_interval: 15s
2024-04-24T23:18:58.480366736Z   scrape_interval: 15s
2024-04-24T23:18:58.480370776Z rule_files:
2024-04-24T23:18:58.480374776Z - osm_metric_rules.yml
2024-04-24T23:18:58.480378772Z - osm_alert_rules.yml
2024-04-24T23:18:58.480382721Z scrape_configs:
2024-04-24T23:18:58.480386638Z - job_name: mon_exporter
2024-04-24T23:18:58.480390557Z   static_configs:
2024-04-24T23:18:58.480394524Z   - targets:
2024-04-24T23:18:58.480398621Z     - mon:8000
2024-04-24T23:18:58.480402598Z - honor_labels: true
2024-04-24T23:18:58.480406478Z   job_name: pushgateway
2024-04-24T23:18:58.480410394Z   scrape_interval: 30s
2024-04-24T23:18:58.480414279Z   static_configs:
2024-04-24T23:18:58.480418211Z   - targets:
2024-04-24T23:18:58.480423630Z     - pushgateway-prometheus-pushgateway:9091
2024-04-24T23:18:58.480427504Z 
2024-04-24T23:18:58.486481831Z Alerts read: 6
2024-04-24T23:18:58.488616361Z Writing new alerts file to /etc/prometheus/osm_alert_rules.yml
2024-04-24T23:18:58.496237618Z groups:
2024-04-24T23:18:58.496250965Z - name: osm_alert_rules
2024-04-24T23:18:58.496254340Z   rules:
2024-04-24T23:18:58.496257883Z   - alert: vdu_down
2024-04-24T23:18:58.496260855Z     annotations:
2024-04-24T23:18:58.496267143Z       description: VDU {{ $labels.vm_id }} in VIM {{ $labels.vim_id }} has been down
2024-04-24T23:18:58.496271499Z         for more than 3 minutes. NS instance id is {{ $labels.ns_id }}
2024-04-24T23:18:58.496274891Z       summary: VDU {{ $labels.vm_id }} in VIM {{ $labels.vim_id }} is down
2024-04-24T23:18:58.496278448Z     expr: vm_status_extended != 1
2024-04-24T23:18:58.496281372Z     for: 3m
2024-04-24T23:18:58.496284287Z - name: _osm_alert_rules_20240424231858_
2024-04-24T23:18:58.496287304Z   rules:
2024-04-24T23:18:58.496291942Z   - alert: scalein_1df09c7d-1d3f-4725-9b53-b496a20b91a2
2024-04-24T23:18:58.496294919Z     expr: (count (osm_cpu_utilization{ns_id="81dfdac3-2fc7-405b-861d-667a23425157",
2024-04-24T23:18:58.496299879Z       vnf_member_index="vnf", vdu_id="hackfest_basic_metrics-VM"}) > 3) and (avg(osm_cpu_utilization{ns_id="81dfdac3-2fc7-405b-861d-667a23425157",
2024-04-24T23:18:58.496316003Z       vnf_member_index="vnf", vdu_id="hackfest_basic_metrics-VM"}) < 10.0000000000)
2024-04-24T23:18:58.496319060Z     for: 1m
2024-04-24T23:18:58.496321933Z     labels:
2024-04-24T23:18:58.496325119Z       ns_id: 81dfdac3-2fc7-405b-861d-667a23425157
2024-04-24T23:18:58.496328093Z       vdu_id: hackfest_basic_metrics-VM
2024-04-24T23:18:58.496332540Z       vnf_member_index: vnf
2024-04-24T23:18:58.496335535Z   - alert: scaleout_7c59814d-8f05-4d26-9fd2-66ec30823798
2024-04-24T23:18:58.496338468Z     expr: (count (osm_cpu_utilization{ns_id="81dfdac3-2fc7-405b-861d-667a23425157",
2024-04-24T23:18:58.496341414Z       vnf_member_index="vnf", vdu_id="hackfest_basic_metrics-VM"}) < 5) and (avg(osm_cpu_utilization{ns_id="81dfdac3-2fc7-405b-861d-667a23425157",
2024-04-24T23:18:58.496344312Z       vnf_member_index="vnf", vdu_id="hackfest_basic_metrics-VM"}) > 60.0000000000)
2024-04-24T23:18:58.496347203Z     for: 1m
2024-04-24T23:18:58.496350023Z     labels:
2024-04-24T23:18:58.496352932Z       ns_id: 81dfdac3-2fc7-405b-861d-667a23425157
2024-04-24T23:18:58.496356096Z       vdu_id: hackfest_basic_metrics-VM
2024-04-24T23:18:58.496359006Z       vnf_member_index: vnf
2024-04-24T23:18:58.496361954Z   - alert: scalein_d2b585ff-71d3-4ae5-86db-267143351d7d
2024-04-24T23:18:58.496364820Z     expr: (count (osm_cpu_utilization{ns_id="81dfdac3-2fc7-405b-861d-667a23425157",
2024-04-24T23:18:58.496367859Z       vnf_member_index="vnf", vdu_id="hackfest_basic_metrics-VM"}) > 3) and (avg(osm_cpu_utilization{ns_id="81dfdac3-2fc7-405b-861d-667a23425157",
2024-04-24T23:18:58.496370736Z       vnf_member_index="vnf", vdu_id="hackfest_basic_metrics-VM"}) < 10.0000000000)
2024-04-24T23:18:58.496373608Z     for: 1m
2024-04-24T23:18:58.496376524Z     labels:
2024-04-24T23:18:58.496379392Z       ns_id: 81dfdac3-2fc7-405b-861d-667a23425157
2024-04-24T23:18:58.496382280Z       vdu_id: hackfest_basic_metrics-VM
2024-04-24T23:18:58.496385163Z       vnf_member_index: vnf
2024-04-24T23:18:58.496388026Z   - alert: scaleout_5ff47137-da8b-432a-b231-acd130a1992f
2024-04-24T23:18:58.496390890Z     expr: (count (osm_cpu_utilization{ns_id="81dfdac3-2fc7-405b-861d-667a23425157",
2024-04-24T23:18:58.496394134Z       vnf_member_index="vnf", vdu_id="hackfest_basic_metrics-VM"}) < 5) and (avg(osm_cpu_utilization{ns_id="81dfdac3-2fc7-405b-861d-667a23425157",
2024-04-24T23:18:58.496396995Z       vnf_member_index="vnf", vdu_id="hackfest_basic_metrics-VM"}) > 60.0000000000)
2024-04-24T23:18:58.496399834Z     for: 1m
2024-04-24T23:18:58.496402649Z     labels:
2024-04-24T23:18:58.496405686Z       ns_id: 81dfdac3-2fc7-405b-861d-667a23425157
2024-04-24T23:18:58.496408515Z       vdu_id: hackfest_basic_metrics-VM
2024-04-24T23:18:58.496411395Z       vnf_member_index: vnf
2024-04-24T23:18:58.496414377Z   - alert: scalein_063ed24b-fdef-4dd6-9df4-7dcbe473b819
2024-04-24T23:18:58.496417162Z     expr: (count (osm_cpu_utilization{ns_id="81dfdac3-2fc7-405b-861d-667a23425157",
2024-04-24T23:18:58.496420271Z       vnf_member_index="vnf", vdu_id="hackfest_basic_metrics-VM"}) > 3) and (avg(osm_cpu_utilization{ns_id="81dfdac3-2fc7-405b-861d-667a23425157",
2024-04-24T23:18:58.496423228Z       vnf_member_index="vnf", vdu_id="hackfest_basic_metrics-VM"}) < 10.0000000000)
2024-04-24T23:18:58.496426076Z     for: 1m
2024-04-24T23:18:58.496428892Z     labels:
2024-04-24T23:18:58.496431752Z       ns_id: 81dfdac3-2fc7-405b-861d-667a23425157
2024-04-24T23:18:58.496434632Z       vdu_id: hackfest_basic_metrics-VM
2024-04-24T23:18:58.496449978Z       vnf_member_index: vnf
2024-04-24T23:18:58.496453080Z   - alert: scaleout_effb3995-4125-4a6f-8d93-f6231f86a25b
2024-04-24T23:18:58.496459998Z     expr: (count (osm_cpu_utilization{ns_id="81dfdac3-2fc7-405b-861d-667a23425157",
2024-04-24T23:18:58.496462963Z       vnf_member_index="vnf", vdu_id="hackfest_basic_metrics-VM"}) < 5) and (avg(osm_cpu_utilization{ns_id="81dfdac3-2fc7-405b-861d-667a23425157",
2024-04-24T23:18:58.496465782Z       vnf_member_index="vnf", vdu_id="hackfest_basic_metrics-VM"}) > 60.0000000000)
2024-04-24T23:18:58.496468586Z     for: 1m
2024-04-24T23:18:58.496471422Z     labels:
2024-04-24T23:18:58.496474306Z       ns_id: 81dfdac3-2fc7-405b-861d-667a23425157
2024-04-24T23:18:58.496477194Z       vdu_id: hackfest_basic_metrics-VM
2024-04-24T23:18:58.496480074Z       vnf_member_index: vnf
2024-04-24T23:18:58.496482794Z 
2024-04-24T23:18:58.503766430Z New config written, updating prometheus
2024-04-24T23:19:03.534142094Z Prometheus config update successful
2024-04-24T23:19:03.534191672Z 
2024-04-24T23:19:03.534241105Z Changes detected, updating prometheus config
2024-04-24T23:19:03.535839955Z Jobs detected: 0
2024-04-24T23:19:03.545820257Z Writing new config file to /etc/prometheus/prometheus.yml
2024-04-24T23:19:03.546752260Z alerting:
2024-04-24T23:19:03.546759792Z   alertmanagers:
2024-04-24T23:19:03.546764384Z   - static_configs:
2024-04-24T23:19:03.546767518Z     - targets:
2024-04-24T23:19:03.546771927Z       - alertmanager:9093
2024-04-24T23:19:03.546774982Z global:
2024-04-24T23:19:03.546778012Z   evaluation_interval: 15s
2024-04-24T23:19:03.546781024Z   scrape_interval: 15s
2024-04-24T23:19:03.546783994Z rule_files:
2024-04-24T23:19:03.546787163Z - osm_metric_rules.yml
2024-04-24T23:19:03.546790133Z - osm_alert_rules.yml
2024-04-24T23:19:03.546793018Z scrape_configs:
2024-04-24T23:19:03.546796105Z - job_name: mon_exporter
2024-04-24T23:19:03.546799151Z   static_configs:
2024-04-24T23:19:03.546802070Z   - targets:
2024-04-24T23:19:03.546805126Z     - mon:8000
2024-04-24T23:19:03.546808057Z - honor_labels: true
2024-04-24T23:19:03.546811002Z   job_name: pushgateway
2024-04-24T23:19:03.546813917Z   scrape_interval: 30s
2024-04-24T23:19:03.546816849Z   static_configs:
2024-04-24T23:19:03.546820118Z   - targets:
2024-04-24T23:19:03.546824449Z     - pushgateway-prometheus-pushgateway:9091
2024-04-24T23:19:03.546827314Z 
2024-04-24T23:19:03.556246950Z Alerts read: 6
2024-04-24T23:19:03.560734214Z Writing new alerts file to /etc/prometheus/osm_alert_rules.yml
2024-04-24T23:19:03.572647591Z groups:
2024-04-24T23:19:03.572672050Z - name: osm_alert_rules
2024-04-24T23:19:03.572676854Z   rules:
2024-04-24T23:19:03.572681495Z   - alert: vdu_down
2024-04-24T23:19:03.572685601Z     annotations:
2024-04-24T23:19:03.572692737Z       description: VDU {{ $labels.vm_id }} in VIM {{ $labels.vim_id }} has been down
2024-04-24T23:19:03.572698256Z         for more than 3 minutes. NS instance id is {{ $labels.ns_id }}
2024-04-24T23:19:03.572702502Z       summary: VDU {{ $labels.vm_id }} in VIM {{ $labels.vim_id }} is down
2024-04-24T23:19:03.572707362Z     expr: vm_status_extended != 1
2024-04-24T23:19:03.572711502Z     for: 3m
2024-04-24T23:19:03.572715619Z - name: _osm_alert_rules_20240424231903_
2024-04-24T23:19:03.572719616Z   rules:
2024-04-24T23:19:03.572725041Z   - alert: scalein_1df09c7d-1d3f-4725-9b53-b496a20b91a2
2024-04-24T23:19:03.572729479Z     expr: (count (osm_cpu_utilization{ns_id="81dfdac3-2fc7-405b-861d-667a23425157",
2024-04-24T23:19:03.572737234Z       vnf_member_index="vnf", vdu_id="hackfest_basic_metrics-VM"}) > 3) and (avg(osm_cpu_utilization{ns_id="81dfdac3-2fc7-405b-861d-667a23425157",
2024-04-24T23:19:03.572741624Z       vnf_member_index="vnf", vdu_id="hackfest_basic_metrics-VM"}) < 10.0000000000)
2024-04-24T23:19:03.572760295Z     for: 1m
2024-04-24T23:19:03.572764479Z     labels:
2024-04-24T23:19:03.572768996Z       ns_id: 81dfdac3-2fc7-405b-861d-667a23425157
2024-04-24T23:19:03.572773003Z       vdu_id: hackfest_basic_metrics-VM
2024-04-24T23:19:03.572777638Z       vnf_member_index: vnf
2024-04-24T23:19:03.572781604Z   - alert: scaleout_7c59814d-8f05-4d26-9fd2-66ec30823798
2024-04-24T23:19:03.572785589Z     expr: (count (osm_cpu_utilization{ns_id="81dfdac3-2fc7-405b-861d-667a23425157",
2024-04-24T23:19:03.572789674Z       vnf_member_index="vnf", vdu_id="hackfest_basic_metrics-VM"}) < 5) and (avg(osm_cpu_utilization{ns_id="81dfdac3-2fc7-405b-861d-667a23425157",
2024-04-24T23:19:03.572793851Z       vnf_member_index="vnf", vdu_id="hackfest_basic_metrics-VM"}) > 60.0000000000)
2024-04-24T23:19:03.572797838Z     for: 1m
2024-04-24T23:19:03.572801810Z     labels:
2024-04-24T23:19:03.572805778Z       ns_id: 81dfdac3-2fc7-405b-861d-667a23425157
2024-04-24T23:19:03.572809651Z       vdu_id: hackfest_basic_metrics-VM
2024-04-24T23:19:03.572813655Z       vnf_member_index: vnf
2024-04-24T23:19:03.572817646Z   - alert: scalein_d2b585ff-71d3-4ae5-86db-267143351d7d
2024-04-24T23:19:03.572821571Z     expr: (count (osm_cpu_utilization{ns_id="81dfdac3-2fc7-405b-861d-667a23425157",
2024-04-24T23:19:03.572825759Z       vnf_member_index="vnf", vdu_id="hackfest_basic_metrics-VM"}) > 3) and (avg(osm_cpu_utilization{ns_id="81dfdac3-2fc7-405b-861d-667a23425157",
2024-04-24T23:19:03.572829721Z       vnf_member_index="vnf", vdu_id="hackfest_basic_metrics-VM"}) < 10.0000000000)
2024-04-24T23:19:03.572833754Z     for: 1m
2024-04-24T23:19:03.572837689Z     labels:
2024-04-24T23:19:03.572841694Z       ns_id: 81dfdac3-2fc7-405b-861d-667a23425157
2024-04-24T23:19:03.572845617Z       vdu_id: hackfest_basic_metrics-VM
2024-04-24T23:19:03.572849606Z       vnf_member_index: vnf
2024-04-24T23:19:03.572853716Z   - alert: scaleout_5ff47137-da8b-432a-b231-acd130a1992f
2024-04-24T23:19:03.572857908Z     expr: (count (osm_cpu_utilization{ns_id="81dfdac3-2fc7-405b-861d-667a23425157",
2024-04-24T23:19:03.572861934Z       vnf_member_index="vnf", vdu_id="hackfest_basic_metrics-VM"}) < 5) and (avg(osm_cpu_utilization{ns_id="81dfdac3-2fc7-405b-861d-667a23425157",
2024-04-24T23:19:03.572865894Z       vnf_member_index="vnf", vdu_id="hackfest_basic_metrics-VM"}) > 60.0000000000)
2024-04-24T23:19:03.572869830Z     for: 1m
2024-04-24T23:19:03.572873767Z     labels:
2024-04-24T23:19:03.572877682Z       ns_id: 81dfdac3-2fc7-405b-861d-667a23425157
2024-04-24T23:19:03.572881618Z       vdu_id: hackfest_basic_metrics-VM
2024-04-24T23:19:03.572885528Z       vnf_member_index: vnf
2024-04-24T23:19:03.572889628Z   - alert: scalein_063ed24b-fdef-4dd6-9df4-7dcbe473b819
2024-04-24T23:19:03.572893552Z     expr: (count (osm_cpu_utilization{ns_id="81dfdac3-2fc7-405b-861d-667a23425157",
2024-04-24T23:19:03.572897557Z       vnf_member_index="vnf", vdu_id="hackfest_basic_metrics-VM"}) > 3) and (avg(osm_cpu_utilization{ns_id="81dfdac3-2fc7-405b-861d-667a23425157",
2024-04-24T23:19:03.572901491Z       vnf_member_index="vnf", vdu_id="hackfest_basic_metrics-VM"}) < 10.0000000000)
2024-04-24T23:19:03.572905525Z     for: 1m
2024-04-24T23:19:03.572909483Z     labels:
2024-04-24T23:19:03.572913420Z       ns_id: 81dfdac3-2fc7-405b-861d-667a23425157
2024-04-24T23:19:03.572917370Z       vdu_id: hackfest_basic_metrics-VM
2024-04-24T23:19:03.572921481Z       vnf_member_index: vnf
2024-04-24T23:19:03.572925435Z   - alert: scaleout_effb3995-4125-4a6f-8d93-f6231f86a25b
2024-04-24T23:19:03.572929402Z     expr: (count (osm_cpu_utilization{ns_id="81dfdac3-2fc7-405b-861d-667a23425157",
2024-04-24T23:19:03.572938235Z       vnf_member_index="vnf", vdu_id="hackfest_basic_metrics-VM"}) < 5) and (avg(osm_cpu_utilization{ns_id="81dfdac3-2fc7-405b-861d-667a23425157",
2024-04-24T23:19:03.572942407Z       vnf_member_index="vnf", vdu_id="hackfest_basic_metrics-VM"}) > 60.0000000000)
2024-04-24T23:19:03.572946319Z     for: 1m
2024-04-24T23:19:03.572950254Z     labels:
2024-04-24T23:19:03.572954153Z       ns_id: 81dfdac3-2fc7-405b-861d-667a23425157
2024-04-24T23:19:03.572958201Z       vdu_id: hackfest_basic_metrics-VM
2024-04-24T23:19:03.572962188Z       vnf_member_index: vnf
2024-04-24T23:19:03.572965958Z 
2024-04-24T23:19:03.583733539Z New config written, updating prometheus
2024-04-24T23:19:08.624212307Z Prometheus config update successful
2024-04-24T23:19:08.624263374Z 
2024-04-24T23:19:08.624288723Z Changes detected, updating prometheus config
2024-04-24T23:19:08.626079538Z Jobs detected: 0
2024-04-24T23:19:08.633107007Z Writing new config file to /etc/prometheus/prometheus.yml
2024-04-24T23:19:08.635456571Z alerting:
2024-04-24T23:19:08.635471497Z   alertmanagers:
2024-04-24T23:19:08.635476671Z   - static_configs:
2024-04-24T23:19:08.635480858Z     - targets:
2024-04-24T23:19:08.635485962Z       - alertmanager:9093
2024-04-24T23:19:08.635490071Z global:
2024-04-24T23:19:08.635494198Z   evaluation_interval: 15s
2024-04-24T23:19:08.635498364Z   scrape_interval: 15s
2024-04-24T23:19:08.635502800Z rule_files:
2024-04-24T23:19:08.635507010Z - osm_metric_rules.yml
2024-04-24T23:19:08.635511095Z - osm_alert_rules.yml
2024-04-24T23:19:08.635515148Z scrape_configs:
2024-04-24T23:19:08.635519541Z - job_name: mon_exporter
2024-04-24T23:19:08.635523669Z   static_configs:
2024-04-24T23:19:08.635527639Z   - targets:
2024-04-24T23:19:08.635531786Z     - mon:8000
2024-04-24T23:19:08.635535777Z - honor_labels: true
2024-04-24T23:19:08.635539865Z   job_name: pushgateway
2024-04-24T23:19:08.635543819Z   scrape_interval: 30s
2024-04-24T23:19:08.635547881Z   static_configs:
2024-04-24T23:19:08.635556289Z   - targets:
2024-04-24T23:19:08.635561159Z     - pushgateway-prometheus-pushgateway:9091
2024-04-24T23:19:08.635565180Z 
2024-04-24T23:19:08.639809786Z Alerts read: 6
2024-04-24T23:19:08.641695259Z Writing new alerts file to /etc/prometheus/osm_alert_rules.yml
2024-04-24T23:19:08.649477335Z groups:
2024-04-24T23:19:08.649491800Z - name: osm_alert_rules
2024-04-24T23:19:08.649495210Z   rules:
2024-04-24T23:19:08.649498152Z   - alert: vdu_down
2024-04-24T23:19:08.649501168Z     annotations:
2024-04-24T23:19:08.649504898Z       description: VDU {{ $labels.vm_id }} in VIM {{ $labels.vim_id }} has been down
2024-04-24T23:19:08.649508805Z         for more than 3 minutes. NS instance id is {{ $labels.ns_id }}
2024-04-24T23:19:08.649511833Z       summary: VDU {{ $labels.vm_id }} in VIM {{ $labels.vim_id }} is down
2024-04-24T23:19:08.649515268Z     expr: vm_status_extended != 1
2024-04-24T23:19:08.649518193Z     for: 3m
2024-04-24T23:19:08.649521126Z - name: _osm_alert_rules_20240424231908_
2024-04-24T23:19:08.649523993Z   rules:
2024-04-24T23:19:08.649527739Z   - alert: scalein_1df09c7d-1d3f-4725-9b53-b496a20b91a2
2024-04-24T23:19:08.649530723Z     expr: (count (osm_cpu_utilization{ns_id="81dfdac3-2fc7-405b-861d-667a23425157",
2024-04-24T23:19:08.649536274Z       vnf_member_index="vnf", vdu_id="hackfest_basic_metrics-VM"}) > 3) and (avg(osm_cpu_utilization{ns_id="81dfdac3-2fc7-405b-861d-667a23425157",
2024-04-24T23:19:08.649539302Z       vnf_member_index="vnf", vdu_id="hackfest_basic_metrics-VM"}) < 10.0000000000)
2024-04-24T23:19:08.649574232Z     for: 1m
2024-04-24T23:19:08.649579592Z     labels:
2024-04-24T23:19:08.649584474Z       ns_id: 81dfdac3-2fc7-405b-861d-667a23425157
2024-04-24T23:19:08.649588740Z       vdu_id: hackfest_basic_metrics-VM
2024-04-24T23:19:08.649592955Z       vnf_member_index: vnf
2024-04-24T23:19:08.649595928Z   - alert: scaleout_7c59814d-8f05-4d26-9fd2-66ec30823798
2024-04-24T23:19:08.649601220Z     expr: (count (osm_cpu_utilization{ns_id="81dfdac3-2fc7-405b-861d-667a23425157",
2024-04-24T23:19:08.649604371Z       vnf_member_index="vnf", vdu_id="hackfest_basic_metrics-VM"}) < 5) and (avg(osm_cpu_utilization{ns_id="81dfdac3-2fc7-405b-861d-667a23425157",
2024-04-24T23:19:08.649607363Z       vnf_member_index="vnf", vdu_id="hackfest_basic_metrics-VM"}) > 60.0000000000)
2024-04-24T23:19:08.649610255Z     for: 1m
2024-04-24T23:19:08.649613245Z     labels:
2024-04-24T23:19:08.649616125Z       ns_id: 81dfdac3-2fc7-405b-861d-667a23425157
2024-04-24T23:19:08.649619014Z       vdu_id: hackfest_basic_metrics-VM
2024-04-24T23:19:08.649622024Z       vnf_member_index: vnf
2024-04-24T23:19:08.649664286Z   - alert: scalein_d2b585ff-71d3-4ae5-86db-267143351d7d
2024-04-24T23:19:08.649668531Z     expr: (count (osm_cpu_utilization{ns_id="81dfdac3-2fc7-405b-861d-667a23425157",
2024-04-24T23:19:08.649671706Z       vnf_member_index="vnf", vdu_id="hackfest_basic_metrics-VM"}) > 3) and (avg(osm_cpu_utilization{ns_id="81dfdac3-2fc7-405b-861d-667a23425157",
2024-04-24T23:19:08.649674674Z       vnf_member_index="vnf", vdu_id="hackfest_basic_metrics-VM"}) < 10.0000000000)
2024-04-24T23:19:08.649677569Z     for: 1m
2024-04-24T23:19:08.649680492Z     labels:
2024-04-24T23:19:08.649683391Z       ns_id: 81dfdac3-2fc7-405b-861d-667a23425157
2024-04-24T23:19:08.649686345Z       vdu_id: hackfest_basic_metrics-VM
2024-04-24T23:19:08.649689442Z       vnf_member_index: vnf
2024-04-24T23:19:08.649692752Z   - alert: scaleout_5ff47137-da8b-432a-b231-acd130a1992f
2024-04-24T23:19:08.649695815Z     expr: (count (osm_cpu_utilization{ns_id="81dfdac3-2fc7-405b-861d-667a23425157",
2024-04-24T23:19:08.649698700Z       vnf_member_index="vnf", vdu_id="hackfest_basic_metrics-VM"}) < 5) and (avg(osm_cpu_utilization{ns_id="81dfdac3-2fc7-405b-861d-667a23425157",
2024-04-24T23:19:08.649701608Z       vnf_member_index="vnf", vdu_id="hackfest_basic_metrics-VM"}) > 60.0000000000)
2024-04-24T23:19:08.649704533Z     for: 1m
2024-04-24T23:19:08.649707362Z     labels:
2024-04-24T23:19:08.649710272Z       ns_id: 81dfdac3-2fc7-405b-861d-667a23425157
2024-04-24T23:19:08.649713287Z       vdu_id: hackfest_basic_metrics-VM
2024-04-24T23:19:08.649716179Z       vnf_member_index: vnf
2024-04-24T23:19:08.649719067Z   - alert: scalein_063ed24b-fdef-4dd6-9df4-7dcbe473b819
2024-04-24T23:19:08.649722096Z     expr: (count (osm_cpu_utilization{ns_id="81dfdac3-2fc7-405b-861d-667a23425157",
2024-04-24T23:19:08.649725116Z       vnf_member_index="vnf", vdu_id="hackfest_basic_metrics-VM"}) > 3) and (avg(osm_cpu_utilization{ns_id="81dfdac3-2fc7-405b-861d-667a23425157",
2024-04-24T23:19:08.649727999Z       vnf_member_index="vnf", vdu_id="hackfest_basic_metrics-VM"}) < 10.0000000000)
2024-04-24T23:19:08.649730888Z     for: 1m
2024-04-24T23:19:08.649733742Z     labels:
2024-04-24T23:19:08.649736610Z       ns_id: 81dfdac3-2fc7-405b-861d-667a23425157
2024-04-24T23:19:08.649739567Z       vdu_id: hackfest_basic_metrics-VM
2024-04-24T23:19:08.649742398Z       vnf_member_index: vnf
2024-04-24T23:19:08.649745600Z   - alert: scaleout_effb3995-4125-4a6f-8d93-f6231f86a25b
2024-04-24T23:19:08.649748534Z     expr: (count (osm_cpu_utilization{ns_id="81dfdac3-2fc7-405b-861d-667a23425157",
2024-04-24T23:19:08.649751453Z       vnf_member_index="vnf", vdu_id="hackfest_basic_metrics-VM"}) < 5) and (avg(osm_cpu_utilization{ns_id="81dfdac3-2fc7-405b-861d-667a23425157",
2024-04-24T23:19:08.649759910Z       vnf_member_index="vnf", vdu_id="hackfest_basic_metrics-VM"}) > 60.0000000000)
2024-04-24T23:19:08.649762874Z     for: 1m
2024-04-24T23:19:08.649765716Z     labels:
2024-04-24T23:19:08.649768609Z       ns_id: 81dfdac3-2fc7-405b-861d-667a23425157
2024-04-24T23:19:08.649771542Z       vdu_id: hackfest_basic_metrics-VM
2024-04-24T23:19:08.649774421Z       vnf_member_index: vnf
2024-04-24T23:19:08.649777149Z 
2024-04-24T23:19:08.655878810Z New config written, updating prometheus
2024-04-24T23:19:13.693757751Z Prometheus config update successful
2024-04-24T23:19:13.693785953Z 
2024-04-24T23:19:13.693790019Z Changes detected, updating prometheus config
2024-04-24T23:19:13.693793254Z Jobs detected: 0
2024-04-24T23:19:13.693796644Z Writing new config file to /etc/prometheus/prometheus.yml
2024-04-24T23:19:13.693799722Z alerting:
2024-04-24T23:19:13.693802737Z   alertmanagers:
2024-04-24T23:19:13.693806389Z   - static_configs:
2024-04-24T23:19:13.693809688Z     - targets:
2024-04-24T23:19:13.693813325Z       - alertmanager:9093
2024-04-24T23:19:13.693816275Z global:
2024-04-24T23:19:13.693819332Z   evaluation_interval: 15s
2024-04-24T23:19:13.693822348Z   scrape_interval: 15s
2024-04-24T23:19:13.693825295Z rule_files:
2024-04-24T23:19:13.693828402Z - osm_metric_rules.yml
2024-04-24T23:19:13.693831692Z - osm_alert_rules.yml
2024-04-24T23:19:13.693834615Z scrape_configs:
2024-04-24T23:19:13.693837768Z - job_name: mon_exporter
2024-04-24T23:19:13.693840691Z   static_configs:
2024-04-24T23:19:13.693843948Z   - targets:
2024-04-24T23:19:13.693847192Z     - mon:8000
2024-04-24T23:19:13.693850120Z - honor_labels: true
2024-04-24T23:19:13.693853069Z   job_name: pushgateway
2024-04-24T23:19:13.693856112Z   scrape_interval: 30s
2024-04-24T23:19:13.693859047Z   static_configs:
2024-04-24T23:19:13.693861903Z   - targets:
2024-04-24T23:19:13.693864978Z     - pushgateway-prometheus-pushgateway:9091
2024-04-24T23:19:13.693867698Z 
2024-04-24T23:19:13.693870675Z Alerts read: 6
2024-04-24T23:19:13.693931445Z Writing new alerts file to /etc/prometheus/osm_alert_rules.yml
2024-04-24T23:19:13.699427510Z groups:
2024-04-24T23:19:13.699442180Z - name: osm_alert_rules
2024-04-24T23:19:13.699445634Z   rules:
2024-04-24T23:19:13.699448592Z   - alert: vdu_down
2024-04-24T23:19:13.699451710Z     annotations:
2024-04-24T23:19:13.699455393Z       description: VDU {{ $labels.vm_id }} in VIM {{ $labels.vim_id }} has been down
2024-04-24T23:19:13.699458726Z         for more than 3 minutes. NS instance id is {{ $labels.ns_id }}
2024-04-24T23:19:13.699461779Z       summary: VDU {{ $labels.vm_id }} in VIM {{ $labels.vim_id }} is down
2024-04-24T23:19:13.699464987Z     expr: vm_status_extended != 1
2024-04-24T23:19:13.699476151Z     for: 3m
2024-04-24T23:19:13.699496261Z - name: _osm_alert_rules_20240424231913_
2024-04-24T23:19:13.699499264Z   rules:
2024-04-24T23:19:13.699502829Z   - alert: scalein_1df09c7d-1d3f-4725-9b53-b496a20b91a2
2024-04-24T23:19:13.699506047Z     expr: (count (osm_cpu_utilization{ns_id="81dfdac3-2fc7-405b-861d-667a23425157",
2024-04-24T23:19:13.699512236Z       vnf_member_index="vnf", vdu_id="hackfest_basic_metrics-VM"}) > 3) and (avg(osm_cpu_utilization{ns_id="81dfdac3-2fc7-405b-861d-667a23425157",
2024-04-24T23:19:13.699515893Z       vnf_member_index="vnf", vdu_id="hackfest_basic_metrics-VM"}) < 10.0000000000)
2024-04-24T23:19:13.699518960Z     for: 1m
2024-04-24T23:19:13.699521974Z     labels:
2024-04-24T23:19:13.699535130Z       ns_id: 81dfdac3-2fc7-405b-861d-667a23425157
2024-04-24T23:19:13.699538157Z       vdu_id: hackfest_basic_metrics-VM
2024-04-24T23:19:13.699541497Z       vnf_member_index: vnf
2024-04-24T23:19:13.699553638Z   - alert: scaleout_7c59814d-8f05-4d26-9fd2-66ec30823798
2024-04-24T23:19:13.699556666Z     expr: (count (osm_cpu_utilization{ns_id="81dfdac3-2fc7-405b-861d-667a23425157",
2024-04-24T23:19:13.699559785Z       vnf_member_index="vnf", vdu_id="hackfest_basic_metrics-VM"}) < 5) and (avg(osm_cpu_utilization{ns_id="81dfdac3-2fc7-405b-861d-667a23425157",
2024-04-24T23:19:13.699562842Z       vnf_member_index="vnf", vdu_id="hackfest_basic_metrics-VM"}) > 60.0000000000)
2024-04-24T23:19:13.699577438Z     for: 1m
2024-04-24T23:19:13.699580152Z     labels:
2024-04-24T23:19:13.699583050Z       ns_id: 81dfdac3-2fc7-405b-861d-667a23425157
2024-04-24T23:19:13.699585855Z       vdu_id: hackfest_basic_metrics-VM
2024-04-24T23:19:13.699588628Z       vnf_member_index: vnf
2024-04-24T23:19:13.699591408Z   - alert: scalein_d2b585ff-71d3-4ae5-86db-267143351d7d
2024-04-24T23:19:13.699594270Z     expr: (count (osm_cpu_utilization{ns_id="81dfdac3-2fc7-405b-861d-667a23425157",
2024-04-24T23:19:13.699597235Z       vnf_member_index="vnf", vdu_id="hackfest_basic_metrics-VM"}) > 3) and (avg(osm_cpu_utilization{ns_id="81dfdac3-2fc7-405b-861d-667a23425157",
2024-04-24T23:19:13.699599977Z       vnf_member_index="vnf", vdu_id="hackfest_basic_metrics-VM"}) < 10.0000000000)
2024-04-24T23:19:13.699602797Z     for: 1m
2024-04-24T23:19:13.699605561Z     labels:
2024-04-24T23:19:13.699608332Z       ns_id: 81dfdac3-2fc7-405b-861d-667a23425157
2024-04-24T23:19:13.699611090Z       vdu_id: hackfest_basic_metrics-VM
2024-04-24T23:19:13.699614123Z       vnf_member_index: vnf
2024-04-24T23:19:13.699617081Z   - alert: scaleout_5ff47137-da8b-432a-b231-acd130a1992f
2024-04-24T23:19:13.699619790Z     expr: (count (osm_cpu_utilization{ns_id="81dfdac3-2fc7-405b-861d-667a23425157",
2024-04-24T23:19:13.699622584Z       vnf_member_index="vnf", vdu_id="hackfest_basic_metrics-VM"}) < 5) and (avg(osm_cpu_utilization{ns_id="81dfdac3-2fc7-405b-861d-667a23425157",
2024-04-24T23:19:13.699625319Z       vnf_member_index="vnf", vdu_id="hackfest_basic_metrics-VM"}) > 60.0000000000)
2024-04-24T23:19:13.699628086Z     for: 1m
2024-04-24T23:19:13.699630766Z     labels:
2024-04-24T23:19:13.699633758Z       ns_id: 81dfdac3-2fc7-405b-861d-667a23425157
2024-04-24T23:19:13.699636502Z       vdu_id: hackfest_basic_metrics-VM
2024-04-24T23:19:13.699639302Z       vnf_member_index: vnf
2024-04-24T23:19:13.699642048Z   - alert: scalein_063ed24b-fdef-4dd6-9df4-7dcbe473b819
2024-04-24T23:19:13.699644795Z     expr: (count (osm_cpu_utilization{ns_id="81dfdac3-2fc7-405b-861d-667a23425157",
2024-04-24T23:19:13.699647762Z       vnf_member_index="vnf", vdu_id="hackfest_basic_metrics-VM"}) > 3) and (avg(osm_cpu_utilization{ns_id="81dfdac3-2fc7-405b-861d-667a23425157",
2024-04-24T23:19:13.699650446Z       vnf_member_index="vnf", vdu_id="hackfest_basic_metrics-VM"}) < 10.0000000000)
2024-04-24T23:19:13.699653215Z     for: 1m
2024-04-24T23:19:13.699655916Z     labels:
2024-04-24T23:19:13.699675670Z       ns_id: 81dfdac3-2fc7-405b-861d-667a23425157
2024-04-24T23:19:13.699678734Z       vdu_id: hackfest_basic_metrics-VM
2024-04-24T23:19:13.699681955Z       vnf_member_index: vnf
2024-04-24T23:19:13.699684872Z   - alert: scaleout_effb3995-4125-4a6f-8d93-f6231f86a25b
2024-04-24T23:19:13.699687728Z     expr: (count (osm_cpu_utilization{ns_id="81dfdac3-2fc7-405b-861d-667a23425157",
2024-04-24T23:19:13.699690666Z       vnf_member_index="vnf", vdu_id="hackfest_basic_metrics-VM"}) < 5) and (avg(osm_cpu_utilization{ns_id="81dfdac3-2fc7-405b-861d-667a23425157",
2024-04-24T23:19:13.699697621Z       vnf_member_index="vnf", vdu_id="hackfest_basic_metrics-VM"}) > 60.0000000000)
2024-04-24T23:19:13.699700584Z     for: 1m
2024-04-24T23:19:13.699703440Z     labels:
2024-04-24T23:19:13.699706728Z       ns_id: 81dfdac3-2fc7-405b-861d-667a23425157
2024-04-24T23:19:13.699709650Z       vdu_id: hackfest_basic_metrics-VM
2024-04-24T23:19:13.699712598Z       vnf_member_index: vnf
2024-04-24T23:19:13.699715319Z 
2024-04-24T23:19:13.715057411Z New config written, updating prometheus
2024-04-24T23:19:18.758279845Z Prometheus config update successful
2024-04-24T23:19:18.758332671Z 
2024-04-24T23:19:18.758343456Z Changes detected, updating prometheus config
2024-04-24T23:19:18.759675929Z Jobs detected: 0
2024-04-24T23:19:18.765076104Z Writing new config file to /etc/prometheus/prometheus.yml
2024-04-24T23:19:18.767703063Z alerting:
2024-04-24T23:19:18.767732708Z   alertmanagers:
2024-04-24T23:19:18.767743094Z   - static_configs:
2024-04-24T23:19:18.767751894Z     - targets:
2024-04-24T23:19:18.767761178Z       - alertmanager:9093
2024-04-24T23:19:18.767769706Z global:
2024-04-24T23:19:18.767778228Z   evaluation_interval: 15s
2024-04-24T23:19:18.767786554Z   scrape_interval: 15s
2024-04-24T23:19:18.767794997Z rule_files:
2024-04-24T23:19:18.767803406Z - osm_metric_rules.yml
2024-04-24T23:19:18.767812098Z - osm_alert_rules.yml
2024-04-24T23:19:18.767820289Z scrape_configs:
2024-04-24T23:19:18.767828543Z - job_name: mon_exporter
2024-04-24T23:19:18.767836848Z   static_configs:
2024-04-24T23:19:18.767845399Z   - targets:
2024-04-24T23:19:18.767853718Z     - mon:8000
2024-04-24T23:19:18.767862019Z - honor_labels: true
2024-04-24T23:19:18.767870309Z   job_name: pushgateway
2024-04-24T23:19:18.767878739Z   scrape_interval: 30s
2024-04-24T23:19:18.767887094Z   static_configs:
2024-04-24T23:19:18.767895329Z   - targets:
2024-04-24T23:19:18.767904849Z     - pushgateway-prometheus-pushgateway:9091
2024-04-24T23:19:18.767913023Z 
2024-04-24T23:19:18.772786863Z Alerts read: 6
2024-04-24T23:19:18.774484685Z Writing new alerts file to /etc/prometheus/osm_alert_rules.yml
2024-04-24T23:19:18.780603644Z groups:
2024-04-24T23:19:18.780635584Z - name: osm_alert_rules
2024-04-24T23:19:18.780644681Z   rules:
2024-04-24T23:19:18.780653167Z   - alert: vdu_down
2024-04-24T23:19:18.780662930Z     annotations:
2024-04-24T23:19:18.780672542Z       description: VDU {{ $labels.vm_id }} in VIM {{ $labels.vim_id }} has been down
2024-04-24T23:19:18.780681865Z         for more than 3 minutes. NS instance id is {{ $labels.ns_id }}
2024-04-24T23:19:18.780690232Z       summary: VDU {{ $labels.vm_id }} in VIM {{ $labels.vim_id }} is down
2024-04-24T23:19:18.780699388Z     expr: vm_status_extended != 1
2024-04-24T23:19:18.780707860Z     for: 3m
2024-04-24T23:19:18.780716321Z - name: _osm_alert_rules_20240424231918_
2024-04-24T23:19:18.780724760Z   rules:
2024-04-24T23:19:18.780734334Z   - alert: scalein_1df09c7d-1d3f-4725-9b53-b496a20b91a2
2024-04-24T23:19:18.780742889Z     expr: (count (osm_cpu_utilization{ns_id="81dfdac3-2fc7-405b-861d-667a23425157",
2024-04-24T23:19:18.780752577Z       vnf_member_index="vnf", vdu_id="hackfest_basic_metrics-VM"}) > 3) and (avg(osm_cpu_utilization{ns_id="81dfdac3-2fc7-405b-861d-667a23425157",
2024-04-24T23:19:18.780761159Z       vnf_member_index="vnf", vdu_id="hackfest_basic_metrics-VM"}) < 10.0000000000)
2024-04-24T23:19:18.780769720Z     for: 1m
2024-04-24T23:19:18.780778080Z     labels:
2024-04-24T23:19:18.780786395Z       ns_id: 81dfdac3-2fc7-405b-861d-667a23425157
2024-04-24T23:19:18.780823280Z       vdu_id: hackfest_basic_metrics-VM
2024-04-24T23:19:18.780833419Z       vnf_member_index: vnf
2024-04-24T23:19:18.780841921Z   - alert: scaleout_7c59814d-8f05-4d26-9fd2-66ec30823798
2024-04-24T23:19:18.780850647Z     expr: (count (osm_cpu_utilization{ns_id="81dfdac3-2fc7-405b-861d-667a23425157",
2024-04-24T23:19:18.780859104Z       vnf_member_index="vnf", vdu_id="hackfest_basic_metrics-VM"}) < 5) and (avg(osm_cpu_utilization{ns_id="81dfdac3-2fc7-405b-861d-667a23425157",
2024-04-24T23:19:18.780867273Z       vnf_member_index="vnf", vdu_id="hackfest_basic_metrics-VM"}) > 60.0000000000)
2024-04-24T23:19:18.780875862Z     for: 1m
2024-04-24T23:19:18.780884183Z     labels:
2024-04-24T23:19:18.780892385Z       ns_id: 81dfdac3-2fc7-405b-861d-667a23425157
2024-04-24T23:19:18.780900619Z       vdu_id: hackfest_basic_metrics-VM
2024-04-24T23:19:18.780908994Z       vnf_member_index: vnf
2024-04-24T23:19:18.780917129Z   - alert: scalein_d2b585ff-71d3-4ae5-86db-267143351d7d
2024-04-24T23:19:18.780925321Z     expr: (count (osm_cpu_utilization{ns_id="81dfdac3-2fc7-405b-861d-667a23425157",
2024-04-24T23:19:18.780933572Z       vnf_member_index="vnf", vdu_id="hackfest_basic_metrics-VM"}) > 3) and (avg(osm_cpu_utilization{ns_id="81dfdac3-2fc7-405b-861d-667a23425157",
2024-04-24T23:19:18.780941809Z       vnf_member_index="vnf", vdu_id="hackfest_basic_metrics-VM"}) < 10.0000000000)
2024-04-24T23:19:18.780950011Z     for: 1m
2024-04-24T23:19:18.780958222Z     labels:
2024-04-24T23:19:18.780987971Z       ns_id: 81dfdac3-2fc7-405b-861d-667a23425157
2024-04-24T23:19:18.780996723Z       vdu_id: hackfest_basic_metrics-VM
2024-04-24T23:19:18.781004933Z       vnf_member_index: vnf
2024-04-24T23:19:18.781013178Z   - alert: scaleout_5ff47137-da8b-432a-b231-acd130a1992f
2024-04-24T23:19:18.781021373Z     expr: (count (osm_cpu_utilization{ns_id="81dfdac3-2fc7-405b-861d-667a23425157",
2024-04-24T23:19:18.781029773Z       vnf_member_index="vnf", vdu_id="hackfest_basic_metrics-VM"}) < 5) and (avg(osm_cpu_utilization{ns_id="81dfdac3-2fc7-405b-861d-667a23425157",
2024-04-24T23:19:18.781038056Z       vnf_member_index="vnf", vdu_id="hackfest_basic_metrics-VM"}) > 60.0000000000)
2024-04-24T23:19:18.781046434Z     for: 1m
2024-04-24T23:19:18.781054951Z     labels:
2024-04-24T23:19:18.781063240Z       ns_id: 81dfdac3-2fc7-405b-861d-667a23425157
2024-04-24T23:19:18.781071706Z       vdu_id: hackfest_basic_metrics-VM
2024-04-24T23:19:18.781079869Z       vnf_member_index: vnf
2024-04-24T23:19:18.781088091Z   - alert: scalein_063ed24b-fdef-4dd6-9df4-7dcbe473b819
2024-04-24T23:19:18.781096210Z     expr: (count (osm_cpu_utilization{ns_id="81dfdac3-2fc7-405b-861d-667a23425157",
2024-04-24T23:19:18.781104543Z       vnf_member_index="vnf", vdu_id="hackfest_basic_metrics-VM"}) > 3) and (avg(osm_cpu_utilization{ns_id="81dfdac3-2fc7-405b-861d-667a23425157",
2024-04-24T23:19:18.781112846Z       vnf_member_index="vnf", vdu_id="hackfest_basic_metrics-VM"}) < 10.0000000000)
2024-04-24T23:19:18.781122801Z     for: 1m
2024-04-24T23:19:18.781148267Z     labels:
2024-04-24T23:19:18.781161966Z       ns_id: 81dfdac3-2fc7-405b-861d-667a23425157
2024-04-24T23:19:18.781174451Z       vdu_id: hackfest_basic_metrics-VM
2024-04-24T23:19:18.781187097Z       vnf_member_index: vnf
2024-04-24T23:19:18.781199728Z   - alert: scaleout_effb3995-4125-4a6f-8d93-f6231f86a25b
2024-04-24T23:19:18.781211773Z     expr: (count (osm_cpu_utilization{ns_id="81dfdac3-2fc7-405b-861d-667a23425157",
2024-04-24T23:19:18.781220890Z       vnf_member_index="vnf", vdu_id="hackfest_basic_metrics-VM"}) < 5) and (avg(osm_cpu_utilization{ns_id="81dfdac3-2fc7-405b-861d-667a23425157",
2024-04-24T23:19:18.781229172Z       vnf_member_index="vnf", vdu_id="hackfest_basic_metrics-VM"}) > 60.0000000000)
2024-04-24T23:19:18.781251716Z     for: 1m
2024-04-24T23:19:18.781260101Z     labels:
2024-04-24T23:19:18.781268325Z       ns_id: 81dfdac3-2fc7-405b-861d-667a23425157
2024-04-24T23:19:18.781276458Z       vdu_id: hackfest_basic_metrics-VM
2024-04-24T23:19:18.781284672Z       vnf_member_index: vnf
2024-04-24T23:19:18.781292490Z 
2024-04-24T23:19:18.786596661Z New config written, updating prometheus
2024-04-24T23:19:23.809885264Z Prometheus config update successful
2024-04-24T23:19:23.809935200Z 
2024-04-24T23:20:20.636941732Z Changes detected, updating prometheus config
2024-04-24T23:20:20.638678836Z Jobs detected: 0
2024-04-24T23:20:20.648981464Z Writing new config file to /etc/prometheus/prometheus.yml
2024-04-24T23:20:20.654806738Z alerting:
2024-04-24T23:20:20.654828140Z   alertmanagers:
2024-04-24T23:20:20.654832897Z   - static_configs:
2024-04-24T23:20:20.654836764Z     - targets:
2024-04-24T23:20:20.654841151Z       - alertmanager:9093
2024-04-24T23:20:20.654862044Z global:
2024-04-24T23:20:20.654866143Z   evaluation_interval: 15s
2024-04-24T23:20:20.654869839Z   scrape_interval: 15s
2024-04-24T23:20:20.654873565Z rule_files:
2024-04-24T23:20:20.654877136Z - osm_metric_rules.yml
2024-04-24T23:20:20.654881509Z - osm_alert_rules.yml
2024-04-24T23:20:20.654885239Z scrape_configs:
2024-04-24T23:20:20.654888820Z - job_name: mon_exporter
2024-04-24T23:20:20.654892512Z   static_configs:
2024-04-24T23:20:20.654896212Z   - targets:
2024-04-24T23:20:20.654900036Z     - mon:8000
2024-04-24T23:20:20.654908999Z - honor_labels: true
2024-04-24T23:20:20.654912762Z   job_name: pushgateway
2024-04-24T23:20:20.654916383Z   scrape_interval: 30s
2024-04-24T23:20:20.654919974Z   static_configs:
2024-04-24T23:20:20.654938599Z   - targets:
2024-04-24T23:20:20.654943446Z     - pushgateway-prometheus-pushgateway:9091
2024-04-24T23:20:20.654947115Z 
2024-04-24T23:20:20.667521779Z Alerts read: 0
2024-04-24T23:20:20.673885639Z Writing new alerts file to /etc/prometheus/osm_alert_rules.yml
2024-04-24T23:20:20.676202423Z groups:
2024-04-24T23:20:20.676214972Z - name: osm_alert_rules
2024-04-24T23:20:20.676219921Z   rules:
2024-04-24T23:20:20.676224323Z   - alert: vdu_down
2024-04-24T23:20:20.676229025Z     annotations:
2024-04-24T23:20:20.676233992Z       description: VDU {{ $labels.vm_id }} in VIM {{ $labels.vim_id }} has been down
2024-04-24T23:20:20.676238921Z         for more than 3 minutes. NS instance id is {{ $labels.ns_id }}
2024-04-24T23:20:20.676243281Z       summary: VDU {{ $labels.vm_id }} in VIM {{ $labels.vim_id }} is down
2024-04-24T23:20:20.676248128Z     expr: vm_status_extended != 1
2024-04-24T23:20:20.676297571Z     for: 3m
2024-04-24T23:20:20.676302616Z 
2024-04-24T23:20:20.677768528Z New config written, updating prometheus
2024-04-24T23:20:25.698142239Z Prometheus config update successful
2024-04-24T23:20:25.698178304Z 
2024-04-24T23:20:25.698183146Z Changes detected, updating prometheus config
2024-04-24T23:20:25.699561658Z Jobs detected: 0
2024-04-24T23:20:25.705076577Z Writing new config file to /etc/prometheus/prometheus.yml
2024-04-24T23:20:25.706915757Z alerting:
2024-04-24T23:20:25.706927463Z   alertmanagers:
2024-04-24T23:20:25.706931387Z   - static_configs:
2024-04-24T23:20:25.706934309Z     - targets:
2024-04-24T23:20:25.706937772Z       - alertmanager:9093
2024-04-24T23:20:25.706940768Z global:
2024-04-24T23:20:25.706943776Z   evaluation_interval: 15s
2024-04-24T23:20:25.706946805Z   scrape_interval: 15s
2024-04-24T23:20:25.706965684Z rule_files:
2024-04-24T23:20:25.706968754Z - osm_metric_rules.yml
2024-04-24T23:20:25.706971669Z - osm_alert_rules.yml
2024-04-24T23:20:25.706974588Z scrape_configs:
2024-04-24T23:20:25.706977429Z - job_name: mon_exporter
2024-04-24T23:20:25.706980706Z   static_configs:
2024-04-24T23:20:25.706983662Z   - targets:
2024-04-24T23:20:25.706986574Z     - mon:8000
2024-04-24T23:20:25.706989432Z - honor_labels: true
2024-04-24T23:20:25.706992233Z   job_name: pushgateway
2024-04-24T23:20:25.706995084Z   scrape_interval: 30s
2024-04-24T23:20:25.706997953Z   static_configs:
2024-04-24T23:20:25.707001119Z   - targets:
2024-04-24T23:20:25.707004985Z     - pushgateway-prometheus-pushgateway:9091
2024-04-24T23:20:25.707007801Z 
2024-04-24T23:20:25.709897837Z Alerts read: 0
2024-04-24T23:20:25.711695366Z Writing new alerts file to /etc/prometheus/osm_alert_rules.yml
2024-04-24T23:20:25.712840422Z groups:
2024-04-24T23:20:25.712850958Z - name: osm_alert_rules
2024-04-24T23:20:25.712854254Z   rules:
2024-04-24T23:20:25.712857351Z   - alert: vdu_down
2024-04-24T23:20:25.712860416Z     annotations:
2024-04-24T23:20:25.712864309Z       description: VDU {{ $labels.vm_id }} in VIM {{ $labels.vim_id }} has been down
2024-04-24T23:20:25.712868417Z         for more than 3 minutes. NS instance id is {{ $labels.ns_id }}
2024-04-24T23:20:25.712871613Z       summary: VDU {{ $labels.vm_id }} in VIM {{ $labels.vim_id }} is down
2024-04-24T23:20:25.712875419Z     expr: vm_status_extended != 1
2024-04-24T23:20:25.712878511Z     for: 3m
2024-04-24T23:20:25.712881350Z 
2024-04-24T23:20:25.713928940Z New config written, updating prometheus
2024-04-24T23:20:30.737743236Z Prometheus config update successful
2024-04-24T23:20:30.737773526Z 
2024-04-24T23:20:30.737778719Z Changes detected, updating prometheus config
2024-04-24T23:20:30.741729795Z Jobs detected: 0
2024-04-24T23:20:30.745331257Z Writing new config file to /etc/prometheus/prometheus.yml
2024-04-24T23:20:30.746088894Z alerting:
2024-04-24T23:20:30.746134497Z   alertmanagers:
2024-04-24T23:20:30.746145515Z   - static_configs:
2024-04-24T23:20:30.746174071Z     - targets:
2024-04-24T23:20:30.746196282Z       - alertmanager:9093
2024-04-24T23:20:30.746202684Z global:
2024-04-24T23:20:30.746209396Z   evaluation_interval: 15s
2024-04-24T23:20:30.746214993Z   scrape_interval: 15s
2024-04-24T23:20:30.746220597Z rule_files:
2024-04-24T23:20:30.746239534Z - osm_metric_rules.yml
2024-04-24T23:20:30.746245876Z - osm_alert_rules.yml
2024-04-24T23:20:30.746252681Z scrape_configs:
2024-04-24T23:20:30.746261036Z - job_name: mon_exporter
2024-04-24T23:20:30.746268793Z   static_configs:
2024-04-24T23:20:30.746276932Z   - targets:
2024-04-24T23:20:30.746305047Z     - mon:8000
2024-04-24T23:20:30.746314156Z - honor_labels: true
2024-04-24T23:20:30.746343718Z   job_name: pushgateway
2024-04-24T23:20:30.746350532Z   scrape_interval: 30s
2024-04-24T23:20:30.746356595Z   static_configs:
2024-04-24T23:20:30.746362216Z   - targets:
2024-04-24T23:20:30.746383801Z     - pushgateway-prometheus-pushgateway:9091
2024-04-24T23:20:30.746389486Z 
2024-04-24T23:20:30.763055545Z Alerts read: 0
2024-04-24T23:20:30.763080922Z Writing new alerts file to /etc/prometheus/osm_alert_rules.yml
2024-04-24T23:20:30.763087608Z groups:
2024-04-24T23:20:30.763094104Z - name: osm_alert_rules
2024-04-24T23:20:30.763098196Z   rules:
2024-04-24T23:20:30.763101841Z   - alert: vdu_down
2024-04-24T23:20:30.763105488Z     annotations:
2024-04-24T23:20:30.763122668Z       description: VDU {{ $labels.vm_id }} in VIM {{ $labels.vim_id }} has been down
2024-04-24T23:20:30.763127638Z         for more than 3 minutes. NS instance id is {{ $labels.ns_id }}
2024-04-24T23:20:30.763131445Z       summary: VDU {{ $labels.vm_id }} in VIM {{ $labels.vim_id }} is down
2024-04-24T23:20:30.763135845Z     expr: vm_status_extended != 1
2024-04-24T23:20:30.763139617Z     for: 3m
2024-04-24T23:20:30.763143206Z 
2024-04-24T23:20:30.775625496Z New config written, updating prometheus
2024-04-24T23:20:35.798074897Z Prometheus config update successful
2024-04-24T23:20:35.798131261Z 
2024-04-24T23:20:35.798149823Z Changes detected, updating prometheus config
2024-04-24T23:20:35.799294808Z Jobs detected: 0
2024-04-24T23:20:35.804239044Z Writing new config file to /etc/prometheus/prometheus.yml
2024-04-24T23:20:35.806009595Z alerting:
2024-04-24T23:20:35.806021940Z   alertmanagers:
2024-04-24T23:20:35.806026747Z   - static_configs:
2024-04-24T23:20:35.806030558Z     - targets:
2024-04-24T23:20:35.806034744Z       - alertmanager:9093
2024-04-24T23:20:35.806038491Z global:
2024-04-24T23:20:35.806042138Z   evaluation_interval: 15s
2024-04-24T23:20:35.806045773Z   scrape_interval: 15s
2024-04-24T23:20:35.806049766Z rule_files:
2024-04-24T23:20:35.806053497Z - osm_metric_rules.yml
2024-04-24T23:20:35.806057169Z - osm_alert_rules.yml
2024-04-24T23:20:35.806060819Z scrape_configs:
2024-04-24T23:20:35.806064506Z - job_name: mon_exporter
2024-04-24T23:20:35.806068112Z   static_configs:
2024-04-24T23:20:35.806071716Z   - targets:
2024-04-24T23:20:35.806075392Z     - mon:8000
2024-04-24T23:20:35.806079068Z - honor_labels: true
2024-04-24T23:20:35.806082741Z   job_name: pushgateway
2024-04-24T23:20:35.806086334Z   scrape_interval: 30s
2024-04-24T23:20:35.806090277Z   static_configs:
2024-04-24T23:20:35.806093857Z   - targets:
2024-04-24T23:20:35.806098408Z     - pushgateway-prometheus-pushgateway:9091
2024-04-24T23:20:35.806101930Z 
2024-04-24T23:20:35.809438334Z Alerts read: 0
2024-04-24T23:20:35.811221108Z Writing new alerts file to /etc/prometheus/osm_alert_rules.yml
2024-04-24T23:20:35.812342254Z groups:
2024-04-24T23:20:35.812355335Z - name: osm_alert_rules
2024-04-24T23:20:35.812360134Z   rules:
2024-04-24T23:20:35.812364317Z   - alert: vdu_down
2024-04-24T23:20:35.812368333Z     annotations:
2024-04-24T23:20:35.812373380Z       description: VDU {{ $labels.vm_id }} in VIM {{ $labels.vim_id }} has been down
2024-04-24T23:20:35.812378220Z         for more than 3 minutes. NS instance id is {{ $labels.ns_id }}
2024-04-24T23:20:35.812382304Z       summary: VDU {{ $labels.vm_id }} in VIM {{ $labels.vim_id }} is down
2024-04-24T23:20:35.812386797Z     expr: vm_status_extended != 1
2024-04-24T23:20:35.812391049Z     for: 3m
2024-04-24T23:20:35.812395201Z 
2024-04-24T23:20:35.813368078Z New config written, updating prometheus
2024-04-24T23:20:40.836328192Z Prometheus config update successful
2024-04-24T23:20:40.836364213Z 
2024-04-24T23:20:40.836405431Z Changes detected, updating prometheus config
2024-04-24T23:20:40.838085437Z Jobs detected: 0
2024-04-24T23:20:40.842854637Z Writing new config file to /etc/prometheus/prometheus.yml
2024-04-24T23:20:40.844620882Z alerting:
2024-04-24T23:20:40.844632681Z   alertmanagers:
2024-04-24T23:20:40.844637106Z   - static_configs:
2024-04-24T23:20:40.844640171Z     - targets:
2024-04-24T23:20:40.844643648Z       - alertmanager:9093
2024-04-24T23:20:40.844646914Z global:
2024-04-24T23:20:40.844650251Z   evaluation_interval: 15s
2024-04-24T23:20:40.844668693Z   scrape_interval: 15s
2024-04-24T23:20:40.844671838Z rule_files:
2024-04-24T23:20:40.844674794Z - osm_metric_rules.yml
2024-04-24T23:20:40.844677746Z - osm_alert_rules.yml
2024-04-24T23:20:40.844681216Z scrape_configs:
2024-04-24T23:20:40.844684101Z - job_name: mon_exporter
2024-04-24T23:20:40.844686922Z   static_configs:
2024-04-24T23:20:40.844690326Z   - targets:
2024-04-24T23:20:40.844693244Z     - mon:8000
2024-04-24T23:20:40.844696145Z - honor_labels: true
2024-04-24T23:20:40.844698977Z   job_name: pushgateway
2024-04-24T23:20:40.844701909Z   scrape_interval: 30s
2024-04-24T23:20:40.844704781Z   static_configs:
2024-04-24T23:20:40.844707600Z   - targets:
2024-04-24T23:20:40.844711155Z     - pushgateway-prometheus-pushgateway:9091
2024-04-24T23:20:40.844714055Z 
2024-04-24T23:20:40.849727077Z Alerts read: 0
2024-04-24T23:20:40.851563149Z Writing new alerts file to /etc/prometheus/osm_alert_rules.yml
2024-04-24T23:20:40.852934693Z groups:
2024-04-24T23:20:40.852944454Z - name: osm_alert_rules
2024-04-24T23:20:40.852948281Z   rules:
2024-04-24T23:20:40.852951391Z   - alert: vdu_down
2024-04-24T23:20:40.852954994Z     annotations:
2024-04-24T23:20:40.852958770Z       description: VDU {{ $labels.vm_id }} in VIM {{ $labels.vim_id }} has been down
2024-04-24T23:20:40.852962415Z         for more than 3 minutes. NS instance id is {{ $labels.ns_id }}
2024-04-24T23:20:40.852965336Z       summary: VDU {{ $labels.vm_id }} in VIM {{ $labels.vim_id }} is down
2024-04-24T23:20:40.852968820Z     expr: vm_status_extended != 1
2024-04-24T23:20:40.852971725Z     for: 3m
2024-04-24T23:20:40.852974473Z 
2024-04-24T23:20:40.853973122Z New config written, updating prometheus
2024-04-24T23:20:45.887647106Z Prometheus config update successful
2024-04-24T23:20:45.887680797Z 
2024-04-24T23:20:45.887693735Z Changes detected, updating prometheus config
2024-04-24T23:20:45.890365585Z Jobs detected: 0
2024-04-24T23:20:45.896615799Z Writing new config file to /etc/prometheus/prometheus.yml
2024-04-24T23:20:45.898694645Z alerting:
2024-04-24T23:20:45.898709969Z   alertmanagers:
2024-04-24T23:20:45.898715211Z   - static_configs:
2024-04-24T23:20:45.898719424Z     - targets:
2024-04-24T23:20:45.898736205Z       - alertmanager:9093
2024-04-24T23:20:45.898741314Z global:
2024-04-24T23:20:45.898745510Z   evaluation_interval: 15s
2024-04-24T23:20:45.898749540Z   scrape_interval: 15s
2024-04-24T23:20:45.898753552Z rule_files:
2024-04-24T23:20:45.898757685Z - osm_metric_rules.yml
2024-04-24T23:20:45.898761640Z - osm_alert_rules.yml
2024-04-24T23:20:45.898765693Z scrape_configs:
2024-04-24T23:20:45.898769694Z - job_name: mon_exporter
2024-04-24T23:20:45.898773695Z   static_configs:
2024-04-24T23:20:45.898777706Z   - targets:
2024-04-24T23:20:45.898781763Z     - mon:8000
2024-04-24T23:20:45.898785953Z - honor_labels: true
2024-04-24T23:20:45.898790102Z   job_name: pushgateway
2024-04-24T23:20:45.898794136Z   scrape_interval: 30s
2024-04-24T23:20:45.898798116Z   static_configs:
2024-04-24T23:20:45.898802022Z   - targets:
2024-04-24T23:20:45.898806884Z     - pushgateway-prometheus-pushgateway:9091
2024-04-24T23:20:45.898810764Z 
2024-04-24T23:20:45.902232884Z Alerts read: 0
2024-04-24T23:20:45.904222896Z Writing new alerts file to /etc/prometheus/osm_alert_rules.yml
2024-04-24T23:20:45.905343822Z groups:
2024-04-24T23:20:45.905357914Z - name: osm_alert_rules
2024-04-24T23:20:45.905377158Z   rules:
2024-04-24T23:20:45.905382738Z   - alert: vdu_down
2024-04-24T23:20:45.905412132Z     annotations:
2024-04-24T23:20:45.905417657Z       description: VDU {{ $labels.vm_id }} in VIM {{ $labels.vim_id }} has been down
2024-04-24T23:20:45.905422652Z         for more than 3 minutes. NS instance id is {{ $labels.ns_id }}
2024-04-24T23:20:45.905427412Z       summary: VDU {{ $labels.vm_id }} in VIM {{ $labels.vim_id }} is down
2024-04-24T23:20:45.905432479Z     expr: vm_status_extended != 1
2024-04-24T23:20:45.905436863Z     for: 3m
2024-04-24T23:20:45.905441009Z 
2024-04-24T23:20:45.906309707Z New config written, updating prometheus
2024-04-24T23:20:50.940147102Z Prometheus config update successful
2024-04-24T23:20:50.940343304Z 
2024-04-25T01:13:26.624734480Z Changes detected, updating prometheus config
2024-04-25T01:13:26.626898988Z Jobs detected: 0
2024-04-25T01:13:26.638842511Z Writing new config file to /etc/prometheus/prometheus.yml
2024-04-25T01:13:26.643097017Z alerting:
2024-04-25T01:13:26.643115189Z   alertmanagers:
2024-04-25T01:13:26.643124531Z   - static_configs:
2024-04-25T01:13:26.643129066Z     - targets:
2024-04-25T01:13:26.643134991Z       - alertmanager:9093
2024-04-25T01:13:26.643139861Z global:
2024-04-25T01:13:26.643144606Z   evaluation_interval: 15s
2024-04-25T01:13:26.643149211Z   scrape_interval: 15s
2024-04-25T01:13:26.643153608Z rule_files:
2024-04-25T01:13:26.643158128Z - osm_metric_rules.yml
2024-04-25T01:13:26.643163151Z - osm_alert_rules.yml
2024-04-25T01:13:26.643167738Z scrape_configs:
2024-04-25T01:13:26.643171983Z - job_name: mon_exporter
2024-04-25T01:13:26.643176034Z   static_configs:
2024-04-25T01:13:26.643180367Z   - targets:
2024-04-25T01:13:26.643185554Z     - mon:8000
2024-04-25T01:13:26.643189882Z - honor_labels: true
2024-04-25T01:13:26.643194683Z   job_name: pushgateway
2024-04-25T01:13:26.643199303Z   scrape_interval: 30s
2024-04-25T01:13:26.643204424Z   static_configs:
2024-04-25T01:13:26.643210374Z   - targets:
2024-04-25T01:13:26.643215160Z     - pushgateway-prometheus-pushgateway:9091
2024-04-25T01:13:26.643219356Z 
2024-04-25T01:13:26.649038341Z Alerts read: 0
2024-04-25T01:13:26.654280705Z Writing new alerts file to /etc/prometheus/osm_alert_rules.yml
2024-04-25T01:13:26.656389042Z groups:
2024-04-25T01:13:26.656405611Z - name: osm_alert_rules
2024-04-25T01:13:26.656410438Z   rules:
2024-04-25T01:13:26.656415257Z   - alert: vdu_down
2024-04-25T01:13:26.656419725Z     annotations:
2024-04-25T01:13:26.656425381Z       description: VDU {{ $labels.vm_id }} in VIM {{ $labels.vim_id }} has been down
2024-04-25T01:13:26.656430891Z         for more than 3 minutes. NS instance id is {{ $labels.ns_id }}
2024-04-25T01:13:26.656435172Z       summary: VDU {{ $labels.vm_id }} in VIM {{ $labels.vim_id }} is down
2024-04-25T01:13:26.656440143Z     expr: vm_status_extended != 1
2024-04-25T01:13:26.656444952Z     for: 3m
2024-04-25T01:13:26.656449459Z 
2024-04-25T01:13:26.658431321Z New config written, updating prometheus
2024-04-25T01:13:31.686277009Z Prometheus config update successful
2024-04-25T01:13:31.686315023Z 
2024-04-25T01:18:39.987556631Z Changes detected, updating prometheus config
2024-04-25T01:18:39.989179869Z Jobs detected: 0
2024-04-25T01:18:39.996488878Z Writing new config file to /etc/prometheus/prometheus.yml
2024-04-25T01:18:39.998650894Z alerting:
2024-04-25T01:18:39.998669178Z   alertmanagers:
2024-04-25T01:18:39.998672942Z   - static_configs:
2024-04-25T01:18:39.998676962Z     - targets:
2024-04-25T01:18:39.998680513Z       - alertmanager:9093
2024-04-25T01:18:39.998683467Z global:
2024-04-25T01:18:39.998686429Z   evaluation_interval: 15s
2024-04-25T01:18:39.998703604Z   scrape_interval: 15s
2024-04-25T01:18:39.998706743Z rule_files:
2024-04-25T01:18:39.998709644Z - osm_metric_rules.yml
2024-04-25T01:18:39.998712606Z - osm_alert_rules.yml
2024-04-25T01:18:39.998715768Z scrape_configs:
2024-04-25T01:18:39.998718651Z - job_name: mon_exporter
2024-04-25T01:18:39.998721601Z   static_configs:
2024-04-25T01:18:39.998724762Z   - targets:
2024-04-25T01:18:39.998727642Z     - mon:8000
2024-04-25T01:18:39.998730569Z - honor_labels: true
2024-04-25T01:18:39.998733655Z   job_name: pushgateway
2024-04-25T01:18:39.998736514Z   scrape_interval: 30s
2024-04-25T01:18:39.998739616Z   static_configs:
2024-04-25T01:18:39.998742464Z   - targets:
2024-04-25T01:18:39.998746113Z     - pushgateway-prometheus-pushgateway:9091
2024-04-25T01:18:39.998748957Z 
2024-04-25T01:18:40.006368877Z Alerts read: 0
2024-04-25T01:18:40.008137821Z Writing new alerts file to /etc/prometheus/osm_alert_rules.yml
2024-04-25T01:18:40.009234385Z groups:
2024-04-25T01:18:40.009245412Z - name: osm_alert_rules
2024-04-25T01:18:40.009249058Z   rules:
2024-04-25T01:18:40.009252065Z   - alert: vdu_down
2024-04-25T01:18:40.009255119Z     annotations:
2024-04-25T01:18:40.009258978Z       description: VDU {{ $labels.vm_id }} in VIM {{ $labels.vim_id }} has been down
2024-04-25T01:18:40.009262704Z         for more than 3 minutes. NS instance id is {{ $labels.ns_id }}
2024-04-25T01:18:40.009266200Z       summary: VDU {{ $labels.vm_id }} in VIM {{ $labels.vim_id }} is down
2024-04-25T01:18:40.009269884Z     expr: vm_status_extended != 1
2024-04-25T01:18:40.009272804Z     for: 3m
2024-04-25T01:18:40.009275535Z 
2024-04-25T01:18:40.010310910Z New config written, updating prometheus
2024-04-25T01:18:45.044355090Z Prometheus config update successful
2024-04-25T01:18:45.044393190Z 
2024-04-25T01:59:17.243834273Z Changes detected, updating prometheus config
2024-04-25T01:59:17.250080130Z Jobs detected: 0
2024-04-25T01:59:17.259562789Z Writing new config file to /etc/prometheus/prometheus.yml
2024-04-25T01:59:17.262354044Z alerting:
2024-04-25T01:59:17.262367289Z   alertmanagers:
2024-04-25T01:59:17.262371007Z   - static_configs:
2024-04-25T01:59:17.262374046Z     - targets:
2024-04-25T01:59:17.262377733Z       - alertmanager:9093
2024-04-25T01:59:17.262380697Z global:
2024-04-25T01:59:17.262383672Z   evaluation_interval: 15s
2024-04-25T01:59:17.262386694Z   scrape_interval: 15s
2024-04-25T01:59:17.262389861Z rule_files:
2024-04-25T01:59:17.262392760Z - osm_metric_rules.yml
2024-04-25T01:59:17.262395726Z - osm_alert_rules.yml
2024-04-25T01:59:17.262398686Z scrape_configs:
2024-04-25T01:59:17.262401591Z - job_name: mon_exporter
2024-04-25T01:59:17.262404515Z   static_configs:
2024-04-25T01:59:17.262407444Z   - targets:
2024-04-25T01:59:17.262410373Z     - mon:8000
2024-04-25T01:59:17.262413607Z - honor_labels: true
2024-04-25T01:59:17.262416499Z   job_name: pushgateway
2024-04-25T01:59:17.262419566Z   scrape_interval: 30s
2024-04-25T01:59:17.262422479Z   static_configs:
2024-04-25T01:59:17.262425316Z   - targets:
2024-04-25T01:59:17.262428862Z     - pushgateway-prometheus-pushgateway:9091
2024-04-25T01:59:17.262431682Z 
2024-04-25T01:59:17.267590994Z Alerts read: 2
2024-04-25T01:59:17.271379411Z Writing new alerts file to /etc/prometheus/osm_alert_rules.yml
2024-04-25T01:59:17.278300832Z groups:
2024-04-25T01:59:17.278316625Z - name: osm_alert_rules
2024-04-25T01:59:17.278320164Z   rules:
2024-04-25T01:59:17.278323347Z   - alert: vdu_down
2024-04-25T01:59:17.278339301Z     annotations:
2024-04-25T01:59:17.278343696Z       description: VDU {{ $labels.vm_id }} in VIM {{ $labels.vim_id }} has been down
2024-04-25T01:59:17.278347489Z         for more than 3 minutes. NS instance id is {{ $labels.ns_id }}
2024-04-25T01:59:17.278350600Z       summary: VDU {{ $labels.vm_id }} in VIM {{ $labels.vim_id }} is down
2024-04-25T01:59:17.278354251Z     expr: vm_status_extended != 1
2024-04-25T01:59:17.278357336Z     for: 3m
2024-04-25T01:59:17.278360565Z - name: _osm_alert_rules_20240425015917_
2024-04-25T01:59:17.278363564Z   rules:
2024-04-25T01:59:17.278367393Z   - alert: scalein_44569f19-e540-4d4f-934c-d0d79f11eeb3
2024-04-25T01:59:17.278370374Z     expr: (count (osm_cpu_utilization{ns_id="2ede8d1c-d187-4400-a3a2-f26c207eb5c7",
2024-04-25T01:59:17.278374385Z       vnf_member_index="vnf", vdu_id="hackfest_basic_metrics-VM"}) > 1) and (avg(osm_cpu_utilization{ns_id="2ede8d1c-d187-4400-a3a2-f26c207eb5c7",
2024-04-25T01:59:17.278377430Z       vnf_member_index="vnf", vdu_id="hackfest_basic_metrics-VM"}) < 10.0000000000)
2024-04-25T01:59:17.278380452Z     for: 1m
2024-04-25T01:59:17.278383435Z     labels:
2024-04-25T01:59:17.278386410Z       ns_id: 2ede8d1c-d187-4400-a3a2-f26c207eb5c7
2024-04-25T01:59:17.278389342Z       vdu_id: hackfest_basic_metrics-VM
2024-04-25T01:59:17.278392937Z       vnf_member_index: vnf
2024-04-25T01:59:17.278395894Z   - alert: scaleout_cf3fbb0c-196f-4006-9e2a-5900ce6cad9f
2024-04-25T01:59:17.278399005Z     expr: (count (osm_cpu_utilization{ns_id="2ede8d1c-d187-4400-a3a2-f26c207eb5c7",
2024-04-25T01:59:17.278402093Z       vnf_member_index="vnf", vdu_id="hackfest_basic_metrics-VM"}) < 2) and (avg(osm_cpu_utilization{ns_id="2ede8d1c-d187-4400-a3a2-f26c207eb5c7",
2024-04-25T01:59:17.278405083Z       vnf_member_index="vnf", vdu_id="hackfest_basic_metrics-VM"}) > 60.0000000000)
2024-04-25T01:59:17.278408060Z     for: 1m
2024-04-25T01:59:17.278411354Z     labels:
2024-04-25T01:59:17.278414331Z       ns_id: 2ede8d1c-d187-4400-a3a2-f26c207eb5c7
2024-04-25T01:59:17.278417354Z       vdu_id: hackfest_basic_metrics-VM
2024-04-25T01:59:17.278420346Z       vnf_member_index: vnf
2024-04-25T01:59:17.278423145Z 
2024-04-25T01:59:17.282905893Z New config written, updating prometheus
2024-04-25T01:59:22.316605765Z Prometheus config update successful
2024-04-25T01:59:22.316642732Z 
2024-04-25T01:59:22.316649498Z Changes detected, updating prometheus config
2024-04-25T01:59:22.318169127Z Jobs detected: 0
2024-04-25T01:59:22.337396732Z Writing new config file to /etc/prometheus/prometheus.yml
2024-04-25T01:59:22.340171225Z alerting:
2024-04-25T01:59:22.340196487Z   alertmanagers:
2024-04-25T01:59:22.340202787Z   - static_configs:
2024-04-25T01:59:22.340207509Z     - targets:
2024-04-25T01:59:22.340212580Z       - alertmanager:9093
2024-04-25T01:59:22.340217105Z global:
2024-04-25T01:59:22.340221622Z   evaluation_interval: 15s
2024-04-25T01:59:22.340226358Z   scrape_interval: 15s
2024-04-25T01:59:22.340230696Z rule_files:
2024-04-25T01:59:22.340235030Z - osm_metric_rules.yml
2024-04-25T01:59:22.340239330Z - osm_alert_rules.yml
2024-04-25T01:59:22.340243759Z scrape_configs:
2024-04-25T01:59:22.340248108Z - job_name: mon_exporter
2024-04-25T01:59:22.340252419Z   static_configs:
2024-04-25T01:59:22.340256816Z   - targets:
2024-04-25T01:59:22.340261159Z     - mon:8000
2024-04-25T01:59:22.340265533Z - honor_labels: true
2024-04-25T01:59:22.340269830Z   job_name: pushgateway
2024-04-25T01:59:22.340274198Z   scrape_interval: 30s
2024-04-25T01:59:22.340278399Z   static_configs:
2024-04-25T01:59:22.340300580Z   - targets:
2024-04-25T01:59:22.340306267Z     - pushgateway-prometheus-pushgateway:9091
2024-04-25T01:59:22.340310831Z 
2024-04-25T01:59:22.347128323Z Alerts read: 2
2024-04-25T01:59:22.349425366Z Writing new alerts file to /etc/prometheus/osm_alert_rules.yml
2024-04-25T01:59:22.353004479Z groups:
2024-04-25T01:59:22.353018346Z - name: osm_alert_rules
2024-04-25T01:59:22.353021844Z   rules:
2024-04-25T01:59:22.353024967Z   - alert: vdu_down
2024-04-25T01:59:22.353028034Z     annotations:
2024-04-25T01:59:22.353031707Z       description: VDU {{ $labels.vm_id }} in VIM {{ $labels.vim_id }} has been down
2024-04-25T01:59:22.353036081Z         for more than 3 minutes. NS instance id is {{ $labels.ns_id }}
2024-04-25T01:59:22.353039203Z       summary: VDU {{ $labels.vm_id }} in VIM {{ $labels.vim_id }} is down
2024-04-25T01:59:22.353042741Z     expr: vm_status_extended != 1
2024-04-25T01:59:22.353046179Z     for: 3m
2024-04-25T01:59:22.353049169Z - name: _osm_alert_rules_20240425015922_
2024-04-25T01:59:22.353052173Z   rules:
2024-04-25T01:59:22.353055738Z   - alert: scalein_44569f19-e540-4d4f-934c-d0d79f11eeb3
2024-04-25T01:59:22.353058781Z     expr: (count (osm_cpu_utilization{ns_id="2ede8d1c-d187-4400-a3a2-f26c207eb5c7",
2024-04-25T01:59:22.353064826Z       vnf_member_index="vnf", vdu_id="hackfest_basic_metrics-VM"}) > 1) and (avg(osm_cpu_utilization{ns_id="2ede8d1c-d187-4400-a3a2-f26c207eb5c7",
2024-04-25T01:59:22.353067955Z       vnf_member_index="vnf", vdu_id="hackfest_basic_metrics-VM"}) < 10.0000000000)
2024-04-25T01:59:22.353070966Z     for: 1m
2024-04-25T01:59:22.353073948Z     labels:
2024-04-25T01:59:22.353076904Z       ns_id: 2ede8d1c-d187-4400-a3a2-f26c207eb5c7
2024-04-25T01:59:22.353080165Z       vdu_id: hackfest_basic_metrics-VM
2024-04-25T01:59:22.353084302Z       vnf_member_index: vnf
2024-04-25T01:59:22.353087255Z   - alert: scaleout_cf3fbb0c-196f-4006-9e2a-5900ce6cad9f
2024-04-25T01:59:22.353090750Z     expr: (count (osm_cpu_utilization{ns_id="2ede8d1c-d187-4400-a3a2-f26c207eb5c7",
2024-04-25T01:59:22.353094164Z       vnf_member_index="vnf", vdu_id="hackfest_basic_metrics-VM"}) < 2) and (avg(osm_cpu_utilization{ns_id="2ede8d1c-d187-4400-a3a2-f26c207eb5c7",
2024-04-25T01:59:22.353097320Z       vnf_member_index="vnf", vdu_id="hackfest_basic_metrics-VM"}) > 60.0000000000)
2024-04-25T01:59:22.353100313Z     for: 1m
2024-04-25T01:59:22.353103202Z     labels:
2024-04-25T01:59:22.353106139Z       ns_id: 2ede8d1c-d187-4400-a3a2-f26c207eb5c7
2024-04-25T01:59:22.353109077Z       vdu_id: hackfest_basic_metrics-VM
2024-04-25T01:59:22.353112127Z       vnf_member_index: vnf
2024-04-25T01:59:22.353114932Z 
2024-04-25T01:59:22.357471200Z New config written, updating prometheus
2024-04-25T01:59:27.391341117Z Prometheus config update successful
2024-04-25T01:59:27.391396761Z 
2024-04-25T02:06:41.377838965Z Changes detected, updating prometheus config
2024-04-25T02:06:41.379754751Z Jobs detected: 0
2024-04-25T02:06:41.388295054Z Writing new config file to /etc/prometheus/prometheus.yml
2024-04-25T02:06:41.391182483Z alerting:
2024-04-25T02:06:41.391219549Z   alertmanagers:
2024-04-25T02:06:41.391229825Z   - static_configs:
2024-04-25T02:06:41.391238767Z     - targets:
2024-04-25T02:06:41.391247878Z       - alertmanager:9093
2024-04-25T02:06:41.391256172Z global:
2024-04-25T02:06:41.391264702Z   evaluation_interval: 15s
2024-04-25T02:06:41.391272979Z   scrape_interval: 15s
2024-04-25T02:06:41.391281279Z rule_files:
2024-04-25T02:06:41.391289584Z - osm_metric_rules.yml
2024-04-25T02:06:41.391298020Z - osm_alert_rules.yml
2024-04-25T02:06:41.391306631Z scrape_configs:
2024-04-25T02:06:41.391342860Z - job_name: mon_exporter
2024-04-25T02:06:41.391351524Z   static_configs:
2024-04-25T02:06:41.391359932Z   - targets:
2024-04-25T02:06:41.391368132Z     - mon:8000
2024-04-25T02:06:41.391376271Z - honor_labels: true
2024-04-25T02:06:41.391384443Z   job_name: pushgateway
2024-04-25T02:06:41.391392647Z   scrape_interval: 30s
2024-04-25T02:06:41.391400772Z   static_configs:
2024-04-25T02:06:41.391409001Z   - targets:
2024-04-25T02:06:41.391418336Z     - pushgateway-prometheus-pushgateway:9091
2024-04-25T02:06:41.391426345Z 
2024-04-25T02:06:41.394625738Z Alerts read: 0
2024-04-25T02:06:41.396759948Z Writing new alerts file to /etc/prometheus/osm_alert_rules.yml
2024-04-25T02:06:41.398123847Z groups:
2024-04-25T02:06:41.398146900Z - name: osm_alert_rules
2024-04-25T02:06:41.398155785Z   rules:
2024-04-25T02:06:41.398164272Z   - alert: vdu_down
2024-04-25T02:06:41.398172532Z     annotations:
2024-04-25T02:06:41.398182036Z       description: VDU {{ $labels.vm_id }} in VIM {{ $labels.vim_id }} has been down
2024-04-25T02:06:41.398191545Z         for more than 3 minutes. NS instance id is {{ $labels.ns_id }}
2024-04-25T02:06:41.398200311Z       summary: VDU {{ $labels.vm_id }} in VIM {{ $labels.vim_id }} is down
2024-04-25T02:06:41.398209432Z     expr: vm_status_extended != 1
2024-04-25T02:06:41.398217785Z     for: 3m
2024-04-25T02:06:41.398225596Z 
2024-04-25T02:06:41.399519258Z New config written, updating prometheus
2024-04-25T02:06:46.436891801Z Prometheus config update successful
2024-04-25T02:06:46.436923386Z 
2024-04-25T02:06:46.436927075Z Changes detected, updating prometheus config
2024-04-25T02:06:46.438643485Z Jobs detected: 0
2024-04-25T02:06:46.444752241Z Writing new config file to /etc/prometheus/prometheus.yml
2024-04-25T02:06:46.447336664Z alerting:
2024-04-25T02:06:46.447360654Z   alertmanagers:
2024-04-25T02:06:46.447368076Z   - static_configs:
2024-04-25T02:06:46.447373993Z     - targets:
2024-04-25T02:06:46.447383123Z       - alertmanager:9093
2024-04-25T02:06:46.447389012Z global:
2024-04-25T02:06:46.447394835Z   evaluation_interval: 15s
2024-04-25T02:06:46.447400386Z   scrape_interval: 15s
2024-04-25T02:06:46.447406093Z rule_files:
2024-04-25T02:06:46.447411660Z - osm_metric_rules.yml
2024-04-25T02:06:46.447417317Z - osm_alert_rules.yml
2024-04-25T02:06:46.447423344Z scrape_configs:
2024-04-25T02:06:46.447428931Z - job_name: mon_exporter
2024-04-25T02:06:46.447434389Z   static_configs:
2024-04-25T02:06:46.447440029Z   - targets:
2024-04-25T02:06:46.447445626Z     - mon:8000
2024-04-25T02:06:46.447451370Z - honor_labels: true
2024-04-25T02:06:46.447456921Z   job_name: pushgateway
2024-04-25T02:06:46.447462479Z   scrape_interval: 30s
2024-04-25T02:06:46.447467993Z   static_configs:
2024-04-25T02:06:46.447473631Z   - targets:
2024-04-25T02:06:46.447480017Z     - pushgateway-prometheus-pushgateway:9091
2024-04-25T02:06:46.447485456Z 
2024-04-25T02:06:46.450537496Z Alerts read: 0
2024-04-25T02:06:46.452248980Z Writing new alerts file to /etc/prometheus/osm_alert_rules.yml
2024-04-25T02:06:46.453326185Z groups:
2024-04-25T02:06:46.453343577Z - name: osm_alert_rules
2024-04-25T02:06:46.453348819Z   rules:
2024-04-25T02:06:46.453353727Z   - alert: vdu_down
2024-04-25T02:06:46.453359100Z     annotations:
2024-04-25T02:06:46.453372952Z       description: VDU {{ $labels.vm_id }} in VIM {{ $labels.vim_id }} has been down
2024-04-25T02:06:46.453381122Z         for more than 3 minutes. NS instance id is {{ $labels.ns_id }}
2024-04-25T02:06:46.453400238Z       summary: VDU {{ $labels.vm_id }} in VIM {{ $labels.vim_id }} is down
2024-04-25T02:06:46.453404865Z     expr: vm_status_extended != 1
2024-04-25T02:06:46.453407940Z     for: 3m
2024-04-25T02:06:46.453410723Z 
2024-04-25T02:06:46.454545803Z New config written, updating prometheus
2024-04-25T02:06:51.478817135Z Prometheus config update successful
2024-04-25T02:06:51.478858729Z 
2024-04-25T02:07:32.165386654Z Changes detected, updating prometheus config
2024-04-25T02:07:32.166781223Z Jobs detected: 0
2024-04-25T02:07:32.175889531Z Writing new config file to /etc/prometheus/prometheus.yml
2024-04-25T02:07:32.179207704Z alerting:
2024-04-25T02:07:32.179258489Z   alertmanagers:
2024-04-25T02:07:32.179269830Z   - static_configs:
2024-04-25T02:07:32.179277642Z     - targets:
2024-04-25T02:07:32.179286198Z       - alertmanager:9093
2024-04-25T02:07:32.179294500Z global:
2024-04-25T02:07:32.179302421Z   evaluation_interval: 15s
2024-04-25T02:07:32.179310001Z   scrape_interval: 15s
2024-04-25T02:07:32.179317761Z rule_files:
2024-04-25T02:07:32.179325464Z - osm_metric_rules.yml
2024-04-25T02:07:32.179333005Z - osm_alert_rules.yml
2024-04-25T02:07:32.179340595Z scrape_configs:
2024-04-25T02:07:32.179348167Z - job_name: mon_exporter
2024-04-25T02:07:32.179355810Z   static_configs:
2024-04-25T02:07:32.179363482Z   - targets:
2024-04-25T02:07:32.179371112Z     - mon:8000
2024-04-25T02:07:32.179379488Z - honor_labels: true
2024-04-25T02:07:32.179387088Z   job_name: pushgateway
2024-04-25T02:07:32.179394661Z   scrape_interval: 30s
2024-04-25T02:07:32.179402172Z   static_configs:
2024-04-25T02:07:32.179409870Z   - targets:
2024-04-25T02:07:32.179418533Z     - pushgateway-prometheus-pushgateway:9091
2024-04-25T02:07:32.179425939Z 
2024-04-25T02:07:32.184671970Z Alerts read: 2
2024-04-25T02:07:32.187958477Z Writing new alerts file to /etc/prometheus/osm_alert_rules.yml
2024-04-25T02:07:32.194270746Z groups:
2024-04-25T02:07:32.194302719Z - name: osm_alert_rules
2024-04-25T02:07:32.194310697Z   rules:
2024-04-25T02:07:32.194317944Z   - alert: vdu_down
2024-04-25T02:07:32.194324947Z     annotations:
2024-04-25T02:07:32.194332768Z       description: VDU {{ $labels.vm_id }} in VIM {{ $labels.vim_id }} has been down
2024-04-25T02:07:32.194340532Z         for more than 3 minutes. NS instance id is {{ $labels.ns_id }}
2024-04-25T02:07:32.194347373Z       summary: VDU {{ $labels.vm_id }} in VIM {{ $labels.vim_id }} is down
2024-04-25T02:07:32.194354983Z     expr: vm_status_extended != 1
2024-04-25T02:07:32.194362093Z     for: 3m
2024-04-25T02:07:32.194369148Z - name: _osm_alert_rules_20240425020732_
2024-04-25T02:07:32.194376000Z   rules:
2024-04-25T02:07:32.194384044Z   - alert: scalein_d7bdb31a-e292-44e1-8479-c664b400f26b
2024-04-25T02:07:32.194391184Z     expr: (count (osm_cpu_utilization{ns_id="a2856130-a67d-4a8f-9ac2-0157054182ba",
2024-04-25T02:07:32.194399017Z       vnf_member_index="vnf", vdu_id="hackfest_basic_metrics-VM"}) > 1) and (avg(osm_cpu_utilization{ns_id="a2856130-a67d-4a8f-9ac2-0157054182ba",
2024-04-25T02:07:32.194406644Z       vnf_member_index="vnf", vdu_id="hackfest_basic_metrics-VM"}) < 10.0000000000)
2024-04-25T02:07:32.194413659Z     for: 1m
2024-04-25T02:07:32.194420619Z     labels:
2024-04-25T02:07:32.194427664Z       ns_id: a2856130-a67d-4a8f-9ac2-0157054182ba
2024-04-25T02:07:32.194434658Z       vdu_id: hackfest_basic_metrics-VM
2024-04-25T02:07:32.194442193Z       vnf_member_index: vnf
2024-04-25T02:07:32.194449123Z   - alert: scaleout_155bc1d2-e136-475b-bae8-9f688ca669d7
2024-04-25T02:07:32.194481950Z     expr: (count (osm_cpu_utilization{ns_id="a2856130-a67d-4a8f-9ac2-0157054182ba",
2024-04-25T02:07:32.194489724Z       vnf_member_index="vnf", vdu_id="hackfest_basic_metrics-VM"}) < 2) and (avg(osm_cpu_utilization{ns_id="a2856130-a67d-4a8f-9ac2-0157054182ba",
2024-04-25T02:07:32.194497255Z       vnf_member_index="vnf", vdu_id="hackfest_basic_metrics-VM"}) > 60.0000000000)
2024-04-25T02:07:32.194504311Z     for: 1m
2024-04-25T02:07:32.194511195Z     labels:
2024-04-25T02:07:32.194518063Z       ns_id: a2856130-a67d-4a8f-9ac2-0157054182ba
2024-04-25T02:07:32.194524978Z       vdu_id: hackfest_basic_metrics-VM
2024-04-25T02:07:32.194532268Z       vnf_member_index: vnf
2024-04-25T02:07:32.194538875Z 
2024-04-25T02:07:32.201573082Z New config written, updating prometheus
2024-04-25T02:07:37.228249315Z Prometheus config update successful
2024-04-25T02:07:37.228308327Z 
2024-04-25T02:07:37.228323016Z Changes detected, updating prometheus config
2024-04-25T02:07:37.229458636Z Jobs detected: 0
2024-04-25T02:07:37.234332308Z Writing new config file to /etc/prometheus/prometheus.yml
2024-04-25T02:07:37.236165886Z alerting:
2024-04-25T02:07:37.236179991Z   alertmanagers:
2024-04-25T02:07:37.236183889Z   - static_configs:
2024-04-25T02:07:37.236187065Z     - targets:
2024-04-25T02:07:37.236193104Z       - alertmanager:9093
2024-04-25T02:07:37.236196171Z global:
2024-04-25T02:07:37.236199245Z   evaluation_interval: 15s
2024-04-25T02:07:37.236202482Z   scrape_interval: 15s
2024-04-25T02:07:37.236205517Z rule_files:
2024-04-25T02:07:37.236208545Z - osm_metric_rules.yml
2024-04-25T02:07:37.236211567Z - osm_alert_rules.yml
2024-04-25T02:07:37.236214599Z scrape_configs:
2024-04-25T02:07:37.236217553Z - job_name: mon_exporter
2024-04-25T02:07:37.236220657Z   static_configs:
2024-04-25T02:07:37.236223746Z   - targets:
2024-04-25T02:07:37.236226798Z     - mon:8000
2024-04-25T02:07:37.236229736Z - honor_labels: true
2024-04-25T02:07:37.236232836Z   job_name: pushgateway
2024-04-25T02:07:37.236235801Z   scrape_interval: 30s
2024-04-25T02:07:37.236238836Z   static_configs:
2024-04-25T02:07:37.236241854Z   - targets:
2024-04-25T02:07:37.236245815Z     - pushgateway-prometheus-pushgateway:9091
2024-04-25T02:07:37.236248696Z 
2024-04-25T02:07:37.239706902Z Alerts read: 2
2024-04-25T02:07:37.241500311Z Writing new alerts file to /etc/prometheus/osm_alert_rules.yml
2024-04-25T02:07:37.245001818Z groups:
2024-04-25T02:07:37.245039439Z - name: osm_alert_rules
2024-04-25T02:07:37.245044026Z   rules:
2024-04-25T02:07:37.245047178Z   - alert: vdu_down
2024-04-25T02:07:37.245050177Z     annotations:
2024-04-25T02:07:37.245053988Z       description: VDU {{ $labels.vm_id }} in VIM {{ $labels.vim_id }} has been down
2024-04-25T02:07:37.245059779Z         for more than 3 minutes. NS instance id is {{ $labels.ns_id }}
2024-04-25T02:07:37.245062820Z       summary: VDU {{ $labels.vm_id }} in VIM {{ $labels.vim_id }} is down
2024-04-25T02:07:37.245066506Z     expr: vm_status_extended != 1
2024-04-25T02:07:37.245069597Z     for: 3m
2024-04-25T02:07:37.245072620Z - name: _osm_alert_rules_20240425020737_
2024-04-25T02:07:37.245075724Z   rules:
2024-04-25T02:07:37.245080743Z   - alert: scalein_d7bdb31a-e292-44e1-8479-c664b400f26b
2024-04-25T02:07:37.245083797Z     expr: (count (osm_cpu_utilization{ns_id="a2856130-a67d-4a8f-9ac2-0157054182ba",
2024-04-25T02:07:37.245103468Z       vnf_member_index="vnf", vdu_id="hackfest_basic_metrics-VM"}) > 1) and (avg(osm_cpu_utilization{ns_id="a2856130-a67d-4a8f-9ac2-0157054182ba",
2024-04-25T02:07:37.245106853Z       vnf_member_index="vnf", vdu_id="hackfest_basic_metrics-VM"}) < 10.0000000000)
2024-04-25T02:07:37.245120717Z     for: 1m
2024-04-25T02:07:37.245123652Z     labels:
2024-04-25T02:07:37.245126682Z       ns_id: a2856130-a67d-4a8f-9ac2-0157054182ba
2024-04-25T02:07:37.245129643Z       vdu_id: hackfest_basic_metrics-VM
2024-04-25T02:07:37.245133335Z       vnf_member_index: vnf
2024-04-25T02:07:37.245136347Z   - alert: scaleout_155bc1d2-e136-475b-bae8-9f688ca669d7
2024-04-25T02:07:37.245139413Z     expr: (count (osm_cpu_utilization{ns_id="a2856130-a67d-4a8f-9ac2-0157054182ba",
2024-04-25T02:07:37.245142390Z       vnf_member_index="vnf", vdu_id="hackfest_basic_metrics-VM"}) < 2) and (avg(osm_cpu_utilization{ns_id="a2856130-a67d-4a8f-9ac2-0157054182ba",
2024-04-25T02:07:37.245145334Z       vnf_member_index="vnf", vdu_id="hackfest_basic_metrics-VM"}) > 60.0000000000)
2024-04-25T02:07:37.245148207Z     for: 1m
2024-04-25T02:07:37.245151115Z     labels:
2024-04-25T02:07:37.245154096Z       ns_id: a2856130-a67d-4a8f-9ac2-0157054182ba
2024-04-25T02:07:37.245157070Z       vdu_id: hackfest_basic_metrics-VM
2024-04-25T02:07:37.245159989Z       vnf_member_index: vnf
2024-04-25T02:07:37.245162680Z 
2024-04-25T02:07:37.248018732Z New config written, updating prometheus
2024-04-25T02:07:42.279383208Z Prometheus config update successful
2024-04-25T02:07:42.279420719Z 
2024-04-25T02:25:04.033791733Z Changes detected, updating prometheus config
2024-04-25T02:25:04.036216723Z Jobs detected: 0
2024-04-25T02:25:04.042925151Z Writing new config file to /etc/prometheus/prometheus.yml
2024-04-25T02:25:04.045346818Z alerting:
2024-04-25T02:25:04.045367599Z   alertmanagers:
2024-04-25T02:25:04.045371827Z   - static_configs:
2024-04-25T02:25:04.045375518Z     - targets:
2024-04-25T02:25:04.045379390Z       - alertmanager:9093
2024-04-25T02:25:04.045382702Z global:
2024-04-25T02:25:04.045385981Z   evaluation_interval: 15s
2024-04-25T02:25:04.045389370Z   scrape_interval: 15s
2024-04-25T02:25:04.045392721Z rule_files:
2024-04-25T02:25:04.045396429Z - osm_metric_rules.yml
2024-04-25T02:25:04.045399843Z - osm_alert_rules.yml
2024-04-25T02:25:04.045403131Z scrape_configs:
2024-04-25T02:25:04.045406583Z - job_name: mon_exporter
2024-04-25T02:25:04.045409913Z   static_configs:
2024-04-25T02:25:04.045413256Z   - targets:
2024-04-25T02:25:04.045416485Z     - mon:8000
2024-04-25T02:25:04.045419713Z - honor_labels: true
2024-04-25T02:25:04.045422972Z   job_name: pushgateway
2024-04-25T02:25:04.045426175Z   scrape_interval: 30s
2024-04-25T02:25:04.045429493Z   static_configs:
2024-04-25T02:25:04.045432746Z   - targets:
2024-04-25T02:25:04.045436669Z     - pushgateway-prometheus-pushgateway:9091
2024-04-25T02:25:04.045439900Z 
2024-04-25T02:25:04.052257652Z Alerts read: 0
2024-04-25T02:25:04.055140957Z Writing new alerts file to /etc/prometheus/osm_alert_rules.yml
2024-04-25T02:25:04.056530395Z groups:
2024-04-25T02:25:04.056547465Z - name: osm_alert_rules
2024-04-25T02:25:04.056551224Z   rules:
2024-04-25T02:25:04.056554384Z   - alert: vdu_down
2024-04-25T02:25:04.056557871Z     annotations:
2024-04-25T02:25:04.056561728Z       description: VDU {{ $labels.vm_id }} in VIM {{ $labels.vim_id }} has been down
2024-04-25T02:25:04.056568107Z         for more than 3 minutes. NS instance id is {{ $labels.ns_id }}
2024-04-25T02:25:04.056571519Z       summary: VDU {{ $labels.vm_id }} in VIM {{ $labels.vim_id }} is down
2024-04-25T02:25:04.056575780Z     expr: vm_status_extended != 1
2024-04-25T02:25:04.056579045Z     for: 3m
2024-04-25T02:25:04.056595791Z 
2024-04-25T02:25:04.057914263Z New config written, updating prometheus
2024-04-25T02:25:09.091248845Z Prometheus config update successful
2024-04-25T02:25:09.091302090Z 
2024-04-25T02:25:09.091424888Z Changes detected, updating prometheus config
2024-04-25T02:25:09.093258794Z Jobs detected: 0
2024-04-25T02:25:09.098528375Z Writing new config file to /etc/prometheus/prometheus.yml
2024-04-25T02:25:09.100415788Z alerting:
2024-04-25T02:25:09.100428569Z   alertmanagers:
2024-04-25T02:25:09.100432585Z   - static_configs:
2024-04-25T02:25:09.100435846Z     - targets:
2024-04-25T02:25:09.100439750Z       - alertmanager:9093
2024-04-25T02:25:09.100442962Z global:
2024-04-25T02:25:09.100446243Z   evaluation_interval: 15s
2024-04-25T02:25:09.100449354Z   scrape_interval: 15s
2024-04-25T02:25:09.100452513Z rule_files:
2024-04-25T02:25:09.100455576Z - osm_metric_rules.yml
2024-04-25T02:25:09.100458639Z - osm_alert_rules.yml
2024-04-25T02:25:09.100461732Z scrape_configs:
2024-04-25T02:25:09.100464755Z - job_name: mon_exporter
2024-04-25T02:25:09.100467926Z   static_configs:
2024-04-25T02:25:09.100471222Z   - targets:
2024-04-25T02:25:09.100474268Z     - mon:8000
2024-04-25T02:25:09.100477300Z - honor_labels: true
2024-04-25T02:25:09.100480330Z   job_name: pushgateway
2024-04-25T02:25:09.100483399Z   scrape_interval: 30s
2024-04-25T02:25:09.100486417Z   static_configs:
2024-04-25T02:25:09.100489393Z   - targets:
2024-04-25T02:25:09.100493058Z     - pushgateway-prometheus-pushgateway:9091
2024-04-25T02:25:09.100507583Z 
2024-04-25T02:25:09.103657668Z Alerts read: 0
2024-04-25T02:25:09.105583815Z Writing new alerts file to /etc/prometheus/osm_alert_rules.yml
2024-04-25T02:25:09.106751691Z groups:
2024-04-25T02:25:09.106782733Z - name: osm_alert_rules
2024-04-25T02:25:09.106786253Z   rules:
2024-04-25T02:25:09.106789350Z   - alert: vdu_down
2024-04-25T02:25:09.106792590Z     annotations:
2024-04-25T02:25:09.106796338Z       description: VDU {{ $labels.vm_id }} in VIM {{ $labels.vim_id }} has been down
2024-04-25T02:25:09.106800110Z         for more than 3 minutes. NS instance id is {{ $labels.ns_id }}
2024-04-25T02:25:09.106803235Z       summary: VDU {{ $labels.vm_id }} in VIM {{ $labels.vim_id }} is down
2024-04-25T02:25:09.106806874Z     expr: vm_status_extended != 1
2024-04-25T02:25:09.106809948Z     for: 3m
2024-04-25T02:25:09.106817519Z 
2024-04-25T02:25:09.107843300Z New config written, updating prometheus
2024-04-25T02:25:14.143459752Z Prometheus config update successful
2024-04-25T02:25:14.143515271Z 
2024-04-25T02:27:19.462044520Z Changes detected, updating prometheus config
2024-04-25T02:27:19.463615359Z Jobs detected: 0
2024-04-25T02:27:19.471491411Z Writing new config file to /etc/prometheus/prometheus.yml
2024-04-25T02:27:19.474378606Z alerting:
2024-04-25T02:27:19.474392795Z   alertmanagers:
2024-04-25T02:27:19.474397246Z   - static_configs:
2024-04-25T02:27:19.474401874Z     - targets:
2024-04-25T02:27:19.474406536Z       - alertmanager:9093
2024-04-25T02:27:19.474415921Z global:
2024-04-25T02:27:19.474420541Z   evaluation_interval: 15s
2024-04-25T02:27:19.474424994Z   scrape_interval: 15s
2024-04-25T02:27:19.474429472Z rule_files:
2024-04-25T02:27:19.474433930Z - osm_metric_rules.yml
2024-04-25T02:27:19.474438165Z - osm_alert_rules.yml
2024-04-25T02:27:19.474442780Z scrape_configs:
2024-04-25T02:27:19.474446396Z - job_name: mon_exporter
2024-04-25T02:27:19.474449367Z   static_configs:
2024-04-25T02:27:19.474452320Z   - targets:
2024-04-25T02:27:19.474455295Z     - mon:8000
2024-04-25T02:27:19.474458241Z - honor_labels: true
2024-04-25T02:27:19.474475867Z   job_name: pushgateway
2024-04-25T02:27:19.474478847Z   scrape_interval: 30s
2024-04-25T02:27:19.474481779Z   static_configs:
2024-04-25T02:27:19.474484591Z   - targets:
2024-04-25T02:27:19.474488369Z     - pushgateway-prometheus-pushgateway:9091
2024-04-25T02:27:19.474491193Z 
2024-04-25T02:27:19.478144203Z Alerts read: 1
2024-04-25T02:27:19.480094651Z Writing new alerts file to /etc/prometheus/osm_alert_rules.yml
2024-04-25T02:27:19.482323061Z groups:
2024-04-25T02:27:19.482335232Z - name: osm_alert_rules
2024-04-25T02:27:19.482339151Z   rules:
2024-04-25T02:27:19.482342913Z   - alert: vdu_down
2024-04-25T02:27:19.482346663Z     annotations:
2024-04-25T02:27:19.482351235Z       description: VDU {{ $labels.vm_id }} in VIM {{ $labels.vim_id }} has been down
2024-04-25T02:27:19.482355605Z         for more than 3 minutes. NS instance id is {{ $labels.ns_id }}
2024-04-25T02:27:19.482359181Z       summary: VDU {{ $labels.vm_id }} in VIM {{ $labels.vim_id }} is down
2024-04-25T02:27:19.482363565Z     expr: vm_status_extended != 1
2024-04-25T02:27:19.482367108Z     for: 3m
2024-04-25T02:27:19.482370674Z - name: _osm_alert_rules_20240425022719_
2024-04-25T02:27:19.482374304Z   rules:
2024-04-25T02:27:19.482382452Z   - alert: vdu_alarm_0fb6ec37-3d55-4f35-8db2-b50ddd3d3cc6
2024-04-25T02:27:19.482386673Z     expr: osm_cpu_utilization{ns_id="e48066f2-bd8c-4e2b-b8fc-8a1b13911af2", vnf_member_index="vnf",
2024-04-25T02:27:19.482390285Z       vdu_id="cirros_vnfd-VM"} < 20.0000
2024-04-25T02:27:19.482393886Z     for: 1m
2024-04-25T02:27:19.482397537Z     labels:
2024-04-25T02:27:19.482401444Z       ns_id: e48066f2-bd8c-4e2b-b8fc-8a1b13911af2
2024-04-25T02:27:19.482405893Z       vdu_id: cirros_vnfd-VM
2024-04-25T02:27:19.482409444Z       vdu_name: '{{ $labels.vdu_name }}'
2024-04-25T02:27:19.482413092Z       vnf_member_index: vnf
2024-04-25T02:27:19.482416402Z 
2024-04-25T02:27:19.484506907Z New config written, updating prometheus
2024-04-25T02:27:24.517819686Z Prometheus config update successful
2024-04-25T02:27:24.517861820Z 
2024-04-25T02:37:09.670483489Z Changes detected, updating prometheus config
2024-04-25T02:37:09.672959843Z Jobs detected: 0
2024-04-25T02:37:09.678357151Z Writing new config file to /etc/prometheus/prometheus.yml
2024-04-25T02:37:09.680342059Z alerting:
2024-04-25T02:37:09.680380884Z   alertmanagers:
2024-04-25T02:37:09.680392825Z   - static_configs:
2024-04-25T02:37:09.680400857Z     - targets:
2024-04-25T02:37:09.680408879Z       - alertmanager:9093
2024-04-25T02:37:09.680417021Z global:
2024-04-25T02:37:09.680424352Z   evaluation_interval: 15s
2024-04-25T02:37:09.680431653Z   scrape_interval: 15s
2024-04-25T02:37:09.680439052Z rule_files:
2024-04-25T02:37:09.680446153Z - osm_metric_rules.yml
2024-04-25T02:37:09.680453078Z - osm_alert_rules.yml
2024-04-25T02:37:09.680460573Z scrape_configs:
2024-04-25T02:37:09.680467792Z - job_name: mon_exporter
2024-04-25T02:37:09.680475465Z   static_configs:
2024-04-25T02:37:09.680483864Z   - targets:
2024-04-25T02:37:09.680492641Z     - mon:8000
2024-04-25T02:37:09.680508795Z - honor_labels: true
2024-04-25T02:37:09.680518842Z   job_name: pushgateway
2024-04-25T02:37:09.680528326Z   scrape_interval: 30s
2024-04-25T02:37:09.680535471Z   static_configs:
2024-04-25T02:37:09.680543995Z   - targets:
2024-04-25T02:37:09.680553518Z     - pushgateway-prometheus-pushgateway:9091
2024-04-25T02:37:09.680560434Z 
2024-04-25T02:37:09.683244898Z Alerts read: 0
2024-04-25T02:37:09.685305349Z Writing new alerts file to /etc/prometheus/osm_alert_rules.yml
2024-04-25T02:37:09.686813095Z groups:
2024-04-25T02:37:09.686825818Z - name: osm_alert_rules
2024-04-25T02:37:09.686852599Z   rules:
2024-04-25T02:37:09.686855930Z   - alert: vdu_down
2024-04-25T02:37:09.686858972Z     annotations:
2024-04-25T02:37:09.686862882Z       description: VDU {{ $labels.vm_id }} in VIM {{ $labels.vim_id }} has been down
2024-04-25T02:37:09.686866801Z         for more than 3 minutes. NS instance id is {{ $labels.ns_id }}
2024-04-25T02:37:09.686869999Z       summary: VDU {{ $labels.vm_id }} in VIM {{ $labels.vim_id }} is down
2024-04-25T02:37:09.686873494Z     expr: vm_status_extended != 1
2024-04-25T02:37:09.686876541Z     for: 3m
2024-04-25T02:37:09.686879370Z 
2024-04-25T02:37:09.688002461Z New config written, updating prometheus
2024-04-25T02:37:14.719533718Z Prometheus config update successful
2024-04-25T02:37:14.719757790Z 
2024-04-25T02:39:22.966362399Z Changes detected, updating prometheus config
2024-04-25T02:39:22.969810233Z Jobs detected: 1
2024-04-25T02:39:22.980971112Z Writing new config file to /etc/prometheus/prometheus.yml
2024-04-25T02:39:22.986957135Z alerting:
2024-04-25T02:39:22.986997815Z   alertmanagers:
2024-04-25T02:39:22.987007888Z   - static_configs:
2024-04-25T02:39:22.987016463Z     - targets:
2024-04-25T02:39:22.987025401Z       - alertmanager:9093
2024-04-25T02:39:22.987034704Z global:
2024-04-25T02:39:22.987043701Z   evaluation_interval: 15s
2024-04-25T02:39:22.987051978Z   scrape_interval: 15s
2024-04-25T02:39:22.987060236Z rule_files:
2024-04-25T02:39:22.987068486Z - osm_metric_rules.yml
2024-04-25T02:39:22.987076658Z - osm_alert_rules.yml
2024-04-25T02:39:22.987084790Z scrape_configs:
2024-04-25T02:39:22.987093121Z - job_name: mon_exporter
2024-04-25T02:39:22.987101271Z   static_configs:
2024-04-25T02:39:22.987109526Z   - targets:
2024-04-25T02:39:22.987118109Z     - mon:8000
2024-04-25T02:39:22.987126448Z - honor_labels: true
2024-04-25T02:39:22.987134629Z   job_name: pushgateway
2024-04-25T02:39:22.987142878Z   scrape_interval: 30s
2024-04-25T02:39:22.987151042Z   static_configs:
2024-04-25T02:39:22.987159178Z   - targets:
2024-04-25T02:39:22.987168652Z     - pushgateway-prometheus-pushgateway:9091
2024-04-25T02:39:22.987177007Z - job_name: 2516a75feabf4541bf7f141a98f5c32d
2024-04-25T02:39:22.987185196Z   metrics_path: /snmp
2024-04-25T02:39:22.987193457Z   params:
2024-04-25T02:39:22.987201721Z     module:
2024-04-25T02:39:22.987210027Z     - osm-snmp
2024-04-25T02:39:22.987218151Z   relabel_configs:
2024-04-25T02:39:22.987226275Z   - source_labels:
2024-04-25T02:39:22.987234467Z     - __address__
2024-04-25T02:39:22.987243184Z     target_label: __param_target
2024-04-25T02:39:22.987251342Z   - source_labels:
2024-04-25T02:39:22.987259452Z     - __param_target
2024-04-25T02:39:22.987267697Z     target_label: instance
2024-04-25T02:39:22.987276860Z   - replacement: eechart-0011706274-snmpexporter.071af83c-ca74-4caa-af92-d0a08f5a1328.svc:80
2024-04-25T02:39:22.987285221Z     target_label: __address__
2024-04-25T02:39:22.987293347Z   static_configs:
2024-04-25T02:39:22.987301935Z   - targets:
2024-04-25T02:39:22.987310250Z     - 172.21.249.230
2024-04-25T02:39:22.987318239Z 
2024-04-25T02:39:22.992774968Z Alerts read: 0
2024-04-25T02:39:22.994876701Z Writing new alerts file to /etc/prometheus/osm_alert_rules.yml
2024-04-25T02:39:22.996331435Z groups:
2024-04-25T02:39:22.996350181Z - name: osm_alert_rules
2024-04-25T02:39:22.996359213Z   rules:
2024-04-25T02:39:22.996397739Z   - alert: vdu_down
2024-04-25T02:39:22.996406482Z     annotations:
2024-04-25T02:39:22.996416211Z       description: VDU {{ $labels.vm_id }} in VIM {{ $labels.vim_id }} has been down
2024-04-25T02:39:22.996425886Z         for more than 3 minutes. NS instance id is {{ $labels.ns_id }}
2024-04-25T02:39:22.996434446Z       summary: VDU {{ $labels.vm_id }} in VIM {{ $labels.vim_id }} is down
2024-04-25T02:39:22.996443484Z     expr: vm_status_extended != 1
2024-04-25T02:39:22.996452469Z     for: 3m
2024-04-25T02:39:22.996460419Z 
2024-04-25T02:39:22.997611749Z New config written, updating prometheus
2024-04-25T02:39:28.041518174Z Prometheus config update successful
2024-04-25T02:39:28.043928123Z 
2024-04-25T02:39:51.049795248Z Changes detected, updating prometheus config
2024-04-25T02:39:51.051220796Z Jobs detected: 0
2024-04-25T02:39:51.059701999Z Writing new config file to /etc/prometheus/prometheus.yml
2024-04-25T02:39:51.062996583Z alerting:
2024-04-25T02:39:51.063053572Z   alertmanagers:
2024-04-25T02:39:51.063064110Z   - static_configs:
2024-04-25T02:39:51.063072658Z     - targets:
2024-04-25T02:39:51.063082252Z       - alertmanager:9093
2024-04-25T02:39:51.063090538Z global:
2024-04-25T02:39:51.063099510Z   evaluation_interval: 15s
2024-04-25T02:39:51.063108041Z   scrape_interval: 15s
2024-04-25T02:39:51.063116831Z rule_files:
2024-04-25T02:39:51.063125361Z - osm_metric_rules.yml
2024-04-25T02:39:51.063133822Z - osm_alert_rules.yml
2024-04-25T02:39:51.063142174Z scrape_configs:
2024-04-25T02:39:51.063150428Z - job_name: mon_exporter
2024-04-25T02:39:51.063158678Z   static_configs:
2024-04-25T02:39:51.063167105Z   - targets:
2024-04-25T02:39:51.063175594Z     - mon:8000
2024-04-25T02:39:51.063183799Z - honor_labels: true
2024-04-25T02:39:51.063192175Z   job_name: pushgateway
2024-04-25T02:39:51.063200350Z   scrape_interval: 30s
2024-04-25T02:39:51.063208544Z   static_configs:
2024-04-25T02:39:51.063216720Z   - targets:
2024-04-25T02:39:51.063225804Z     - pushgateway-prometheus-pushgateway:9091
2024-04-25T02:39:51.063233922Z 
2024-04-25T02:39:51.068659008Z Alerts read: 0
2024-04-25T02:39:51.072273402Z Writing new alerts file to /etc/prometheus/osm_alert_rules.yml
2024-04-25T02:39:51.074584324Z groups:
2024-04-25T02:39:51.074617946Z - name: osm_alert_rules
2024-04-25T02:39:51.074627368Z   rules:
2024-04-25T02:39:51.074636520Z   - alert: vdu_down
2024-04-25T02:39:51.074645212Z     annotations:
2024-04-25T02:39:51.074654930Z       description: VDU {{ $labels.vm_id }} in VIM {{ $labels.vim_id }} has been down
2024-04-25T02:39:51.074664034Z         for more than 3 minutes. NS instance id is {{ $labels.ns_id }}
2024-04-25T02:39:51.074672274Z       summary: VDU {{ $labels.vm_id }} in VIM {{ $labels.vim_id }} is down
2024-04-25T02:39:51.074681276Z     expr: vm_status_extended != 1
2024-04-25T02:39:51.074689673Z     for: 3m
2024-04-25T02:39:51.074697644Z 
2024-04-25T02:39:51.076648180Z New config written, updating prometheus
2024-04-25T02:39:56.108312729Z Prometheus config update successful
2024-04-25T02:39:56.108394847Z 
2024-04-25T02:52:43.358162983Z Changes detected, updating prometheus config
2024-04-25T02:52:43.362837568Z Jobs detected: 0
2024-04-25T02:52:43.375641512Z Writing new config file to /etc/prometheus/prometheus.yml
2024-04-25T02:52:43.378945868Z alerting:
2024-04-25T02:52:43.378967112Z   alertmanagers:
2024-04-25T02:52:43.378973349Z   - static_configs:
2024-04-25T02:52:43.378978649Z     - targets:
2024-04-25T02:52:43.378984660Z       - alertmanager:9093
2024-04-25T02:52:43.379004222Z global:
2024-04-25T02:52:43.379009861Z   evaluation_interval: 15s
2024-04-25T02:52:43.379015098Z   scrape_interval: 15s
2024-04-25T02:52:43.379020314Z rule_files:
2024-04-25T02:52:43.379025567Z - osm_metric_rules.yml
2024-04-25T02:52:43.379030638Z - osm_alert_rules.yml
2024-04-25T02:52:43.379035737Z scrape_configs:
2024-04-25T02:52:43.379041218Z - job_name: mon_exporter
2024-04-25T02:52:43.379046428Z   static_configs:
2024-04-25T02:52:43.379051646Z   - targets:
2024-04-25T02:52:43.379056765Z     - mon:8000
2024-04-25T02:52:43.379061893Z - honor_labels: true
2024-04-25T02:52:43.379067248Z   job_name: pushgateway
2024-04-25T02:52:43.379072358Z   scrape_interval: 30s
2024-04-25T02:52:43.379077446Z   static_configs:
2024-04-25T02:52:43.379082883Z   - targets:
2024-04-25T02:52:43.379088721Z     - pushgateway-prometheus-pushgateway:9091
2024-04-25T02:52:43.379093819Z 
2024-04-25T02:52:43.383658356Z Alerts read: 2
2024-04-25T02:52:43.386070409Z Writing new alerts file to /etc/prometheus/osm_alert_rules.yml
2024-04-25T02:52:43.389277561Z groups:
2024-04-25T02:52:43.389290589Z - name: osm_alert_rules
2024-04-25T02:52:43.389294756Z   rules:
2024-04-25T02:52:43.389298401Z   - alert: vdu_down
2024-04-25T02:52:43.389311668Z     annotations:
2024-04-25T02:52:43.389316426Z       description: VDU {{ $labels.vm_id }} in VIM {{ $labels.vim_id }} has been down
2024-04-25T02:52:43.389320483Z         for more than 3 minutes. NS instance id is {{ $labels.ns_id }}
2024-04-25T02:52:43.389324173Z       summary: VDU {{ $labels.vm_id }} in VIM {{ $labels.vim_id }} is down
2024-04-25T02:52:43.389328255Z     expr: vm_status_extended != 1
2024-04-25T02:52:43.389331907Z     for: 3m
2024-04-25T02:52:43.389335423Z - name: _osm_alert_rules_20240425025243_
2024-04-25T02:52:43.389339008Z   rules:
2024-04-25T02:52:43.389342987Z   - alert: scalein_f362784a-451d-4c39-b93d-4977078d8d78
2024-04-25T02:52:43.389346527Z     expr: (count (osm_cpu_utilization{ns_id="ddd9fecd-5a16-4117-afe4-7ed9c78ea099",
2024-04-25T02:52:43.389350637Z       vnf_member_index="1", vdu_id="hackfest_basic_metrics-VM"}) > 1) and (avg(osm_cpu_utilization{ns_id="ddd9fecd-5a16-4117-afe4-7ed9c78ea099",
2024-04-25T02:52:43.389354194Z       vnf_member_index="1", vdu_id="hackfest_basic_metrics-VM"}) < 10.0000000000)
2024-04-25T02:52:43.389357764Z     for: 1m
2024-04-25T02:52:43.389361293Z     labels:
2024-04-25T02:52:43.389364801Z       ns_id: ddd9fecd-5a16-4117-afe4-7ed9c78ea099
2024-04-25T02:52:43.389368369Z       vdu_id: hackfest_basic_metrics-VM
2024-04-25T02:52:43.389372354Z       vnf_member_index: '1'
2024-04-25T02:52:43.389375992Z   - alert: scaleout_74c2ba6a-ccb8-44e2-98c5-d4e5950d4278
2024-04-25T02:52:43.389379929Z     expr: (count (osm_cpu_utilization{ns_id="ddd9fecd-5a16-4117-afe4-7ed9c78ea099",
2024-04-25T02:52:43.389383659Z       vnf_member_index="1", vdu_id="hackfest_basic_metrics-VM"}) < 2) and (avg(osm_cpu_utilization{ns_id="ddd9fecd-5a16-4117-afe4-7ed9c78ea099",
2024-04-25T02:52:43.389387249Z       vnf_member_index="1", vdu_id="hackfest_basic_metrics-VM"}) > 60.0000000000)
2024-04-25T02:52:43.389390810Z     for: 1m
2024-04-25T02:52:43.389394277Z     labels:
2024-04-25T02:52:43.389397904Z       ns_id: ddd9fecd-5a16-4117-afe4-7ed9c78ea099
2024-04-25T02:52:43.389401471Z       vdu_id: hackfest_basic_metrics-VM
2024-04-25T02:52:43.389404985Z       vnf_member_index: '1'
2024-04-25T02:52:43.389408278Z 
2024-04-25T02:52:43.404004269Z New config written, updating prometheus
2024-04-25T02:52:48.442805150Z Prometheus config update successful
2024-04-25T02:52:48.442844398Z 
2024-04-25T02:52:48.443124799Z Changes detected, updating prometheus config
2024-04-25T02:52:48.445690547Z Jobs detected: 0
2024-04-25T02:52:48.452563943Z Writing new config file to /etc/prometheus/prometheus.yml
2024-04-25T02:52:48.455348766Z alerting:
2024-04-25T02:52:48.455362008Z   alertmanagers:
2024-04-25T02:52:48.455365734Z   - static_configs:
2024-04-25T02:52:48.455388899Z     - targets:
2024-04-25T02:52:48.455393098Z       - alertmanager:9093
2024-04-25T02:52:48.455396436Z global:
2024-04-25T02:52:48.455399648Z   evaluation_interval: 15s
2024-04-25T02:52:48.455402691Z   scrape_interval: 15s
2024-04-25T02:52:48.455405598Z rule_files:
2024-04-25T02:52:48.455408820Z - osm_metric_rules.yml
2024-04-25T02:52:48.455411702Z - osm_alert_rules.yml
2024-04-25T02:52:48.455414659Z scrape_configs:
2024-04-25T02:52:48.455417597Z - job_name: mon_exporter
2024-04-25T02:52:48.455420651Z   static_configs:
2024-04-25T02:52:48.455423574Z   - targets:
2024-04-25T02:52:48.455426447Z     - mon:8000
2024-04-25T02:52:48.455429306Z - honor_labels: true
2024-04-25T02:52:48.455432269Z   job_name: pushgateway
2024-04-25T02:52:48.455435046Z   scrape_interval: 30s
2024-04-25T02:52:48.455437908Z   static_configs:
2024-04-25T02:52:48.455441265Z   - targets:
2024-04-25T02:52:48.455446150Z     - pushgateway-prometheus-pushgateway:9091
2024-04-25T02:52:48.455450291Z 
2024-04-25T02:52:48.459120022Z Alerts read: 2
2024-04-25T02:52:48.461282257Z Writing new alerts file to /etc/prometheus/osm_alert_rules.yml
2024-04-25T02:52:48.464671765Z groups:
2024-04-25T02:52:48.464684619Z - name: osm_alert_rules
2024-04-25T02:52:48.464688203Z   rules:
2024-04-25T02:52:48.464709764Z   - alert: vdu_down
2024-04-25T02:52:48.464713417Z     annotations:
2024-04-25T02:52:48.464717670Z       description: VDU {{ $labels.vm_id }} in VIM {{ $labels.vim_id }} has been down
2024-04-25T02:52:48.464721597Z         for more than 3 minutes. NS instance id is {{ $labels.ns_id }}
2024-04-25T02:52:48.464724591Z       summary: VDU {{ $labels.vm_id }} in VIM {{ $labels.vim_id }} is down
2024-04-25T02:52:48.464728868Z     expr: vm_status_extended != 1
2024-04-25T02:52:48.464731991Z     for: 3m
2024-04-25T02:52:48.464735084Z - name: _osm_alert_rules_20240425025248_
2024-04-25T02:52:48.464738109Z   rules:
2024-04-25T02:52:48.464741583Z   - alert: scalein_f362784a-451d-4c39-b93d-4977078d8d78
2024-04-25T02:52:48.464744626Z     expr: (count (osm_cpu_utilization{ns_id="ddd9fecd-5a16-4117-afe4-7ed9c78ea099",
2024-04-25T02:52:48.464748349Z       vnf_member_index="1", vdu_id="hackfest_basic_metrics-VM"}) > 1) and (avg(osm_cpu_utilization{ns_id="ddd9fecd-5a16-4117-afe4-7ed9c78ea099",
2024-04-25T02:52:48.464751517Z       vnf_member_index="1", vdu_id="hackfest_basic_metrics-VM"}) < 10.0000000000)
2024-04-25T02:52:48.464754526Z     for: 1m
2024-04-25T02:52:48.464757492Z     labels:
2024-04-25T02:52:48.464760498Z       ns_id: ddd9fecd-5a16-4117-afe4-7ed9c78ea099
2024-04-25T02:52:48.464763560Z       vdu_id: hackfest_basic_metrics-VM
2024-04-25T02:52:48.464767045Z       vnf_member_index: '1'
2024-04-25T02:52:48.464769996Z   - alert: scaleout_74c2ba6a-ccb8-44e2-98c5-d4e5950d4278
2024-04-25T02:52:48.464773026Z     expr: (count (osm_cpu_utilization{ns_id="ddd9fecd-5a16-4117-afe4-7ed9c78ea099",
2024-04-25T02:52:48.464776104Z       vnf_member_index="1", vdu_id="hackfest_basic_metrics-VM"}) < 2) and (avg(osm_cpu_utilization{ns_id="ddd9fecd-5a16-4117-afe4-7ed9c78ea099",
2024-04-25T02:52:48.464779072Z       vnf_member_index="1", vdu_id="hackfest_basic_metrics-VM"}) > 60.0000000000)
2024-04-25T02:52:48.464793401Z     for: 1m
2024-04-25T02:52:48.464796299Z     labels:
2024-04-25T02:52:48.464812744Z       ns_id: ddd9fecd-5a16-4117-afe4-7ed9c78ea099
2024-04-25T02:52:48.464815666Z       vdu_id: hackfest_basic_metrics-VM
2024-04-25T02:52:48.464818502Z       vnf_member_index: '1'
2024-04-25T02:52:48.464821278Z 
2024-04-25T02:52:48.468089842Z New config written, updating prometheus
2024-04-25T02:52:53.497369451Z Prometheus config update successful
2024-04-25T02:52:53.497411557Z 
2024-04-25T02:53:37.185892496Z Changes detected, updating prometheus config
2024-04-25T02:53:37.187184807Z Jobs detected: 0
2024-04-25T02:53:37.191768699Z Writing new config file to /etc/prometheus/prometheus.yml
2024-04-25T02:53:37.194176284Z alerting:
2024-04-25T02:53:37.194192620Z   alertmanagers:
2024-04-25T02:53:37.194197829Z   - static_configs:
2024-04-25T02:53:37.194201925Z     - targets:
2024-04-25T02:53:37.194206807Z       - alertmanager:9093
2024-04-25T02:53:37.194210886Z global:
2024-04-25T02:53:37.194214966Z   evaluation_interval: 15s
2024-04-25T02:53:37.194218869Z   scrape_interval: 15s
2024-04-25T02:53:37.194222821Z rule_files:
2024-04-25T02:53:37.194226798Z - osm_metric_rules.yml
2024-04-25T02:53:37.194230680Z - osm_alert_rules.yml
2024-04-25T02:53:37.194234702Z scrape_configs:
2024-04-25T02:53:37.194238705Z - job_name: mon_exporter
2024-04-25T02:53:37.194242597Z   static_configs:
2024-04-25T02:53:37.194246582Z   - targets:
2024-04-25T02:53:37.194250540Z     - mon:8000
2024-04-25T02:53:37.194254435Z - honor_labels: true
2024-04-25T02:53:37.194258481Z   job_name: pushgateway
2024-04-25T02:53:37.194262334Z   scrape_interval: 30s
2024-04-25T02:53:37.194266244Z   static_configs:
2024-04-25T02:53:37.194270210Z   - targets:
2024-04-25T02:53:37.194274947Z     - pushgateway-prometheus-pushgateway:9091
2024-04-25T02:53:37.194354091Z 
2024-04-25T02:53:37.198196731Z Alerts read: 0
2024-04-25T02:53:37.200229458Z Writing new alerts file to /etc/prometheus/osm_alert_rules.yml
2024-04-25T02:53:37.201395515Z groups:
2024-04-25T02:53:37.201406336Z - name: osm_alert_rules
2024-04-25T02:53:37.201410006Z   rules:
2024-04-25T02:53:37.201413106Z   - alert: vdu_down
2024-04-25T02:53:37.201416131Z     annotations:
2024-04-25T02:53:37.201420129Z       description: VDU {{ $labels.vm_id }} in VIM {{ $labels.vim_id }} has been down
2024-04-25T02:53:37.201423620Z         for more than 3 minutes. NS instance id is {{ $labels.ns_id }}
2024-04-25T02:53:37.201426894Z       summary: VDU {{ $labels.vm_id }} in VIM {{ $labels.vim_id }} is down
2024-04-25T02:53:37.201430542Z     expr: vm_status_extended != 1
2024-04-25T02:53:37.201433733Z     for: 3m
2024-04-25T02:53:37.201502647Z 
2024-04-25T02:53:37.202680588Z New config written, updating prometheus
2024-04-25T02:53:42.233493452Z Prometheus config update successful
2024-04-25T02:53:42.233679384Z 
2024-04-25T02:53:42.233945678Z Changes detected, updating prometheus config
2024-04-25T02:53:42.235489049Z Jobs detected: 0
2024-04-25T02:53:42.242103410Z Writing new config file to /etc/prometheus/prometheus.yml
2024-04-25T02:53:42.244324632Z alerting:
2024-04-25T02:53:42.244334941Z   alertmanagers:
2024-04-25T02:53:42.244339077Z   - static_configs:
2024-04-25T02:53:42.244342250Z     - targets:
2024-04-25T02:53:42.244345814Z       - alertmanager:9093
2024-04-25T02:53:42.244348896Z global:
2024-04-25T02:53:42.244351913Z   evaluation_interval: 15s
2024-04-25T02:53:42.244355160Z   scrape_interval: 15s
2024-04-25T02:53:42.244358203Z rule_files:
2024-04-25T02:53:42.244361264Z - osm_metric_rules.yml
2024-04-25T02:53:42.244364690Z - osm_alert_rules.yml
2024-04-25T02:53:42.244382053Z scrape_configs:
2024-04-25T02:53:42.244385278Z - job_name: mon_exporter
2024-04-25T02:53:42.244388302Z   static_configs:
2024-04-25T02:53:42.244391212Z   - targets:
2024-04-25T02:53:42.244394356Z     - mon:8000
2024-04-25T02:53:42.244397314Z - honor_labels: true
2024-04-25T02:53:42.244400231Z   job_name: pushgateway
2024-04-25T02:53:42.244403520Z   scrape_interval: 30s
2024-04-25T02:53:42.244406574Z   static_configs:
2024-04-25T02:53:42.244409466Z   - targets:
2024-04-25T02:53:42.244413205Z     - pushgateway-prometheus-pushgateway:9091
2024-04-25T02:53:42.244490175Z 
2024-04-25T02:53:42.249921475Z Alerts read: 0
2024-04-25T02:53:42.252025094Z Writing new alerts file to /etc/prometheus/osm_alert_rules.yml
2024-04-25T02:53:42.253313308Z groups:
2024-04-25T02:53:42.253323551Z - name: osm_alert_rules
2024-04-25T02:53:42.253327024Z   rules:
2024-04-25T02:53:42.253330274Z   - alert: vdu_down
2024-04-25T02:53:42.253333710Z     annotations:
2024-04-25T02:53:42.253338119Z       description: VDU {{ $labels.vm_id }} in VIM {{ $labels.vim_id }} has been down
2024-04-25T02:53:42.253341870Z         for more than 3 minutes. NS instance id is {{ $labels.ns_id }}
2024-04-25T02:53:42.253344976Z       summary: VDU {{ $labels.vm_id }} in VIM {{ $labels.vim_id }} is down
2024-04-25T02:53:42.253348551Z     expr: vm_status_extended != 1
2024-04-25T02:53:42.253351624Z     for: 3m
2024-04-25T02:53:42.253462230Z 
2024-04-25T02:53:42.254606730Z New config written, updating prometheus
2024-04-25T02:53:47.276873548Z Prometheus config update successful
2024-04-25T02:53:47.276918131Z 
2024-04-24T22:29:41.233420651Z level=info ts=2024-04-24T22:29:41.233Z caller=main.go:389 msg="No time or size retention was set so using the default time retention" duration=15d
2024-04-24T22:29:41.233753401Z level=info ts=2024-04-24T22:29:41.233Z caller=main.go:443 msg="Starting Prometheus" version="(version=2.28.1, branch=HEAD, revision=b0944590a1c9a6b35dc5a696869f75f422b107a1)"
2024-04-24T22:29:41.233833548Z level=info ts=2024-04-24T22:29:41.233Z caller=main.go:448 build_context="(go=go1.16.5, user=root@2915dd495090, date=20210701-15:20:10)"
2024-04-24T22:29:41.233953004Z level=info ts=2024-04-24T22:29:41.233Z caller=main.go:449 host_details="(Linux 5.15.0-47-generic #51-Ubuntu SMP Thu Aug 11 07:51:15 UTC 2022 x86_64 prometheus-0 (none))"
2024-04-24T22:29:41.234110346Z level=info ts=2024-04-24T22:29:41.234Z caller=main.go:450 fd_limits="(soft=1048576, hard=1048576)"
2024-04-24T22:29:41.234198265Z level=info ts=2024-04-24T22:29:41.234Z caller=main.go:451 vm_limits="(soft=unlimited, hard=unlimited)"
2024-04-24T22:29:41.236305077Z level=info ts=2024-04-24T22:29:41.236Z caller=web.go:541 component=web msg="Start listening for connections" address=0.0.0.0:9090
2024-04-24T22:29:41.237206815Z level=info ts=2024-04-24T22:29:41.237Z caller=main.go:824 msg="Starting TSDB ..."
2024-04-24T22:29:41.238133829Z level=info ts=2024-04-24T22:29:41.238Z caller=tls_config.go:191 component=web msg="TLS is disabled." http2=false
2024-04-24T22:29:41.245149662Z level=info ts=2024-04-24T22:29:41.245Z caller=head.go:780 component=tsdb msg="Replaying on-disk memory mappable chunks if any"
2024-04-24T22:29:41.245269011Z level=info ts=2024-04-24T22:29:41.245Z caller=head.go:794 component=tsdb msg="On-disk memory mappable chunks replay completed" duration=9.403µs
2024-04-24T22:29:41.245385394Z level=info ts=2024-04-24T22:29:41.245Z caller=head.go:800 component=tsdb msg="Replaying WAL, this may take a while"
2024-04-24T22:29:41.245871053Z level=info ts=2024-04-24T22:29:41.245Z caller=head.go:854 component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0
2024-04-24T22:29:41.245986028Z level=info ts=2024-04-24T22:29:41.245Z caller=head.go:860 component=tsdb msg="WAL replay completed" checkpoint_replay_duration=61.256µs wal_replay_duration=433.015µs total_replay_duration=704.93µs
2024-04-24T22:29:41.249304944Z level=info ts=2024-04-24T22:29:41.249Z caller=main.go:851 fs_type=EXT4_SUPER_MAGIC
2024-04-24T22:29:41.249316614Z level=info ts=2024-04-24T22:29:41.249Z caller=main.go:854 msg="TSDB started"
2024-04-24T22:29:41.249320674Z level=info ts=2024-04-24T22:29:41.249Z caller=main.go:981 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml
2024-04-24T22:29:41.252060985Z level=info ts=2024-04-24T22:29:41.251Z caller=main.go:1012 msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=2.663975ms remote_storage=12.95µs web_handler=917ns query_engine=4.162µs scrape=441.648µs scrape_sd=59.986µs notify=69.714µs notify_sd=20.912µs rules=1.415506ms
2024-04-24T22:29:41.252071954Z level=info ts=2024-04-24T22:29:41.251Z caller=main.go:796 msg="Server is ready to receive web requests."
2024-04-24T22:33:28.707235156Z level=info ts=2024-04-24T22:33:28.707Z caller=main.go:981 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml
2024-04-24T22:33:28.720563821Z level=info ts=2024-04-24T22:33:28.719Z caller=main.go:1012 msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=12.787774ms remote_storage=2.447µs web_handler=4.973µs query_engine=1.81µs scrape=3.369771ms scrape_sd=126.089µs notify=29.197µs notify_sd=10.836µs rules=8.403272ms
2024-04-24T22:52:13.136501325Z level=warn ts=2024-04-24T22:52:13.132Z caller=manager.go:603 component="rule manager" group=osm_metric_rules msg="Evaluating rule failed" rule="record: vm_status_extended\nexpr: (last_over_time(vm_status[1m]) * on(vm_id, vim_id) group_left(ns_id, vnf_id,\n  vdu_id, project_id, job, vdu_name, vnf_member_index) last_over_time(ns_topology[1m]))\n  or (last_over_time(ns_topology[1m]) * -1)\nlabels:\n  job: osm_prometheus\n" err="found duplicate series for the match group {vim_id=\"dba3fb20-5a29-4377-aaf0-fef0fc52706a\", vm_id=\"9a2144be-be32-404f-8a3b-b83acae7fe7e\"} on the right hand-side of the operation: [{__name__=\"ns_topology\", job=\"airflow_osm_ns_topology\", ns_id=\"3921c043-0790-4b02-b8f0-cdb826653411\", project_id=\"df61d3ec-64e7-4cf7-827d-442bd4e0a3e4\", vdu_id=\"2da2558a-3a85-4969-85bb-971800fd9740\", vdu_name=\"basic_07_secure_-vnf2-mgmtVM-0\", vim_id=\"dba3fb20-5a29-4377-aaf0-fef0fc52706a\", vm_id=\"9a2144be-be32-404f-8a3b-b83acae7fe7e\", vnf_id=\"0d871a47-a087-4655-a333-d009c7197519\", vnf_member_index=\"vnf2\"}, {__name__=\"ns_topology\", job=\"airflow_osm_ns_topology\", ns_id=\"3921c043-0790-4b02-b8f0-cdb826653411\", project_id=\"df61d3ec-64e7-4cf7-827d-442bd4e0a3e4\", vdu_id=\"2da2558a-3a85-4969-85bb-971800fd9740\", vdu_name=\"UNKNOWN\", vim_id=\"dba3fb20-5a29-4377-aaf0-fef0fc52706a\", vm_id=\"9a2144be-be32-404f-8a3b-b83acae7fe7e\", vnf_id=\"0d871a47-a087-4655-a333-d009c7197519\", vnf_member_index=\"vnf2\"}];many-to-many matching not allowed: matching labels must be unique on one side"
2024-04-24T22:52:28.129261524Z level=warn ts=2024-04-24T22:52:28.128Z caller=manager.go:603 component="rule manager" group=osm_metric_rules msg="Evaluating rule failed" rule="record: vm_status_extended\nexpr: (last_over_time(vm_status[1m]) * on(vm_id, vim_id) group_left(ns_id, vnf_id,\n  vdu_id, project_id, job, vdu_name, vnf_member_index) last_over_time(ns_topology[1m]))\n  or (last_over_time(ns_topology[1m]) * -1)\nlabels:\n  job: osm_prometheus\n" err="found duplicate series for the match group {vim_id=\"dba3fb20-5a29-4377-aaf0-fef0fc52706a\", vm_id=\"9a2144be-be32-404f-8a3b-b83acae7fe7e\"} on the right hand-side of the operation: [{__name__=\"ns_topology\", job=\"airflow_osm_ns_topology\", ns_id=\"3921c043-0790-4b02-b8f0-cdb826653411\", project_id=\"df61d3ec-64e7-4cf7-827d-442bd4e0a3e4\", vdu_id=\"2da2558a-3a85-4969-85bb-971800fd9740\", vdu_name=\"basic_07_secure_-vnf2-mgmtVM-0\", vim_id=\"dba3fb20-5a29-4377-aaf0-fef0fc52706a\", vm_id=\"9a2144be-be32-404f-8a3b-b83acae7fe7e\", vnf_id=\"0d871a47-a087-4655-a333-d009c7197519\", vnf_member_index=\"vnf2\"}, {__name__=\"ns_topology\", job=\"airflow_osm_ns_topology\", ns_id=\"3921c043-0790-4b02-b8f0-cdb826653411\", project_id=\"df61d3ec-64e7-4cf7-827d-442bd4e0a3e4\", vdu_id=\"2da2558a-3a85-4969-85bb-971800fd9740\", vdu_name=\"UNKNOWN\", vim_id=\"dba3fb20-5a29-4377-aaf0-fef0fc52706a\", vm_id=\"9a2144be-be32-404f-8a3b-b83acae7fe7e\", vnf_id=\"0d871a47-a087-4655-a333-d009c7197519\", vnf_member_index=\"vnf2\"}];many-to-many matching not allowed: matching labels must be unique on one side"
2024-04-24T22:56:57.605863315Z level=info ts=2024-04-24T22:56:57.605Z caller=main.go:981 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml
2024-04-24T22:56:57.612574411Z level=info ts=2024-04-24T22:56:57.612Z caller=main.go:1012 msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=6.731988ms remote_storage=3.449µs web_handler=10.28µs query_engine=2.974µs scrape=278.498µs scrape_sd=121.551µs notify=63.364µs notify_sd=22.701µs rules=4.752077ms
2024-04-24T22:57:02.675247164Z level=info ts=2024-04-24T22:57:02.675Z caller=main.go:981 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml
2024-04-24T22:57:02.677284939Z level=info ts=2024-04-24T22:57:02.677Z caller=main.go:1012 msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=2.069834ms remote_storage=2.483µs web_handler=913ns query_engine=1.406µs scrape=86.434µs scrape_sd=32.765µs notify=15.015µs notify_sd=9.216µs rules=1.498999ms
2024-04-24T22:58:26.549452620Z level=info ts=2024-04-24T22:58:26.549Z caller=main.go:981 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml
2024-04-24T22:58:26.552149442Z level=info ts=2024-04-24T22:58:26.552Z caller=main.go:1012 msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=2.745227ms remote_storage=4.654µs web_handler=5.75µs query_engine=1.664µs scrape=105.044µs scrape_sd=51.993µs notify=17.262µs notify_sd=12.877µs rules=1.388144ms
2024-04-24T22:58:31.612313069Z level=info ts=2024-04-24T22:58:31.611Z caller=main.go:981 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml
2024-04-24T22:58:31.617587971Z level=info ts=2024-04-24T22:58:31.617Z caller=main.go:1012 msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=5.615121ms remote_storage=11.762µs web_handler=1.149µs query_engine=14.184µs scrape=627.485µs scrape_sd=158.507µs notify=96.209µs notify_sd=29.644µs rules=3.109867ms
2024-04-24T23:18:53.443519711Z level=info ts=2024-04-24T23:18:53.443Z caller=main.go:981 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml
2024-04-24T23:18:53.453748985Z level=info ts=2024-04-24T23:18:53.448Z caller=main.go:1012 msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=5.938883ms remote_storage=15.131µs web_handler=4.746µs query_engine=1.428µs scrape=124.752µs scrape_sd=55.979µs notify=39.35µs notify_sd=10.236µs rules=5.087224ms
2024-04-24T23:18:58.506756110Z level=info ts=2024-04-24T23:18:58.506Z caller=main.go:981 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml
2024-04-24T23:18:58.514405482Z level=info ts=2024-04-24T23:18:58.514Z caller=main.go:1012 msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=7.648466ms remote_storage=3.519µs web_handler=1.232µs query_engine=1.944µs scrape=155.423µs scrape_sd=63.534µs notify=20.538µs notify_sd=19.934µs rules=6.609659ms
2024-04-24T23:19:03.588477984Z level=info ts=2024-04-24T23:19:03.588Z caller=main.go:981 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml
2024-04-24T23:19:03.593472968Z level=info ts=2024-04-24T23:19:03.593Z caller=main.go:1012 msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=5.012197ms remote_storage=2.567µs web_handler=539ns query_engine=1.642µs scrape=151.996µs scrape_sd=64.035µs notify=18.095µs notify_sd=46.33µs rules=4.247714ms
2024-04-24T23:19:08.658777854Z level=info ts=2024-04-24T23:19:08.658Z caller=main.go:981 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml
2024-04-24T23:19:08.663254134Z level=info ts=2024-04-24T23:19:08.663Z caller=main.go:1012 msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=4.401815ms remote_storage=2.227µs web_handler=563ns query_engine=1.375µs scrape=88.503µs scrape_sd=35.46µs notify=12.355µs notify_sd=9.275µs rules=3.871925ms
2024-04-24T23:19:13.721430908Z level=info ts=2024-04-24T23:19:13.721Z caller=main.go:981 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml
2024-04-24T23:19:13.725446558Z level=info ts=2024-04-24T23:19:13.724Z caller=main.go:1012 msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=3.284923ms remote_storage=2.195µs web_handler=887ns query_engine=1.118µs scrape=164.396µs scrape_sd=34.342µs notify=12.355µs notify_sd=9.618µs rules=2.776584ms
2024-04-24T23:19:18.788981393Z level=info ts=2024-04-24T23:19:18.788Z caller=main.go:981 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml
2024-04-24T23:19:18.792925431Z level=info ts=2024-04-24T23:19:18.792Z caller=main.go:1012 msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=4.030854ms remote_storage=2.172µs web_handler=1.039µs query_engine=1.452µs scrape=87.609µs scrape_sd=35.221µs notify=12.671µs notify_sd=9.239µs rules=2.695294ms
2024-04-24T23:20:20.681067196Z level=info ts=2024-04-24T23:20:20.680Z caller=main.go:981 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml
2024-04-24T23:20:20.684413147Z level=info ts=2024-04-24T23:20:20.684Z caller=main.go:1012 msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=3.287502ms remote_storage=2.247µs web_handler=903ns query_engine=1.46µs scrape=212.549µs scrape_sd=69.033µs notify=18.555µs notify_sd=87.767µs rules=1.927123ms
2024-04-24T23:20:25.717529997Z level=info ts=2024-04-24T23:20:25.717Z caller=main.go:981 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml
2024-04-24T23:20:25.720585334Z level=info ts=2024-04-24T23:20:25.720Z caller=main.go:1012 msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=3.061863ms remote_storage=2.842µs web_handler=852ns query_engine=1.995µs scrape=159.497µs scrape_sd=51.986µs notify=19.174µs notify_sd=16.343µs rules=2.232446ms
2024-04-24T23:20:30.778165280Z level=info ts=2024-04-24T23:20:30.778Z caller=main.go:981 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml
2024-04-24T23:20:30.781250865Z level=info ts=2024-04-24T23:20:30.781Z caller=main.go:1012 msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=3.089805ms remote_storage=2.111µs web_handler=573ns query_engine=1.228µs scrape=96.557µs scrape_sd=44.216µs notify=12.075µs notify_sd=9.682µs rules=2.542719ms
2024-04-24T23:20:35.816536775Z level=info ts=2024-04-24T23:20:35.816Z caller=main.go:981 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml
2024-04-24T23:20:35.819190756Z level=info ts=2024-04-24T23:20:35.819Z caller=main.go:1012 msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=2.65875ms remote_storage=2.676µs web_handler=798ns query_engine=1.473µs scrape=152.67µs scrape_sd=78.532µs notify=41.528µs notify_sd=22.02µs rules=1.716479ms
2024-04-24T23:20:40.857023479Z level=info ts=2024-04-24T23:20:40.856Z caller=main.go:981 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml
2024-04-24T23:20:40.859121746Z level=info ts=2024-04-24T23:20:40.859Z caller=main.go:1012 msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=2.099546ms remote_storage=2.168µs web_handler=563ns query_engine=12.502µs scrape=89.467µs scrape_sd=31.652µs notify=28.463µs notify_sd=9.512µs rules=1.411499ms
2024-04-24T23:20:45.911038093Z level=info ts=2024-04-24T23:20:45.910Z caller=main.go:981 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml
2024-04-24T23:20:45.913599396Z level=info ts=2024-04-24T23:20:45.913Z caller=main.go:1012 msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=2.552851ms remote_storage=3.16µs web_handler=760ns query_engine=1.56µs scrape=113.281µs scrape_sd=52.322µs notify=23.972µs notify_sd=12.312µs rules=1.735235ms
2024-04-25T00:00:13.130507077Z level=warn ts=2024-04-25T00:00:13.129Z caller=manager.go:603 component="rule manager" group=osm_metric_rules msg="Evaluating rule failed" rule="record: vm_status_extended\nexpr: (last_over_time(vm_status[1m]) * on(vm_id, vim_id) group_left(ns_id, vnf_id,\n  vdu_id, project_id, job, vdu_name, vnf_member_index) last_over_time(ns_topology[1m]))\n  or (last_over_time(ns_topology[1m]) * -1)\nlabels:\n  job: osm_prometheus\n" err="found duplicate series for the match group {vim_id=\"dba3fb20-5a29-4377-aaf0-fef0fc52706a\", vm_id=\"e0babc98-8242-4582-8690-b8eaa0f31254\"} on the right hand-side of the operation: [{__name__=\"ns_topology\", job=\"airflow_osm_ns_topology\", ns_id=\"480bf56d-8787-4f48-a28c-29d05580ceac\", project_id=\"df61d3ec-64e7-4cf7-827d-442bd4e0a3e4\", vdu_id=\"a2d3bce3-c60b-4c85-9a63-9ac3ccc722eb\", vdu_name=\"basic_27_helm_ee-simple-mgmtVM-0\", vim_id=\"dba3fb20-5a29-4377-aaf0-fef0fc52706a\", vm_id=\"e0babc98-8242-4582-8690-b8eaa0f31254\", vnf_id=\"28117f64-a239-472e-b65a-b140dd051a22\", vnf_member_index=\"simple\"}, {__name__=\"ns_topology\", job=\"airflow_osm_ns_topology\", ns_id=\"480bf56d-8787-4f48-a28c-29d05580ceac\", project_id=\"df61d3ec-64e7-4cf7-827d-442bd4e0a3e4\", vdu_id=\"a2d3bce3-c60b-4c85-9a63-9ac3ccc722eb\", vdu_name=\"UNKNOWN\", vim_id=\"dba3fb20-5a29-4377-aaf0-fef0fc52706a\", vm_id=\"e0babc98-8242-4582-8690-b8eaa0f31254\", vnf_id=\"28117f64-a239-472e-b65a-b140dd051a22\", vnf_member_index=\"simple\"}];many-to-many matching not allowed: matching labels must be unique on one side"
2024-04-25T00:00:28.133674554Z level=warn ts=2024-04-25T00:00:28.132Z caller=manager.go:603 component="rule manager" group=osm_metric_rules msg="Evaluating rule failed" rule="record: vm_status_extended\nexpr: (last_over_time(vm_status[1m]) * on(vm_id, vim_id) group_left(ns_id, vnf_id,\n  vdu_id, project_id, job, vdu_name, vnf_member_index) last_over_time(ns_topology[1m]))\n  or (last_over_time(ns_topology[1m]) * -1)\nlabels:\n  job: osm_prometheus\n" err="found duplicate series for the match group {vim_id=\"dba3fb20-5a29-4377-aaf0-fef0fc52706a\", vm_id=\"e0babc98-8242-4582-8690-b8eaa0f31254\"} on the right hand-side of the operation: [{__name__=\"ns_topology\", job=\"airflow_osm_ns_topology\", ns_id=\"480bf56d-8787-4f48-a28c-29d05580ceac\", project_id=\"df61d3ec-64e7-4cf7-827d-442bd4e0a3e4\", vdu_id=\"a2d3bce3-c60b-4c85-9a63-9ac3ccc722eb\", vdu_name=\"basic_27_helm_ee-simple-mgmtVM-0\", vim_id=\"dba3fb20-5a29-4377-aaf0-fef0fc52706a\", vm_id=\"e0babc98-8242-4582-8690-b8eaa0f31254\", vnf_id=\"28117f64-a239-472e-b65a-b140dd051a22\", vnf_member_index=\"simple\"}, {__name__=\"ns_topology\", job=\"airflow_osm_ns_topology\", ns_id=\"480bf56d-8787-4f48-a28c-29d05580ceac\", project_id=\"df61d3ec-64e7-4cf7-827d-442bd4e0a3e4\", vdu_id=\"a2d3bce3-c60b-4c85-9a63-9ac3ccc722eb\", vdu_name=\"UNKNOWN\", vim_id=\"dba3fb20-5a29-4377-aaf0-fef0fc52706a\", vm_id=\"e0babc98-8242-4582-8690-b8eaa0f31254\", vnf_id=\"28117f64-a239-472e-b65a-b140dd051a22\", vnf_member_index=\"simple\"}];many-to-many matching not allowed: matching labels must be unique on one side"
2024-04-25T00:58:13.127935176Z level=warn ts=2024-04-25T00:58:13.127Z caller=manager.go:603 component="rule manager" group=osm_metric_rules msg="Evaluating rule failed" rule="record: vm_status_extended\nexpr: (last_over_time(vm_status[1m]) * on(vm_id, vim_id) group_left(ns_id, vnf_id,\n  vdu_id, project_id, job, vdu_name, vnf_member_index) last_over_time(ns_topology[1m]))\n  or (last_over_time(ns_topology[1m]) * -1)\nlabels:\n  job: osm_prometheus\n" err="found duplicate series for the match group {vim_id=\"dba3fb20-5a29-4377-aaf0-fef0fc52706a\", vm_id=\"b4222579-f193-4ee8-a21a-d66fb6c18c53\"} on the right hand-side of the operation: [{__name__=\"ns_topology\", job=\"airflow_osm_ns_topology\", ns_id=\"784dfed7-1fd1-477a-9cb3-a8c9bcda7993\", project_id=\"df61d3ec-64e7-4cf7-827d-442bd4e0a3e4\", vdu_id=\"0f0b4279-8b6a-4cf6-80f5-643d52586971\", vdu_name=\"heal_03-several_volumes_-several_volumes-VM-0\", vim_id=\"dba3fb20-5a29-4377-aaf0-fef0fc52706a\", vm_id=\"b4222579-f193-4ee8-a21a-d66fb6c18c53\", vnf_id=\"e65c88b2-fd93-4084-847c-a7610f6ba17a\", vnf_member_index=\"several_volumes_vnf\"}, {__name__=\"ns_topology\", job=\"airflow_osm_ns_topology\", ns_id=\"784dfed7-1fd1-477a-9cb3-a8c9bcda7993\", project_id=\"df61d3ec-64e7-4cf7-827d-442bd4e0a3e4\", vdu_id=\"0f0b4279-8b6a-4cf6-80f5-643d52586971\", vdu_name=\"UNKNOWN\", vim_id=\"dba3fb20-5a29-4377-aaf0-fef0fc52706a\", vm_id=\"b4222579-f193-4ee8-a21a-d66fb6c18c53\", vnf_id=\"e65c88b2-fd93-4084-847c-a7610f6ba17a\", vnf_member_index=\"several_volumes_vnf\"}];many-to-many matching not allowed: matching labels must be unique on one side"
2024-04-25T00:58:28.129434960Z level=warn ts=2024-04-25T00:58:28.128Z caller=manager.go:603 component="rule manager" group=osm_metric_rules msg="Evaluating rule failed" rule="record: vm_status_extended\nexpr: (last_over_time(vm_status[1m]) * on(vm_id, vim_id) group_left(ns_id, vnf_id,\n  vdu_id, project_id, job, vdu_name, vnf_member_index) last_over_time(ns_topology[1m]))\n  or (last_over_time(ns_topology[1m]) * -1)\nlabels:\n  job: osm_prometheus\n" err="found duplicate series for the match group {vim_id=\"dba3fb20-5a29-4377-aaf0-fef0fc52706a\", vm_id=\"b4222579-f193-4ee8-a21a-d66fb6c18c53\"} on the right hand-side of the operation: [{__name__=\"ns_topology\", job=\"airflow_osm_ns_topology\", ns_id=\"784dfed7-1fd1-477a-9cb3-a8c9bcda7993\", project_id=\"df61d3ec-64e7-4cf7-827d-442bd4e0a3e4\", vdu_id=\"0f0b4279-8b6a-4cf6-80f5-643d52586971\", vdu_name=\"heal_03-several_volumes_-several_volumes-VM-0\", vim_id=\"dba3fb20-5a29-4377-aaf0-fef0fc52706a\", vm_id=\"b4222579-f193-4ee8-a21a-d66fb6c18c53\", vnf_id=\"e65c88b2-fd93-4084-847c-a7610f6ba17a\", vnf_member_index=\"several_volumes_vnf\"}, {__name__=\"ns_topology\", job=\"airflow_osm_ns_topology\", ns_id=\"784dfed7-1fd1-477a-9cb3-a8c9bcda7993\", project_id=\"df61d3ec-64e7-4cf7-827d-442bd4e0a3e4\", vdu_id=\"0f0b4279-8b6a-4cf6-80f5-643d52586971\", vdu_name=\"UNKNOWN\", vim_id=\"dba3fb20-5a29-4377-aaf0-fef0fc52706a\", vm_id=\"b4222579-f193-4ee8-a21a-d66fb6c18c53\", vnf_id=\"e65c88b2-fd93-4084-847c-a7610f6ba17a\", vnf_member_index=\"several_volumes_vnf\"}];many-to-many matching not allowed: matching labels must be unique on one side"
2024-04-25T01:13:26.666106069Z level=info ts=2024-04-25T01:13:26.665Z caller=main.go:981 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml
2024-04-25T01:13:26.670458308Z level=info ts=2024-04-25T01:13:26.670Z caller=main.go:1012 msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=4.432302ms remote_storage=3.944µs web_handler=9.403µs query_engine=3.022µs scrape=250.386µs scrape_sd=95.27µs notify=48.681µs notify_sd=21.679µs rules=2.739869ms
2024-04-25T01:18:40.014855082Z level=info ts=2024-04-25T01:18:40.014Z caller=main.go:981 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml
2024-04-25T01:18:40.016884478Z level=info ts=2024-04-25T01:18:40.016Z caller=main.go:1012 msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=2.090393ms remote_storage=3.097µs web_handler=5.477µs query_engine=1.879µs scrape=108.74µs scrape_sd=42.595µs notify=20.183µs notify_sd=13.968µs rules=1.255165ms
2024-04-25T01:29:54.217552402Z level=info ts=2024-04-25T01:29:54.216Z caller=compact.go:518 component=tsdb msg="write block" mint=1713997794124 maxt=1714003200000 ulid=01HW9DZ5QDWEV2P54E47JRE3J9 duration=59.267293ms
2024-04-25T01:29:54.221440955Z level=info ts=2024-04-25T01:29:54.221Z caller=head.go:967 component=tsdb msg="Head GC completed" duration=2.697094ms
2024-04-25T01:50:13.128417328Z level=warn ts=2024-04-25T01:50:13.128Z caller=manager.go:603 component="rule manager" group=osm_metric_rules msg="Evaluating rule failed" rule="record: vm_status_extended\nexpr: (last_over_time(vm_status[1m]) * on(vm_id, vim_id) group_left(ns_id, vnf_id,\n  vdu_id, project_id, job, vdu_name, vnf_member_index) last_over_time(ns_topology[1m]))\n  or (last_over_time(ns_topology[1m]) * -1)\nlabels:\n  job: osm_prometheus\n" err="found duplicate series for the match group {vim_id=\"dba3fb20-5a29-4377-aaf0-fef0fc52706a\", vm_id=\"ecaa166f-49c8-4c21-b343-08cda1b1d949\"} on the right hand-side of the operation: [{__name__=\"ns_topology\", job=\"airflow_osm_ns_topology\", ns_id=\"21f35c50-4a83-4ed4-b48c-b46ba41b7fa3\", project_id=\"df61d3ec-64e7-4cf7-827d-442bd4e0a3e4\", vdu_id=\"744f26e0-031a-46b1-a7b3-d60663983b9b\", vdu_name=\"k8s_10-k8s_proxy-vnf2-mgmtVM-0\", vim_id=\"dba3fb20-5a29-4377-aaf0-fef0fc52706a\", vm_id=\"ecaa166f-49c8-4c21-b343-08cda1b1d949\", vnf_id=\"30479525-d9d2-4019-8f1a-c7ed4b5716a3\", vnf_member_index=\"vnf2\"}, {__name__=\"ns_topology\", job=\"airflow_osm_ns_topology\", ns_id=\"21f35c50-4a83-4ed4-b48c-b46ba41b7fa3\", project_id=\"df61d3ec-64e7-4cf7-827d-442bd4e0a3e4\", vdu_id=\"744f26e0-031a-46b1-a7b3-d60663983b9b\", vdu_name=\"UNKNOWN\", vim_id=\"dba3fb20-5a29-4377-aaf0-fef0fc52706a\", vm_id=\"ecaa166f-49c8-4c21-b343-08cda1b1d949\", vnf_id=\"30479525-d9d2-4019-8f1a-c7ed4b5716a3\", vnf_member_index=\"vnf2\"}];many-to-many matching not allowed: matching labels must be unique on one side"
2024-04-25T01:50:28.129496230Z level=warn ts=2024-04-25T01:50:28.129Z caller=manager.go:603 component="rule manager" group=osm_metric_rules msg="Evaluating rule failed" rule="record: vm_status_extended\nexpr: (last_over_time(vm_status[1m]) * on(vm_id, vim_id) group_left(ns_id, vnf_id,\n  vdu_id, project_id, job, vdu_name, vnf_member_index) last_over_time(ns_topology[1m]))\n  or (last_over_time(ns_topology[1m]) * -1)\nlabels:\n  job: osm_prometheus\n" err="found duplicate series for the match group {vim_id=\"dba3fb20-5a29-4377-aaf0-fef0fc52706a\", vm_id=\"ecaa166f-49c8-4c21-b343-08cda1b1d949\"} on the right hand-side of the operation: [{__name__=\"ns_topology\", job=\"airflow_osm_ns_topology\", ns_id=\"21f35c50-4a83-4ed4-b48c-b46ba41b7fa3\", project_id=\"df61d3ec-64e7-4cf7-827d-442bd4e0a3e4\", vdu_id=\"744f26e0-031a-46b1-a7b3-d60663983b9b\", vdu_name=\"k8s_10-k8s_proxy-vnf2-mgmtVM-0\", vim_id=\"dba3fb20-5a29-4377-aaf0-fef0fc52706a\", vm_id=\"ecaa166f-49c8-4c21-b343-08cda1b1d949\", vnf_id=\"30479525-d9d2-4019-8f1a-c7ed4b5716a3\", vnf_member_index=\"vnf2\"}, {__name__=\"ns_topology\", job=\"airflow_osm_ns_topology\", ns_id=\"21f35c50-4a83-4ed4-b48c-b46ba41b7fa3\", project_id=\"df61d3ec-64e7-4cf7-827d-442bd4e0a3e4\", vdu_id=\"744f26e0-031a-46b1-a7b3-d60663983b9b\", vdu_name=\"UNKNOWN\", vim_id=\"dba3fb20-5a29-4377-aaf0-fef0fc52706a\", vm_id=\"ecaa166f-49c8-4c21-b343-08cda1b1d949\", vnf_id=\"30479525-d9d2-4019-8f1a-c7ed4b5716a3\", vnf_member_index=\"vnf2\"}];many-to-many matching not allowed: matching labels must be unique on one side"
2024-04-25T01:59:17.288367851Z level=info ts=2024-04-25T01:59:17.288Z caller=main.go:981 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml
2024-04-25T01:59:17.290964719Z level=info ts=2024-04-25T01:59:17.290Z caller=main.go:1012 msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=2.663116ms remote_storage=2.469µs web_handler=4.672µs query_engine=1.551µs scrape=164.766µs scrape_sd=46.28µs notify=23.461µs notify_sd=10.097µs rules=1.705144ms
2024-04-25T01:59:22.362163537Z level=info ts=2024-04-25T01:59:22.362Z caller=main.go:981 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml
2024-04-25T01:59:22.366756700Z level=info ts=2024-04-25T01:59:22.366Z caller=main.go:1012 msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=4.527332ms remote_storage=2.327µs web_handler=685ns query_engine=1.224µs scrape=86.852µs scrape_sd=152.135µs notify=25.5µs notify_sd=12.362µs rules=2.831261ms
2024-04-25T02:06:41.406169843Z level=info ts=2024-04-25T02:06:41.405Z caller=main.go:981 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml
2024-04-25T02:06:41.410022041Z level=info ts=2024-04-25T02:06:41.409Z caller=main.go:1012 msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=4.014208ms remote_storage=3.249µs web_handler=10.033µs query_engine=2.868µs scrape=276.014µs scrape_sd=129.192µs notify=53.154µs notify_sd=50.21µs rules=2.122189ms
2024-04-25T02:06:46.457373263Z level=info ts=2024-04-25T02:06:46.457Z caller=main.go:981 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml
2024-04-25T02:06:46.459873414Z level=info ts=2024-04-25T02:06:46.459Z caller=main.go:1012 msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=2.530073ms remote_storage=2.062µs web_handler=535ns query_engine=1.413µs scrape=121.797µs scrape_sd=65.982µs notify=33.441µs notify_sd=23.992µs rules=1.757067ms
2024-04-25T02:07:32.205523089Z level=info ts=2024-04-25T02:07:32.205Z caller=main.go:981 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml
2024-04-25T02:07:32.210505475Z level=info ts=2024-04-25T02:07:32.210Z caller=main.go:1012 msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=4.952984ms remote_storage=3.568µs web_handler=1.115µs query_engine=2.043µs scrape=187.758µs scrape_sd=67.456µs notify=134.1µs notify_sd=37.294µs rules=3.706394ms
2024-04-25T02:07:37.250512043Z level=info ts=2024-04-25T02:07:37.250Z caller=main.go:981 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml
2024-04-25T02:07:37.255818242Z level=info ts=2024-04-25T02:07:37.255Z caller=main.go:1012 msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=5.247876ms remote_storage=3.685µs web_handler=952ns query_engine=1.993µs scrape=181.527µs scrape_sd=64.091µs notify=22.788µs notify_sd=21.102µs rules=4.255855ms
2024-04-25T02:24:13.129863799Z level=warn ts=2024-04-25T02:24:13.129Z caller=manager.go:603 component="rule manager" group=osm_metric_rules msg="Evaluating rule failed" rule="record: vm_status_extended\nexpr: (last_over_time(vm_status[1m]) * on(vm_id, vim_id) group_left(ns_id, vnf_id,\n  vdu_id, project_id, job, vdu_name, vnf_member_index) last_over_time(ns_topology[1m]))\n  or (last_over_time(ns_topology[1m]) * -1)\nlabels:\n  job: osm_prometheus\n" err="found duplicate series for the match group {vim_id=\"dba3fb20-5a29-4377-aaf0-fef0fc52706a\", vm_id=\"273b4068-b40c-42a1-881b-a3d102d427b7\"} on the right hand-side of the operation: [{__name__=\"ns_topology\", job=\"airflow_osm_ns_topology\", ns_id=\"a2856130-a67d-4a8f-9ac2-0157054182ba\", project_id=\"df61d3ec-64e7-4cf7-827d-442bd4e0a3e4\", vdu_id=\"f63a62a3-1d73-4efb-b1e6-f9c9723aef63\", vdu_name=\"sa_02-vnf-hackfest_basic_metrics-VM-1\", vim_id=\"dba3fb20-5a29-4377-aaf0-fef0fc52706a\", vm_id=\"273b4068-b40c-42a1-881b-a3d102d427b7\", vnf_id=\"fdbc09db-a607-4511-84e4-a0cd44f9544c\", vnf_member_index=\"vnf\"}, {__name__=\"ns_topology\", job=\"airflow_osm_ns_topology\", ns_id=\"a2856130-a67d-4a8f-9ac2-0157054182ba\", project_id=\"df61d3ec-64e7-4cf7-827d-442bd4e0a3e4\", vdu_id=\"f63a62a3-1d73-4efb-b1e6-f9c9723aef63\", vdu_name=\"sa_02-vnf-hackfest_basic_metrics-VM-0\", vim_id=\"dba3fb20-5a29-4377-aaf0-fef0fc52706a\", vm_id=\"273b4068-b40c-42a1-881b-a3d102d427b7\", vnf_id=\"fdbc09db-a607-4511-84e4-a0cd44f9544c\", vnf_member_index=\"vnf\"}];many-to-many matching not allowed: matching labels must be unique on one side"
2024-04-25T02:24:28.128729257Z level=warn ts=2024-04-25T02:24:28.128Z caller=manager.go:603 component="rule manager" group=osm_metric_rules msg="Evaluating rule failed" rule="record: vm_status_extended\nexpr: (last_over_time(vm_status[1m]) * on(vm_id, vim_id) group_left(ns_id, vnf_id,\n  vdu_id, project_id, job, vdu_name, vnf_member_index) last_over_time(ns_topology[1m]))\n  or (last_over_time(ns_topology[1m]) * -1)\nlabels:\n  job: osm_prometheus\n" err="found duplicate series for the match group {vim_id=\"dba3fb20-5a29-4377-aaf0-fef0fc52706a\", vm_id=\"273b4068-b40c-42a1-881b-a3d102d427b7\"} on the right hand-side of the operation: [{__name__=\"ns_topology\", job=\"airflow_osm_ns_topology\", ns_id=\"a2856130-a67d-4a8f-9ac2-0157054182ba\", project_id=\"df61d3ec-64e7-4cf7-827d-442bd4e0a3e4\", vdu_id=\"f63a62a3-1d73-4efb-b1e6-f9c9723aef63\", vdu_name=\"sa_02-vnf-hackfest_basic_metrics-VM-1\", vim_id=\"dba3fb20-5a29-4377-aaf0-fef0fc52706a\", vm_id=\"273b4068-b40c-42a1-881b-a3d102d427b7\", vnf_id=\"fdbc09db-a607-4511-84e4-a0cd44f9544c\", vnf_member_index=\"vnf\"}, {__name__=\"ns_topology\", job=\"airflow_osm_ns_topology\", ns_id=\"a2856130-a67d-4a8f-9ac2-0157054182ba\", project_id=\"df61d3ec-64e7-4cf7-827d-442bd4e0a3e4\", vdu_id=\"f63a62a3-1d73-4efb-b1e6-f9c9723aef63\", vdu_name=\"sa_02-vnf-hackfest_basic_metrics-VM-0\", vim_id=\"dba3fb20-5a29-4377-aaf0-fef0fc52706a\", vm_id=\"273b4068-b40c-42a1-881b-a3d102d427b7\", vnf_id=\"fdbc09db-a607-4511-84e4-a0cd44f9544c\", vnf_member_index=\"vnf\"}];many-to-many matching not allowed: matching labels must be unique on one side"
2024-04-25T02:25:04.062520840Z level=info ts=2024-04-25T02:25:04.062Z caller=main.go:981 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml
2024-04-25T02:25:04.065296348Z level=info ts=2024-04-25T02:25:04.065Z caller=main.go:1012 msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=2.79428ms remote_storage=2.587µs web_handler=17.305µs query_engine=1.733µs scrape=163.941µs scrape_sd=48.917µs notify=64.558µs notify_sd=12.66µs rules=1.811134ms
2024-04-25T02:25:09.110690882Z level=info ts=2024-04-25T02:25:09.110Z caller=main.go:981 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml
2024-04-25T02:25:09.113612459Z level=info ts=2024-04-25T02:25:09.113Z caller=main.go:1012 msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=2.912979ms remote_storage=2.874µs web_handler=947ns query_engine=1.666µs scrape=142.031µs scrape_sd=190.337µs notify=20.093µs notify_sd=21.388µs rules=1.93621ms
2024-04-25T02:27:19.488450354Z level=info ts=2024-04-25T02:27:19.488Z caller=main.go:981 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml
2024-04-25T02:27:19.493052935Z level=info ts=2024-04-25T02:27:19.492Z caller=main.go:1012 msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=4.601416ms remote_storage=5.022µs web_handler=1.357µs query_engine=2.588µs scrape=241.859µs scrape_sd=97.863µs notify=20.933µs notify_sd=20.893µs rules=3.480804ms
2024-04-25T02:37:09.692398305Z level=info ts=2024-04-25T02:37:09.692Z caller=main.go:981 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml
2024-04-25T02:37:09.694761376Z level=info ts=2024-04-25T02:37:09.694Z caller=main.go:1012 msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=2.403084ms remote_storage=2.498µs web_handler=4.584µs query_engine=1.606µs scrape=112.064µs scrape_sd=42.204µs notify=16.899µs notify_sd=13.916µs rules=1.492551ms
2024-04-25T02:39:23.005071261Z level=info ts=2024-04-25T02:39:23.004Z caller=main.go:981 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml
2024-04-25T02:39:23.009383969Z level=info ts=2024-04-25T02:39:23.009Z caller=main.go:1012 msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=4.384482ms remote_storage=3.464µs web_handler=10.331µs query_engine=2.601µs scrape=262.416µs scrape_sd=144.171µs notify=48.365µs notify_sd=32.692µs rules=2.234764ms
2024-04-25T02:39:51.080963142Z level=info ts=2024-04-25T02:39:51.080Z caller=main.go:981 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml
2024-04-25T02:39:51.086129398Z level=info ts=2024-04-25T02:39:51.085Z caller=main.go:1012 msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=5.185348ms remote_storage=3.854µs web_handler=1.221µs query_engine=2.055µs scrape=3.289161ms scrape_sd=60.519µs notify=13.811µs notify_sd=10.104µs rules=1.038296ms
2024-04-25T02:48:13.128802968Z level=warn ts=2024-04-25T02:48:13.128Z caller=manager.go:603 component="rule manager" group=osm_metric_rules msg="Evaluating rule failed" rule="record: vm_status_extended\nexpr: (last_over_time(vm_status[1m]) * on(vm_id, vim_id) group_left(ns_id, vnf_id,\n  vdu_id, project_id, job, vdu_name, vnf_member_index) last_over_time(ns_topology[1m]))\n  or (last_over_time(ns_topology[1m]) * -1)\nlabels:\n  job: osm_prometheus\n" err="found duplicate series for the match group {vim_id=\"dba3fb20-5a29-4377-aaf0-fef0fc52706a\", vm_id=\"107014f7-0e46-4ef1-a425-b1c3bb7bdceb\"} on the right hand-side of the operation: [{__name__=\"ns_topology\", job=\"airflow_osm_ns_topology\", ns_id=\"3b2185a9-935c-4bd2-9953-7798240639db\", project_id=\"df61d3ec-64e7-4cf7-827d-442bd4e0a3e4\", vdu_id=\"0584a0bb-5c98-40c1-a0e0-7b6976a00ce5\", vdu_name=\"slicebasic.slice-middle-ubuntu_slice-VM-0\", vim_id=\"dba3fb20-5a29-4377-aaf0-fef0fc52706a\", vm_id=\"107014f7-0e46-4ef1-a425-b1c3bb7bdceb\", vnf_id=\"f619a73f-1767-49fe-8b55-23134e45ec86\", vnf_member_index=\"middle\"}, {__name__=\"ns_topology\", job=\"airflow_osm_ns_topology\", ns_id=\"3b2185a9-935c-4bd2-9953-7798240639db\", project_id=\"df61d3ec-64e7-4cf7-827d-442bd4e0a3e4\", vdu_id=\"0584a0bb-5c98-40c1-a0e0-7b6976a00ce5\", vdu_name=\"UNKNOWN\", vim_id=\"dba3fb20-5a29-4377-aaf0-fef0fc52706a\", vm_id=\"107014f7-0e46-4ef1-a425-b1c3bb7bdceb\", vnf_id=\"f619a73f-1767-49fe-8b55-23134e45ec86\", vnf_member_index=\"middle\"}];many-to-many matching not allowed: matching labels must be unique on one side"
2024-04-25T02:48:28.128181633Z level=warn ts=2024-04-25T02:48:28.127Z caller=manager.go:603 component="rule manager" group=osm_metric_rules msg="Evaluating rule failed" rule="record: vm_status_extended\nexpr: (last_over_time(vm_status[1m]) * on(vm_id, vim_id) group_left(ns_id, vnf_id,\n  vdu_id, project_id, job, vdu_name, vnf_member_index) last_over_time(ns_topology[1m]))\n  or (last_over_time(ns_topology[1m]) * -1)\nlabels:\n  job: osm_prometheus\n" err="found duplicate series for the match group {vim_id=\"dba3fb20-5a29-4377-aaf0-fef0fc52706a\", vm_id=\"107014f7-0e46-4ef1-a425-b1c3bb7bdceb\"} on the right hand-side of the operation: [{__name__=\"ns_topology\", job=\"airflow_osm_ns_topology\", ns_id=\"3b2185a9-935c-4bd2-9953-7798240639db\", project_id=\"df61d3ec-64e7-4cf7-827d-442bd4e0a3e4\", vdu_id=\"0584a0bb-5c98-40c1-a0e0-7b6976a00ce5\", vdu_name=\"slicebasic.slice-middle-ubuntu_slice-VM-0\", vim_id=\"dba3fb20-5a29-4377-aaf0-fef0fc52706a\", vm_id=\"107014f7-0e46-4ef1-a425-b1c3bb7bdceb\", vnf_id=\"f619a73f-1767-49fe-8b55-23134e45ec86\", vnf_member_index=\"middle\"}, {__name__=\"ns_topology\", job=\"airflow_osm_ns_topology\", ns_id=\"3b2185a9-935c-4bd2-9953-7798240639db\", project_id=\"df61d3ec-64e7-4cf7-827d-442bd4e0a3e4\", vdu_id=\"0584a0bb-5c98-40c1-a0e0-7b6976a00ce5\", vdu_name=\"UNKNOWN\", vim_id=\"dba3fb20-5a29-4377-aaf0-fef0fc52706a\", vm_id=\"107014f7-0e46-4ef1-a425-b1c3bb7bdceb\", vnf_id=\"f619a73f-1767-49fe-8b55-23134e45ec86\", vnf_member_index=\"middle\"}];many-to-many matching not allowed: matching labels must be unique on one side"
2024-04-25T02:52:43.410071773Z level=info ts=2024-04-25T02:52:43.409Z caller=main.go:981 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml
2024-04-25T02:52:43.412748594Z level=info ts=2024-04-25T02:52:43.412Z caller=main.go:1012 msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=2.730257ms remote_storage=2.826µs web_handler=5.789µs query_engine=1.818µs scrape=136.28µs scrape_sd=50.981µs notify=24.393µs notify_sd=13.511µs rules=1.957745ms
2024-04-25T02:52:48.472097110Z level=info ts=2024-04-25T02:52:48.471Z caller=main.go:981 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml
2024-04-25T02:52:48.474396893Z level=info ts=2024-04-25T02:52:48.474Z caller=main.go:1012 msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=2.46691ms remote_storage=2.703µs web_handler=585ns query_engine=1.217µs scrape=108.332µs scrape_sd=31.492µs notify=13.651µs notify_sd=11.68µs rules=1.660873ms
2024-04-25T02:53:37.205427277Z level=info ts=2024-04-25T02:53:37.205Z caller=main.go:981 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml
2024-04-25T02:53:37.207000913Z level=info ts=2024-04-25T02:53:37.206Z caller=main.go:1012 msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=1.611477ms remote_storage=2.491µs web_handler=582ns query_engine=1.183µs scrape=88.128µs scrape_sd=38.53µs notify=41.143µs notify_sd=11.478µs rules=1.039282ms
2024-04-25T02:53:42.258041739Z level=info ts=2024-04-25T02:53:42.257Z caller=main.go:981 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml
2024-04-25T02:53:42.259671167Z level=info ts=2024-04-25T02:53:42.259Z caller=main.go:1012 msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=1.654746ms remote_storage=2.188µs web_handler=520ns query_engine=1.23µs scrape=88.878µs scrape_sd=34.212µs notify=13.589µs notify_sd=14.887µs rules=1.187316ms