2024-04-06T01:27:33.162183447Z nc: kafka (10.98.179.224:9092): Operation timed out 2024-04-06T01:27:37.165168227Z nc: kafka (10.98.179.224:9092): Operation timed out 2024-04-06T01:27:41.172710444Z nc: kafka (10.98.179.224:9092): Operation timed out 2024-04-06T01:27:45.175279809Z nc: kafka (10.98.179.224:9092): Operation timed out 2024-04-06T01:27:49.187926582Z nc: kafka (10.98.179.224:9092): Operation timed out 2024-04-06T01:27:53.194218578Z nc: kafka (10.98.179.224:9092): Operation timed out 2024-04-06T01:27:57.200189748Z nc: kafka (10.98.179.224:9092): Operation timed out 2024-04-06T01:28:01.204543508Z nc: kafka (10.98.179.224:9092): Operation timed out 2024-04-06T01:28:05.210071851Z nc: kafka (10.98.179.224:9092): Operation timed out 2024-04-06T01:28:09.222072675Z nc: kafka (10.98.179.224:9092): Operation timed out 2024-04-06T01:28:13.225071608Z nc: kafka (10.98.179.224:9092): Operation timed out 2024-04-06T01:28:17.234348030Z nc: kafka (10.98.179.224:9092): Operation timed out 2024-04-06T01:28:21.241670939Z nc: kafka (10.98.179.224:9092): Operation timed out 2024-04-06T01:29:22.449192433Z nc: kafka (10.98.179.224:9092): Operation timed out 2024-04-06T01:29:25.466422037Z kafka (10.98.179.224:9092) open 2024-04-06T01:29:25.474528392Z mongodb-k8s (10.244.0.16:27017) open 2024-04-06T01:30:33.317627535Z 04/06/2024 01:30:33 AM - kafka.conn - INFO - [IPv4 ('10.98.179.224', 9092)]>: connecting to kafka:9092 [('10.98.179.224', 9092) IPv4] 2024-04-06T01:30:33.317656278Z 04/06/2024 01:30:33 AM - kafka.conn - INFO - Probing node bootstrap-0 broker version 2024-04-06T01:30:33.317660360Z 04/06/2024 01:30:33 AM - kafka.conn - INFO - [IPv4 ('10.98.179.224', 9092)]>: Connection complete. 2024-04-06T01:30:33.418494424Z 04/06/2024 01:30:33 AM - kafka.conn - INFO - Broker version identified as 2.5.0 2024-04-06T01:30:33.418561462Z 04/06/2024 01:30:33 AM - kafka.conn - INFO - Set configuration api_version=(2, 5, 0) to skip auto check_version requests on startup 2024-04-06T01:30:33.436697410Z 04/06/2024 01:30:33 AM - kafka.conn - INFO - [IPv4 ('10.98.179.224', 9092)]>: connecting to kafka:9092 [('10.98.179.224', 9092) IPv4] 2024-04-06T01:30:33.436707258Z 04/06/2024 01:30:33 AM - kafka.conn - INFO - Probing node bootstrap-0 broker version 2024-04-06T01:30:33.439630055Z 04/06/2024 01:30:33 AM - kafka.conn - INFO - [IPv4 ('10.98.179.224', 9092)]>: Connection complete. 2024-04-06T01:30:33.557914031Z 04/06/2024 01:30:33 AM - kafka.conn - INFO - Broker version identified as 2.5.0 2024-04-06T01:30:33.557929940Z 04/06/2024 01:30:33 AM - kafka.conn - INFO - Set configuration api_version=(2, 5, 0) to skip auto check_version requests on startup 2024-04-06T01:30:33.717949212Z 04/06/2024 01:30:33 AM - kafka.conn - INFO - [IPv4 ('10.244.0.33', 9092)]>: connecting to kafka-controller-1.kafka-controller-headless.osm.svc.cluster.local:9092 [('10.244.0.33', 9092) IPv4] 2024-04-06T01:30:33.717965335Z 04/06/2024 01:30:33 AM - kafka.conn - INFO - [IPv4 ('10.244.0.33', 9092)]>: Connection complete. 2024-04-06T01:30:33.717969662Z 04/06/2024 01:30:33 AM - kafka.conn - INFO - [IPv4 ('10.98.179.224', 9092)]>: Closing connection. 2024-04-06T01:30:33.835089206Z 04/06/2024 01:30:33 AM - kafka.conn - INFO - [IPv4 ('10.244.0.32', 9092)]>: connecting to kafka-controller-0.kafka-controller-headless.osm.svc.cluster.local:9092 [('10.244.0.32', 9092) IPv4] 2024-04-06T01:30:33.835105725Z 04/06/2024 01:30:33 AM - kafka.conn - INFO - [IPv4 ('10.244.0.32', 9092)]>: Connection complete. 2024-04-06T01:30:33.841904108Z 04/06/2024 01:30:33 AM - kafka.producer.record_accumulator - WARNING - Produced messages to topic-partition TopicPartition(topic='alarm_request', partition=0) with base offset -1 log start offset -1 and error None. 2024-04-06T01:30:33.937915950Z 04/06/2024 01:30:33 AM - kafka.cluster - ERROR - Topic users not found in cluster metadata 2024-04-06T01:30:34.046637331Z 04/06/2024 01:30:34 AM - kafka.cluster - ERROR - Topic users not found in cluster metadata 2024-04-06T01:30:34.155074695Z 04/06/2024 01:30:34 AM - kafka.cluster - ERROR - Topic users not found in cluster metadata 2024-04-06T01:30:34.277097185Z 04/06/2024 01:30:34 AM - kafka.cluster - ERROR - Topic users not found in cluster metadata 2024-04-06T01:30:34.381876442Z 04/06/2024 01:30:34 AM - kafka.cluster - ERROR - Topic users not found in cluster metadata 2024-04-06T01:30:34.589882510Z 04/06/2024 01:30:34 AM - kafka.cluster - ERROR - Topic project not found in cluster metadata 2024-04-06T01:30:34.691521153Z 04/06/2024 01:30:34 AM - kafka.cluster - ERROR - Topic project not found in cluster metadata 2024-04-06T01:30:34.853763770Z 04/06/2024 01:30:34 AM - kafka.conn - INFO - [IPv4 ('10.244.0.33', 9092)]>: Closing connection. 2024-04-06T01:30:34.853773869Z 04/06/2024 01:30:34 AM - kafka.conn - INFO - [IPv4 ('10.244.0.32', 9092)]>: Closing connection. 2024-04-06T01:30:34.853779856Z 04/06/2024 01:30:34 AM - osm_mon.cmd.mon_dashboarder - INFO - Starting MON Dashboarder... 2024-04-06T01:30:34.935114978Z 04/06/2024 01:30:34 AM - db - INFO - Connected to database osm version None 2024-04-06T01:30:34.935133301Z 04/06/2024 01:30:34 AM - osm_mon.cmd.mon_dashboarder - INFO - Starting MON kafka Consumer... 2024-04-06T01:30:34.940449183Z 04/06/2024 01:30:34 AM - osm_mon.cmd.mon_dashboarder - INFO - Starting MON Dashboarder... 2024-04-06T01:30:34.989992713Z 04/06/2024 01:30:34 AM - aiokafka.consumer.subscription_state - INFO - Updating subscribed topics to: frozenset({'users', 'project'}) 2024-04-06T01:30:34.990002387Z 04/06/2024 01:30:34 AM - aiokafka.consumer.consumer - INFO - Subscribed to topic(s): {'users', 'project'} 2024-04-06T01:30:35.002055357Z 04/06/2024 01:30:34 AM - aiokafka.consumer.group_coordinator - ERROR - Group Coordinator Request failed: [Error 15] CoordinatorNotAvailableError 2024-04-06T01:30:35.148754959Z 04/06/2024 01:30:35 AM - aiokafka.consumer.group_coordinator - ERROR - Group Coordinator Request failed: [Error 15] CoordinatorNotAvailableError 2024-04-06T01:30:35.332774509Z 04/06/2024 01:30:35 AM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 1 for group mon-consumer 2024-04-06T01:30:35.332790681Z 04/06/2024 01:30:35 AM - aiokafka.consumer.group_coordinator - INFO - Revoking previously assigned partitions set() for group mon-consumer 2024-04-06T01:30:35.332794911Z 04/06/2024 01:30:35 AM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-04-06T01:30:35.401007803Z 04/06/2024 01:30:35 AM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 1)for group mon-consumer. 2024-04-06T01:30:35.514748799Z 04/06/2024 01:30:35 AM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 1 for group mon-consumer 2024-04-06T01:30:35.514765368Z 04/06/2024 01:30:35 AM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-04-06T01:30:35.517745093Z 04/06/2024 01:30:35 AM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 1)for group mon-consumer. 2024-04-06T01:30:35.631233433Z 04/06/2024 01:30:35 AM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 1 for group mon-consumer 2024-04-06T01:30:35.631244003Z 04/06/2024 01:30:35 AM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-04-06T01:30:35.643328258Z 04/06/2024 01:30:35 AM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 1)for group mon-consumer. 2024-04-06T01:30:35.777787904Z 04/06/2024 01:30:35 AM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 1 for group mon-consumer 2024-04-06T01:30:35.777807714Z 04/06/2024 01:30:35 AM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-04-06T01:30:35.777820681Z 04/06/2024 01:30:35 AM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 1)for group mon-consumer. 2024-04-06T01:30:35.888799653Z 04/06/2024 01:30:35 AM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 1 for group mon-consumer 2024-04-06T01:30:35.888828195Z 04/06/2024 01:30:35 AM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-04-06T01:30:35.889760188Z 04/06/2024 01:30:35 AM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 1)for group mon-consumer. 2024-04-06T01:30:35.997597271Z 04/06/2024 01:30:35 AM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 1 for group mon-consumer 2024-04-06T01:30:35.997613183Z 04/06/2024 01:30:35 AM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-04-06T01:30:36.028384111Z 04/06/2024 01:30:36 AM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 1)for group mon-consumer. 2024-04-06T01:30:36.138929047Z 04/06/2024 01:30:36 AM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 1 for group mon-consumer 2024-04-06T01:30:36.138964059Z 04/06/2024 01:30:36 AM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-04-06T01:30:36.146346039Z 04/06/2024 01:30:36 AM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 1)for group mon-consumer. 2024-04-06T01:30:36.265587238Z 04/06/2024 01:30:36 AM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 1 for group mon-consumer 2024-04-06T01:30:36.265599452Z 04/06/2024 01:30:36 AM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-04-06T01:30:36.269940079Z 04/06/2024 01:30:36 AM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 1)for group mon-consumer. 2024-04-06T01:30:36.391111968Z 04/06/2024 01:30:36 AM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 1 for group mon-consumer 2024-04-06T01:30:36.391134658Z 04/06/2024 01:30:36 AM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-04-06T01:30:36.412680238Z 04/06/2024 01:30:36 AM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 1)for group mon-consumer. 2024-04-06T01:30:36.550186023Z 04/06/2024 01:30:36 AM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 1 for group mon-consumer 2024-04-06T01:30:36.558101912Z 04/06/2024 01:30:36 AM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-04-06T01:30:36.570587334Z 04/06/2024 01:30:36 AM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 1)for group mon-consumer. 2024-04-06T01:30:36.676027192Z 04/06/2024 01:30:36 AM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 1 for group mon-consumer 2024-04-06T01:30:36.676043856Z 04/06/2024 01:30:36 AM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-04-06T01:30:36.753263335Z 04/06/2024 01:30:36 AM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 1)for group mon-consumer. 2024-04-06T01:30:36.881157354Z 04/06/2024 01:30:36 AM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 1 for group mon-consumer 2024-04-06T01:30:36.881173602Z 04/06/2024 01:30:36 AM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-04-06T01:30:36.886018586Z 04/06/2024 01:30:36 AM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 1)for group mon-consumer. 2024-04-06T01:30:37.018758207Z 04/06/2024 01:30:37 AM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 1 for group mon-consumer 2024-04-06T01:30:37.018828637Z 04/06/2024 01:30:37 AM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-04-06T01:30:37.042680182Z 04/06/2024 01:30:37 AM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 1)for group mon-consumer. 2024-04-06T01:30:37.171016697Z 04/06/2024 01:30:37 AM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 1 for group mon-consumer 2024-04-06T01:30:37.171036576Z 04/06/2024 01:30:37 AM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-04-06T01:30:37.181926359Z 04/06/2024 01:30:37 AM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 1)for group mon-consumer. 2024-04-06T01:30:37.295691438Z 04/06/2024 01:30:37 AM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 1 for group mon-consumer 2024-04-06T01:30:37.302593127Z 04/06/2024 01:30:37 AM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-04-06T01:30:37.302608602Z 04/06/2024 01:30:37 AM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 1)for group mon-consumer. 2024-04-06T01:30:37.419607839Z 04/06/2024 01:30:37 AM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 1 for group mon-consumer 2024-04-06T01:30:37.419921749Z 04/06/2024 01:30:37 AM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-04-06T01:30:37.439501172Z 04/06/2024 01:30:37 AM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 1)for group mon-consumer. 2024-04-06T01:30:37.559698851Z 04/06/2024 01:30:37 AM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 1 for group mon-consumer 2024-04-06T01:30:37.564864786Z 04/06/2024 01:30:37 AM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-04-06T01:30:37.584016277Z 04/06/2024 01:30:37 AM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 1)for group mon-consumer. 2024-04-06T01:30:37.713895788Z 04/06/2024 01:30:37 AM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 1 for group mon-consumer 2024-04-06T01:30:37.713906345Z 04/06/2024 01:30:37 AM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-04-06T01:30:37.718594423Z 04/06/2024 01:30:37 AM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 1)for group mon-consumer. 2024-04-06T01:30:37.900567142Z 04/06/2024 01:30:37 AM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 1 for group mon-consumer 2024-04-06T01:30:37.900583757Z 04/06/2024 01:30:37 AM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-04-06T01:30:37.908286577Z 04/06/2024 01:30:37 AM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 1)for group mon-consumer. 2024-04-06T01:30:38.017876042Z 04/06/2024 01:30:38 AM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 1 for group mon-consumer 2024-04-06T01:30:38.017893858Z 04/06/2024 01:30:38 AM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-04-06T01:30:38.037214034Z 04/06/2024 01:30:38 AM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 1)for group mon-consumer. 2024-04-06T01:30:38.155430964Z 04/06/2024 01:30:38 AM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 1 for group mon-consumer 2024-04-06T01:30:38.155447834Z 04/06/2024 01:30:38 AM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-04-06T01:30:38.254527301Z 04/06/2024 01:30:38 AM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 1)for group mon-consumer. 2024-04-06T01:30:38.481414937Z 04/06/2024 01:30:38 AM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 1 for group mon-consumer 2024-04-06T01:30:38.481454311Z 04/06/2024 01:30:38 AM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-04-06T01:30:38.733060712Z 04/06/2024 01:30:38 AM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 1)for group mon-consumer. 2024-04-06T01:30:38.872129744Z 04/06/2024 01:30:38 AM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 1 for group mon-consumer 2024-04-06T01:30:38.872145997Z 04/06/2024 01:30:38 AM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-04-06T01:30:38.910993475Z 04/06/2024 01:30:38 AM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 1)for group mon-consumer. 2024-04-06T01:30:39.025190357Z 04/06/2024 01:30:39 AM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 1 for group mon-consumer 2024-04-06T01:30:39.025212289Z 04/06/2024 01:30:39 AM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-04-06T01:30:39.034744580Z 04/06/2024 01:30:39 AM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 1)for group mon-consumer. 2024-04-06T01:30:39.172290064Z 04/06/2024 01:30:39 AM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 1 for group mon-consumer 2024-04-06T01:30:39.175123309Z 04/06/2024 01:30:39 AM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-04-06T01:30:42.266056672Z 04/06/2024 01:30:42 AM - aiokafka.consumer.group_coordinator - INFO - Joined group 'mon-consumer' (generation 1) with member_id aiokafka-0.8.1-7893282c-7a34-46c2-9aea-d7092943d786 2024-04-06T01:30:42.266159415Z 04/06/2024 01:30:42 AM - aiokafka.consumer.group_coordinator - INFO - Elected group leader -- performing partition assignments using roundrobin 2024-04-06T01:30:42.402028789Z 04/06/2024 01:30:42 AM - aiokafka.consumer.group_coordinator - INFO - Successfully synced group mon-consumer with generation 1 2024-04-06T01:30:42.402047219Z 04/06/2024 01:30:42 AM - aiokafka.consumer.group_coordinator - INFO - Setting newly assigned partitions {TopicPartition(topic='project', partition=0), TopicPartition(topic='users', partition=0)} for group mon-consumer 2024-04-06T01:31:05.473082111Z 04/06/2024 01:31:05 AM - osm_mon.dashboarder.backends.grafana - INFO - Dashboard admin is created in Grafana 2024-04-06T01:37:37.081914815Z 04/06/2024 01:37:37 AM - osm_mon.dashboarder.backends.grafana - INFO - Datasource prom-operator-osm is created in Grafana 2024-04-06T01:37:37.082003294Z 04/06/2024 01:37:37 AM - osm_mon.dashboarder.backends.grafana - INFO - ************* response: {'_content': b'{"datasource":{"id":3,"uid":"c78bfbe9-5804-46de-992f-5281d47ce918","orgId":1,"name":"prom-operator-osm","type":"prometheus","typeLogoUrl":"public/app/plugins/datasource/prometheus/img/prometheus_logo.svg","access":"proxy","url":"http://172.21.248.125:9090","user":"","database":"","basicAuth":false,"basicAuthUser":"","withCredentials":false,"isDefault":false,"jsonData":{},"secureJsonFields":{},"version":1,"readOnly":false},"id":3,"message":"Datasource added","name":"prom-operator-osm"}', '_content_consumed': True, '_next': None, 'status_code': 200, 'headers': {'Cache-Control': 'no-store', 'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff', 'X-Frame-Options': 'deny', 'X-Xss-Protection': '1; mode=block', 'Date': 'Sat, 06 Apr 2024 01:37:37 GMT', 'Content-Length': '489'}, 'raw': , 'url': 'http://grafana:3000/api/datasources', 'encoding': 'utf-8', 'history': [], 'reason': 'OK', 'cookies': , 'elapsed': datetime.timedelta(microseconds=63629), 'request': , 'connection': } 2024-04-06T01:37:37.339648418Z 04/06/2024 01:37:37 AM - osm_mon.dashboarder.backends.grafana - INFO - Dashboard osm is created in Grafana 2024-04-06T01:43:38.828998022Z 04/06/2024 01:43:38 AM - osm_mon.dashboarder.backends.grafana - INFO - Dashboard admin is created in Grafana 2024-04-06T01:45:30.478194199Z 04/06/2024 01:45:30 AM - osm_mon.dashboarder.backends.grafana - INFO - New user basic_15_test_user created in Grafana 2024-04-06T01:45:33.750754623Z 04/06/2024 01:45:33 AM - osm_mon.dashboarder.backends.grafana - INFO - User basic_15_test_user is assigned Admin permission 2024-04-06T01:45:38.587121388Z 04/06/2024 01:45:38 AM - osm_mon.dashboarder.dashboarder - ERROR - Exception processing message: 2024-04-06T01:45:38.587139958Z Traceback (most recent call last): 2024-04-06T01:45:38.587143844Z File "/usr/lib/python3/dist-packages/osm_mon/dashboarder/dashboarder.py", line 116, in _user_msg 2024-04-06T01:45:38.587147492Z project_data = values["changes"]["add_project_role_mappings"] 2024-04-06T01:45:38.587150759Z KeyError: 'add_project_role_mappings' 2024-04-06T01:45:39.850282838Z 04/06/2024 01:45:39 AM - osm_mon.dashboarder.backends.grafana - INFO - New team basic_15_test_project created in Grafana 2024-04-06T01:45:41.528115007Z 04/06/2024 01:45:41 AM - osm_mon.dashboarder.backends.grafana - INFO - Grafana team updated {"message":"Team updated"} 2024-04-06T01:45:49.772302047Z 04/06/2024 01:45:49 AM - osm_mon.dashboarder.backends.grafana - INFO - User basic_15_test_user deleted in Grafana 2024-04-06T01:45:51.405658334Z 04/06/2024 01:45:51 AM - osm_mon.dashboarder.backends.grafana - INFO - Team basic_15_project_test deleted in Grafana 2024-04-06T01:46:09.622157619Z 04/06/2024 01:46:09 AM - osm_mon.dashboarder.backends.grafana - INFO - Dashboard admin is created in Grafana 2024-04-06T02:54:24.866085890Z 04/06/2024 02:54:24 AM - osm_mon.dashboarder.backends.grafana - INFO - Datasource prom-operator-k8s02 is created in Grafana 2024-04-06T02:54:24.866185740Z 04/06/2024 02:54:24 AM - osm_mon.dashboarder.backends.grafana - INFO - ************* response: {'_content': b'{"datasource":{"id":4,"uid":"b2d57f3c-c176-42ba-b311-7160f62b9916","orgId":1,"name":"prom-operator-k8s02","type":"prometheus","typeLogoUrl":"public/app/plugins/datasource/prometheus/img/prometheus_logo.svg","access":"proxy","url":"http://172.21.248.125:9090","user":"","database":"","basicAuth":false,"basicAuthUser":"","withCredentials":false,"isDefault":false,"jsonData":{},"secureJsonFields":{},"version":1,"readOnly":false},"id":4,"message":"Datasource added","name":"prom-operator-k8s02"}', '_content_consumed': True, '_next': None, 'status_code': 200, 'headers': {'Cache-Control': 'no-store', 'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff', 'X-Frame-Options': 'deny', 'X-Xss-Protection': '1; mode=block', 'Date': 'Sat, 06 Apr 2024 02:54:24 GMT', 'Content-Length': '493'}, 'raw': , 'url': 'http://grafana:3000/api/datasources', 'encoding': 'utf-8', 'history': [], 'reason': 'OK', 'cookies': , 'elapsed': datetime.timedelta(microseconds=58896), 'request': , 'connection': } 2024-04-06T02:54:25.032383910Z 04/06/2024 02:54:25 AM - osm_mon.dashboarder.backends.grafana - INFO - Dashboard k8s02 is created in Grafana 2024-04-06T02:58:56.099128092Z 04/06/2024 02:58:56 AM - osm_mon.dashboarder.backends.grafana - INFO - Datasource prom-operator-k8s07 is created in Grafana 2024-04-06T02:58:56.099918877Z 04/06/2024 02:58:56 AM - osm_mon.dashboarder.backends.grafana - INFO - ************* response: {'_content': b'{"datasource":{"id":5,"uid":"d6e9c762-e728-46c2-84e1-99fe72f68799","orgId":1,"name":"prom-operator-k8s07","type":"prometheus","typeLogoUrl":"public/app/plugins/datasource/prometheus/img/prometheus_logo.svg","access":"proxy","url":"http://172.21.248.125:9090","user":"","database":"","basicAuth":false,"basicAuthUser":"","withCredentials":false,"isDefault":false,"jsonData":{},"secureJsonFields":{},"version":1,"readOnly":false},"id":5,"message":"Datasource added","name":"prom-operator-k8s07"}', '_content_consumed': True, '_next': None, 'status_code': 200, 'headers': {'Cache-Control': 'no-store', 'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff', 'X-Frame-Options': 'deny', 'X-Xss-Protection': '1; mode=block', 'Date': 'Sat, 06 Apr 2024 02:58:56 GMT', 'Content-Length': '493'}, 'raw': , 'url': 'http://grafana:3000/api/datasources', 'encoding': 'utf-8', 'history': [], 'reason': 'OK', 'cookies': , 'elapsed': datetime.timedelta(microseconds=38105), 'request': , 'connection': } 2024-04-06T02:58:56.266716826Z 04/06/2024 02:58:56 AM - osm_mon.dashboarder.backends.grafana - INFO - Dashboard k8s07 is created in Grafana 2024-04-06T03:07:11.950808123Z 04/06/2024 03:07:11 AM - osm_mon.dashboarder.backends.grafana - INFO - New team quotas_01_proj_1 created in Grafana 2024-04-06T03:07:18.346062967Z 04/06/2024 03:07:18 AM - osm_mon.dashboarder.backends.grafana - INFO - New team quotas_01_proj_2 created in Grafana 2024-04-06T03:07:24.784144816Z 04/06/2024 03:07:24 AM - osm_mon.dashboarder.backends.grafana - INFO - New user quotas_01_user created in Grafana 2024-04-06T03:07:24.938548180Z 04/06/2024 03:07:24 AM - osm_mon.dashboarder.backends.grafana - INFO - User quotas_01_user is assigned Editor permission 2024-04-06T03:07:25.116199021Z 04/06/2024 03:07:25 AM - osm_mon.dashboarder.backends.grafana - INFO - User quotas_01_user is assigned Editor permission 2024-04-06T03:07:28.202385186Z 04/06/2024 03:07:28 AM - osm_mon.dashboarder.backends.grafana - INFO - Dashboard folder quotas_01_proj_1 created 2024-04-06T03:07:28.383521165Z 04/06/2024 03:07:28 AM - osm_mon.dashboarder.backends.grafana - INFO - Dashboard quotas_01_proj_1 is created in Grafana 2024-04-06T03:07:28.434003287Z 04/06/2024 03:07:28 AM - osm_mon.dashboarder.backends.grafana - INFO - Dashboard folder quotas_01_proj_2 created 2024-04-06T03:07:28.605990901Z 04/06/2024 03:07:28 AM - osm_mon.dashboarder.backends.grafana - INFO - Dashboard quotas_01_proj_2 is created in Grafana 2024-04-06T03:07:29.574351755Z 04/06/2024 03:07:29 AM - osm_mon.dashboarder.dashboarder - ERROR - Exception processing message: 2024-04-06T03:07:29.574393791Z Traceback (most recent call last): 2024-04-06T03:07:29.574404505Z File "/usr/lib/python3/dist-packages/osm_mon/dashboarder/dashboarder.py", line 116, in _user_msg 2024-04-06T03:07:29.574414289Z project_data = values["changes"]["add_project_role_mappings"] 2024-04-06T03:07:29.574423293Z KeyError: 'add_project_role_mappings' 2024-04-06T03:07:31.337660581Z 04/06/2024 03:07:31 AM - osm_mon.dashboarder.backends.grafana - INFO - Grafana team updated {"message":"Team updated"} 2024-04-06T03:07:41.090179991Z 04/06/2024 03:07:41 AM - osm_mon.dashboarder.dashboarder - ERROR - Exception processing message: 2024-04-06T03:07:41.090257201Z Traceback (most recent call last): 2024-04-06T03:07:41.090265817Z File "/usr/lib/python3/dist-packages/osm_mon/dashboarder/dashboarder.py", line 129, in _user_msg 2024-04-06T03:07:41.090272681Z project_new_name = values["changes"]["name"] 2024-04-06T03:07:41.090279570Z KeyError: 'name' 2024-04-06T03:08:11.711204398Z 04/06/2024 03:08:11 AM - osm_mon.dashboarder.backends.grafana - INFO - Team quotas_01_proj_2 deleted in Grafana 2024-04-06T03:08:13.522340504Z 04/06/2024 03:08:13 AM - osm_mon.dashboarder.backends.grafana - INFO - User quotas_01_user deleted in Grafana 2024-04-06T03:08:15.061003529Z 04/06/2024 03:08:15 AM - osm_mon.dashboarder.backends.grafana - INFO - Team quotas_01_proj_3 deleted in Grafana 2024-04-06T03:08:29.004677490Z 04/06/2024 03:08:29 AM - osm_mon.dashboarder.backends.grafana - INFO - Dashboard admin is created in Grafana 2024-04-06T03:17:01.173140092Z 04/06/2024 03:17:01 AM - osm_mon.dashboarder.backends.grafana - INFO - Dashboard admin is created in Grafana 2024-04-06T03:33:05.270189465Z 04/06/2024 03:33:05 AM - osm_mon.dashboarder.backends.grafana - INFO - Dashboard admin is created in Grafana 2024-04-06T03:57:10.852978125Z 04/06/2024 03:57:10 AM - osm_mon.dashboarder.backends.grafana - INFO - Dashboard admin is created in Grafana