2024-04-23T01:22:39.401159191Z nc: kafka (10.100.161.242:9092): Operation timed out 2024-04-23T01:22:43.405346424Z nc: kafka (10.100.161.242:9092): Operation timed out 2024-04-23T01:22:47.411279048Z nc: kafka (10.100.161.242:9092): Operation timed out 2024-04-23T01:22:51.415347353Z nc: kafka (10.100.161.242:9092): Operation timed out 2024-04-23T01:22:55.423258592Z nc: kafka (10.100.161.242:9092): Operation timed out 2024-04-23T01:22:59.434082178Z nc: kafka (10.100.161.242:9092): Operation timed out 2024-04-23T01:23:03.435312137Z nc: kafka (10.100.161.242:9092): Operation timed out 2024-04-23T01:23:07.437145115Z nc: kafka (10.100.161.242:9092): Operation timed out 2024-04-23T01:23:11.445059157Z nc: kafka (10.100.161.242:9092): Operation timed out 2024-04-23T01:23:15.450601360Z nc: kafka (10.100.161.242:9092): Operation timed out 2024-04-23T01:23:19.459462799Z nc: kafka (10.100.161.242:9092): Operation timed out 2024-04-23T01:24:28.752690701Z kafka (10.100.161.242:9092) open 2024-04-23T01:24:28.760938902Z mongodb-k8s (10.244.0.24:27017) open 2024-04-23T01:25:24.882924173Z 04/23/2024 01:25:24 AM - kafka.conn - INFO - [IPv4 ('10.100.161.242', 9092)]>: connecting to kafka:9092 [('10.100.161.242', 9092) IPv4] 2024-04-23T01:25:24.884577237Z 04/23/2024 01:25:24 AM - kafka.conn - INFO - Probing node bootstrap-0 broker version 2024-04-23T01:25:24.884598222Z 04/23/2024 01:25:24 AM - kafka.conn - INFO - [IPv4 ('10.100.161.242', 9092)]>: Connection complete. 2024-04-23T01:25:24.994364462Z 04/23/2024 01:25:24 AM - kafka.conn - INFO - Broker version identified as 2.5.0 2024-04-23T01:25:24.994394068Z 04/23/2024 01:25:24 AM - kafka.conn - INFO - Set configuration api_version=(2, 5, 0) to skip auto check_version requests on startup 2024-04-23T01:25:25.012670046Z 04/23/2024 01:25:25 AM - kafka.conn - INFO - [IPv4 ('10.100.161.242', 9092)]>: connecting to kafka:9092 [('10.100.161.242', 9092) IPv4] 2024-04-23T01:25:25.012687065Z 04/23/2024 01:25:25 AM - kafka.conn - INFO - Probing node bootstrap-0 broker version 2024-04-23T01:25:25.012692696Z 04/23/2024 01:25:25 AM - kafka.conn - INFO - [IPv4 ('10.100.161.242', 9092)]>: Connection complete. 2024-04-23T01:25:25.119889156Z 04/23/2024 01:25:25 AM - kafka.conn - INFO - Broker version identified as 2.5.0 2024-04-23T01:25:25.120297076Z 04/23/2024 01:25:25 AM - kafka.conn - INFO - Set configuration api_version=(2, 5, 0) to skip auto check_version requests on startup 2024-04-23T01:25:25.297047069Z 04/23/2024 01:25:25 AM - kafka.conn - INFO - [IPv4 ('10.244.0.28', 9092)]>: connecting to kafka-controller-0.kafka-controller-headless.osm.svc.cluster.local:9092 [('10.244.0.28', 9092) IPv4] 2024-04-23T01:25:25.297511992Z 04/23/2024 01:25:25 AM - kafka.conn - INFO - [IPv4 ('10.244.0.28', 9092)]>: Connection complete. 2024-04-23T01:25:25.297725273Z 04/23/2024 01:25:25 AM - kafka.conn - INFO - [IPv4 ('10.100.161.242', 9092)]>: Closing connection. 2024-04-23T01:25:25.926969254Z 04/23/2024 01:25:25 AM - kafka.conn - INFO - [IPv4 ('10.244.0.36', 9092)]>: connecting to kafka-controller-2.kafka-controller-headless.osm.svc.cluster.local:9092 [('10.244.0.36', 9092) IPv4] 2024-04-23T01:25:25.926982059Z 04/23/2024 01:25:25 AM - kafka.conn - INFO - [IPv4 ('10.244.0.36', 9092)]>: Connection complete. 2024-04-23T01:25:26.042378009Z 04/23/2024 01:25:26 AM - kafka.cluster - ERROR - Topic users not found in cluster metadata 2024-04-23T01:25:26.168576772Z 04/23/2024 01:25:26 AM - kafka.cluster - ERROR - Topic users not found in cluster metadata 2024-04-23T01:25:26.310719560Z 04/23/2024 01:25:26 AM - kafka.cluster - ERROR - Topic users not found in cluster metadata 2024-04-23T01:25:26.426491650Z 04/23/2024 01:25:26 AM - kafka.conn - INFO - [IPv4 ('10.244.0.40', 9092)]>: connecting to kafka-controller-1.kafka-controller-headless.osm.svc.cluster.local:9092 [('10.244.0.40', 9092) IPv4] 2024-04-23T01:25:26.426886833Z 04/23/2024 01:25:26 AM - kafka.conn - INFO - [IPv4 ('10.244.0.40', 9092)]>: Connection complete. 2024-04-23T01:25:26.546561172Z 04/23/2024 01:25:26 AM - kafka.cluster - ERROR - Topic project not found in cluster metadata 2024-04-23T01:25:26.678860515Z 04/23/2024 01:25:26 AM - kafka.conn - INFO - [IPv4 ('10.244.0.28', 9092)]>: Closing connection. 2024-04-23T01:25:26.678879999Z 04/23/2024 01:25:26 AM - kafka.conn - INFO - [IPv4 ('10.244.0.36', 9092)]>: Closing connection. 2024-04-23T01:25:26.678883121Z 04/23/2024 01:25:26 AM - kafka.conn - INFO - [IPv4 ('10.244.0.40', 9092)]>: Closing connection. 2024-04-23T01:25:26.678888157Z 04/23/2024 01:25:26 AM - osm_mon.cmd.mon_dashboarder - INFO - Starting MON Dashboarder... 2024-04-23T01:25:26.793790823Z 04/23/2024 01:25:26 AM - db - INFO - Connected to database osm version 1.2 2024-04-23T01:25:26.793806665Z 04/23/2024 01:25:26 AM - osm_mon.cmd.mon_dashboarder - INFO - Starting MON kafka Consumer... 2024-04-23T01:25:26.797149464Z 04/23/2024 01:25:26 AM - osm_mon.cmd.mon_dashboarder - INFO - Starting MON Dashboarder... 2024-04-23T01:25:26.884104414Z 04/23/2024 01:25:26 AM - aiokafka.consumer.subscription_state - INFO - Updating subscribed topics to: frozenset({'users', 'project'}) 2024-04-23T01:25:26.884119847Z 04/23/2024 01:25:26 AM - aiokafka.consumer.consumer - INFO - Subscribed to topic(s): {'users', 'project'} 2024-04-23T01:25:26.940805758Z 04/23/2024 01:25:26 AM - aiokafka.consumer.group_coordinator - ERROR - Group Coordinator Request failed: [Error 15] CoordinatorNotAvailableError 2024-04-23T01:25:27.105681348Z 04/23/2024 01:25:27 AM - aiokafka.consumer.group_coordinator - ERROR - Group Coordinator Request failed: [Error 15] CoordinatorNotAvailableError 2024-04-23T01:25:27.228065746Z 04/23/2024 01:25:27 AM - aiokafka.consumer.group_coordinator - ERROR - Group Coordinator Request failed: [Error 15] CoordinatorNotAvailableError 2024-04-23T01:25:27.334186006Z 04/23/2024 01:25:27 AM - aiokafka.consumer.group_coordinator - ERROR - Group Coordinator Request failed: [Error 15] CoordinatorNotAvailableError 2024-04-23T01:25:27.513830477Z 04/23/2024 01:25:27 AM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 1 for group mon-consumer 2024-04-23T01:25:27.513853131Z 04/23/2024 01:25:27 AM - aiokafka.consumer.group_coordinator - INFO - Revoking previously assigned partitions set() for group mon-consumer 2024-04-23T01:25:27.513908262Z 04/23/2024 01:25:27 AM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-04-23T01:25:27.542172922Z 04/23/2024 01:25:27 AM - osm_mon.dashboarder.backends.grafana - INFO - Dashboard admin is created in Grafana 2024-04-23T01:25:27.562221679Z 04/23/2024 01:25:27 AM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 1)for group mon-consumer. 2024-04-23T01:25:27.792230036Z 04/23/2024 01:25:27 AM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 1 for group mon-consumer 2024-04-23T01:25:27.793090143Z 04/23/2024 01:25:27 AM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-04-23T01:25:27.802808668Z 04/23/2024 01:25:27 AM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 1)for group mon-consumer. 2024-04-23T01:25:27.958795276Z 04/23/2024 01:25:27 AM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 1 for group mon-consumer 2024-04-23T01:25:27.958813788Z 04/23/2024 01:25:27 AM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-04-23T01:25:27.982726536Z 04/23/2024 01:25:27 AM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 1)for group mon-consumer. 2024-04-23T01:25:28.091531850Z 04/23/2024 01:25:28 AM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 1 for group mon-consumer 2024-04-23T01:25:28.091544646Z 04/23/2024 01:25:28 AM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-04-23T01:25:28.110327037Z 04/23/2024 01:25:28 AM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 1)for group mon-consumer. 2024-04-23T01:25:28.233354988Z 04/23/2024 01:25:28 AM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 1 for group mon-consumer 2024-04-23T01:25:28.233372444Z 04/23/2024 01:25:28 AM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-04-23T01:25:28.234929403Z 04/23/2024 01:25:28 AM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 1)for group mon-consumer. 2024-04-23T01:25:28.349654272Z 04/23/2024 01:25:28 AM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 1 for group mon-consumer 2024-04-23T01:25:28.349671375Z 04/23/2024 01:25:28 AM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-04-23T01:25:28.351472633Z 04/23/2024 01:25:28 AM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 1)for group mon-consumer. 2024-04-23T01:25:28.461433516Z 04/23/2024 01:25:28 AM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 1 for group mon-consumer 2024-04-23T01:25:28.461443444Z 04/23/2024 01:25:28 AM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-04-23T01:25:28.573275341Z 04/23/2024 01:25:28 AM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 1)for group mon-consumer. 2024-04-23T01:25:28.724577178Z 04/23/2024 01:25:28 AM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 1 for group mon-consumer 2024-04-23T01:25:28.724643710Z 04/23/2024 01:25:28 AM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-04-23T01:25:28.725995690Z 04/23/2024 01:25:28 AM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 1)for group mon-consumer. 2024-04-23T01:25:28.830862820Z 04/23/2024 01:25:28 AM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 1 for group mon-consumer 2024-04-23T01:25:28.830903097Z 04/23/2024 01:25:28 AM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-04-23T01:25:28.836306208Z 04/23/2024 01:25:28 AM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 1)for group mon-consumer. 2024-04-23T01:25:28.961562740Z 04/23/2024 01:25:28 AM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 1 for group mon-consumer 2024-04-23T01:25:28.961581003Z 04/23/2024 01:25:28 AM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-04-23T01:25:28.967249397Z 04/23/2024 01:25:28 AM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 1)for group mon-consumer. 2024-04-23T01:25:29.080152416Z 04/23/2024 01:25:29 AM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 1 for group mon-consumer 2024-04-23T01:25:29.080218702Z 04/23/2024 01:25:29 AM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-04-23T01:25:29.100215260Z 04/23/2024 01:25:29 AM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 1)for group mon-consumer. 2024-04-23T01:25:29.205557403Z 04/23/2024 01:25:29 AM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 1 for group mon-consumer 2024-04-23T01:25:29.205643525Z 04/23/2024 01:25:29 AM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-04-23T01:25:29.240651673Z 04/23/2024 01:25:29 AM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 1)for group mon-consumer. 2024-04-23T01:25:29.360148964Z 04/23/2024 01:25:29 AM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 1 for group mon-consumer 2024-04-23T01:25:29.360197447Z 04/23/2024 01:25:29 AM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-04-23T01:25:29.366727264Z 04/23/2024 01:25:29 AM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 1)for group mon-consumer. 2024-04-23T01:25:29.477450961Z 04/23/2024 01:25:29 AM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 1 for group mon-consumer 2024-04-23T01:25:29.477467662Z 04/23/2024 01:25:29 AM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-04-23T01:25:29.493053138Z 04/23/2024 01:25:29 AM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 1)for group mon-consumer. 2024-04-23T01:25:29.675445926Z 04/23/2024 01:25:29 AM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 1 for group mon-consumer 2024-04-23T01:25:29.675501650Z 04/23/2024 01:25:29 AM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-04-23T01:25:29.677780510Z 04/23/2024 01:25:29 AM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 1)for group mon-consumer. 2024-04-23T01:25:29.786775674Z 04/23/2024 01:25:29 AM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 1 for group mon-consumer 2024-04-23T01:25:29.786791162Z 04/23/2024 01:25:29 AM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-04-23T01:25:29.798494731Z 04/23/2024 01:25:29 AM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 1)for group mon-consumer. 2024-04-23T01:25:29.906081555Z 04/23/2024 01:25:29 AM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 1 for group mon-consumer 2024-04-23T01:25:29.906115176Z 04/23/2024 01:25:29 AM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-04-23T01:25:29.920323414Z 04/23/2024 01:25:29 AM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 1)for group mon-consumer. 2024-04-23T01:25:30.039305108Z 04/23/2024 01:25:30 AM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 1 for group mon-consumer 2024-04-23T01:25:30.039324087Z 04/23/2024 01:25:30 AM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-04-23T01:25:30.054339605Z 04/23/2024 01:25:30 AM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 1)for group mon-consumer. 2024-04-23T01:25:30.171920804Z 04/23/2024 01:25:30 AM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 1 for group mon-consumer 2024-04-23T01:25:30.171933722Z 04/23/2024 01:25:30 AM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-04-23T01:25:30.193703133Z 04/23/2024 01:25:30 AM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 1)for group mon-consumer. 2024-04-23T01:25:30.320008777Z 04/23/2024 01:25:30 AM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 1 for group mon-consumer 2024-04-23T01:25:30.320024823Z 04/23/2024 01:25:30 AM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-04-23T01:25:33.499243095Z 04/23/2024 01:25:33 AM - aiokafka.consumer.group_coordinator - INFO - Joined group 'mon-consumer' (generation 1) with member_id aiokafka-0.8.1-f7a28873-71fc-4ff4-be8e-ad0ef05cd4ff 2024-04-23T01:25:33.499277481Z 04/23/2024 01:25:33 AM - aiokafka.consumer.group_coordinator - INFO - Elected group leader -- performing partition assignments using roundrobin 2024-04-23T01:25:33.626686851Z 04/23/2024 01:25:33 AM - aiokafka.consumer.group_coordinator - INFO - Successfully synced group mon-consumer with generation 1 2024-04-23T01:25:33.626884268Z 04/23/2024 01:25:33 AM - aiokafka.consumer.group_coordinator - INFO - Setting newly assigned partitions {TopicPartition(topic='users', partition=0), TopicPartition(topic='project', partition=0)} for group mon-consumer 2024-04-23T01:32:29.350459979Z 04/23/2024 01:32:29 AM - osm_mon.dashboarder.backends.grafana - INFO - Datasource prom-operator-osm is created in Grafana 2024-04-23T01:32:29.351498715Z 04/23/2024 01:32:29 AM - osm_mon.dashboarder.backends.grafana - INFO - ************* response: {'_content': b'{"datasource":{"id":3,"uid":"b9a84c77-6fba-4414-97df-569a914dc4c6","orgId":1,"name":"prom-operator-osm","type":"prometheus","typeLogoUrl":"public/app/plugins/datasource/prometheus/img/prometheus_logo.svg","access":"proxy","url":"http://172.21.248.125:9090","user":"","database":"","basicAuth":false,"basicAuthUser":"","withCredentials":false,"isDefault":false,"jsonData":{},"secureJsonFields":{},"version":1,"readOnly":false},"id":3,"message":"Datasource added","name":"prom-operator-osm"}', '_content_consumed': True, '_next': None, 'status_code': 200, 'headers': {'Cache-Control': 'no-store', 'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff', 'X-Frame-Options': 'deny', 'X-Xss-Protection': '1; mode=block', 'Date': 'Tue, 23 Apr 2024 01:32:29 GMT', 'Content-Length': '489'}, 'raw': , 'url': 'http://grafana:3000/api/datasources', 'encoding': 'utf-8', 'history': [], 'reason': 'OK', 'cookies': , 'elapsed': datetime.timedelta(microseconds=45787), 'request': , 'connection': } 2024-04-23T01:32:29.569310974Z 04/23/2024 01:32:29 AM - osm_mon.dashboarder.backends.grafana - INFO - Dashboard osm is created in Grafana 2024-04-23T01:38:31.168128165Z 04/23/2024 01:38:31 AM - osm_mon.dashboarder.backends.grafana - INFO - Dashboard admin is created in Grafana 2024-04-23T01:40:34.307092518Z 04/23/2024 01:40:34 AM - osm_mon.dashboarder.backends.grafana - INFO - New user basic_15_test_user created in Grafana 2024-04-23T01:40:37.573919676Z 04/23/2024 01:40:37 AM - osm_mon.dashboarder.backends.grafana - INFO - User basic_15_test_user is assigned Admin permission 2024-04-23T01:40:42.556322123Z 04/23/2024 01:40:42 AM - osm_mon.dashboarder.dashboarder - ERROR - Exception processing message: 2024-04-23T01:40:42.556344624Z Traceback (most recent call last): 2024-04-23T01:40:42.556348333Z File "/usr/lib/python3/dist-packages/osm_mon/dashboarder/dashboarder.py", line 116, in _user_msg 2024-04-23T01:40:42.556352074Z project_data = values["changes"]["add_project_role_mappings"] 2024-04-23T01:40:42.556355333Z KeyError: 'add_project_role_mappings' 2024-04-23T01:40:43.875777897Z 04/23/2024 01:40:43 AM - osm_mon.dashboarder.backends.grafana - INFO - New team basic_15_test_project created in Grafana 2024-04-23T01:40:45.495018015Z 04/23/2024 01:40:45 AM - osm_mon.dashboarder.backends.grafana - INFO - Grafana team updated {"message":"Team updated"} 2024-04-23T01:40:53.653828705Z 04/23/2024 01:40:53 AM - osm_mon.dashboarder.backends.grafana - INFO - User basic_15_test_user deleted in Grafana 2024-04-23T01:40:55.361463652Z 04/23/2024 01:40:55 AM - osm_mon.dashboarder.backends.grafana - INFO - Team basic_15_project_test deleted in Grafana 2024-04-23T01:41:32.322059321Z 04/23/2024 01:41:32 AM - osm_mon.dashboarder.backends.grafana - INFO - Dashboard admin is created in Grafana 2024-04-23T02:50:18.472403347Z 04/23/2024 02:50:18 AM - osm_mon.dashboarder.backends.grafana - INFO - Datasource prom-operator-k8s02 is created in Grafana 2024-04-23T02:50:18.472635536Z 04/23/2024 02:50:18 AM - osm_mon.dashboarder.backends.grafana - INFO - ************* response: {'_content': b'{"datasource":{"id":4,"uid":"a7f49a61-95a8-4069-94ab-16b84b75da83","orgId":1,"name":"prom-operator-k8s02","type":"prometheus","typeLogoUrl":"public/app/plugins/datasource/prometheus/img/prometheus_logo.svg","access":"proxy","url":"http://172.21.248.125:9090","user":"","database":"","basicAuth":false,"basicAuthUser":"","withCredentials":false,"isDefault":false,"jsonData":{},"secureJsonFields":{},"version":1,"readOnly":false},"id":4,"message":"Datasource added","name":"prom-operator-k8s02"}', '_content_consumed': True, '_next': None, 'status_code': 200, 'headers': {'Cache-Control': 'no-store', 'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff', 'X-Frame-Options': 'deny', 'X-Xss-Protection': '1; mode=block', 'Date': 'Tue, 23 Apr 2024 02:50:18 GMT', 'Content-Length': '493'}, 'raw': , 'url': 'http://grafana:3000/api/datasources', 'encoding': 'utf-8', 'history': [], 'reason': 'OK', 'cookies': , 'elapsed': datetime.timedelta(microseconds=42852), 'request': , 'connection': } 2024-04-23T02:50:18.633273907Z 04/23/2024 02:50:18 AM - osm_mon.dashboarder.backends.grafana - INFO - Dashboard k8s02 is created in Grafana 2024-04-23T02:54:49.735178195Z 04/23/2024 02:54:49 AM - osm_mon.dashboarder.backends.grafana - INFO - Datasource prom-operator-k8s07 is created in Grafana 2024-04-23T02:54:49.735251172Z 04/23/2024 02:54:49 AM - osm_mon.dashboarder.backends.grafana - INFO - ************* response: {'_content': b'{"datasource":{"id":5,"uid":"ad79f0c5-a101-4f13-8b91-8f96977c235f","orgId":1,"name":"prom-operator-k8s07","type":"prometheus","typeLogoUrl":"public/app/plugins/datasource/prometheus/img/prometheus_logo.svg","access":"proxy","url":"http://172.21.248.125:9090","user":"","database":"","basicAuth":false,"basicAuthUser":"","withCredentials":false,"isDefault":false,"jsonData":{},"secureJsonFields":{},"version":1,"readOnly":false},"id":5,"message":"Datasource added","name":"prom-operator-k8s07"}', '_content_consumed': True, '_next': None, 'status_code': 200, 'headers': {'Cache-Control': 'no-store', 'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff', 'X-Frame-Options': 'deny', 'X-Xss-Protection': '1; mode=block', 'Date': 'Tue, 23 Apr 2024 02:54:49 GMT', 'Content-Length': '493'}, 'raw': , 'url': 'http://grafana:3000/api/datasources', 'encoding': 'utf-8', 'history': [], 'reason': 'OK', 'cookies': , 'elapsed': datetime.timedelta(microseconds=50830), 'request': , 'connection': } 2024-04-23T02:54:49.924624994Z 04/23/2024 02:54:49 AM - osm_mon.dashboarder.backends.grafana - INFO - Dashboard k8s07 is created in Grafana 2024-04-23T03:02:30.438625840Z 04/23/2024 03:02:30 AM - osm_mon.dashboarder.backends.grafana - INFO - New team quotas_01_proj_1 created in Grafana 2024-04-23T03:02:36.772898627Z 04/23/2024 03:02:36 AM - osm_mon.dashboarder.backends.grafana - INFO - New team quotas_01_proj_2 created in Grafana 2024-04-23T03:02:43.111193973Z 04/23/2024 03:02:43 AM - osm_mon.dashboarder.backends.grafana - INFO - New user quotas_01_user created in Grafana 2024-04-23T03:02:43.339614870Z 04/23/2024 03:02:43 AM - osm_mon.dashboarder.backends.grafana - INFO - User quotas_01_user is assigned Editor permission 2024-04-23T03:02:43.527321775Z 04/23/2024 03:02:43 AM - osm_mon.dashboarder.backends.grafana - INFO - User quotas_01_user is assigned Editor permission 2024-04-23T03:02:47.794548027Z 04/23/2024 03:02:47 AM - osm_mon.dashboarder.dashboarder - ERROR - Exception processing message: 2024-04-23T03:02:47.794595092Z Traceback (most recent call last): 2024-04-23T03:02:47.794600634Z File "/usr/lib/python3/dist-packages/osm_mon/dashboarder/dashboarder.py", line 116, in _user_msg 2024-04-23T03:02:47.794606494Z project_data = values["changes"]["add_project_role_mappings"] 2024-04-23T03:02:47.794611192Z KeyError: 'add_project_role_mappings' 2024-04-23T03:02:49.512533272Z 04/23/2024 03:02:49 AM - osm_mon.dashboarder.backends.grafana - INFO - Grafana team updated {"message":"Team updated"} 2024-04-23T03:02:51.858827245Z 04/23/2024 03:02:51 AM - osm_mon.dashboarder.backends.grafana - INFO - Dashboard folder quotas_01_proj_3 created 2024-04-23T03:02:52.068926622Z 04/23/2024 03:02:52 AM - osm_mon.dashboarder.backends.grafana - INFO - Dashboard quotas_01_proj_3 is created in Grafana 2024-04-23T03:02:52.127274236Z 04/23/2024 03:02:52 AM - osm_mon.dashboarder.backends.grafana - INFO - Dashboard folder quotas_01_proj_2 created 2024-04-23T03:02:52.316804500Z 04/23/2024 03:02:52 AM - osm_mon.dashboarder.backends.grafana - INFO - Dashboard quotas_01_proj_2 is created in Grafana 2024-04-23T03:02:59.780616423Z 04/23/2024 03:02:59 AM - osm_mon.dashboarder.dashboarder - ERROR - Exception processing message: 2024-04-23T03:02:59.780644709Z Traceback (most recent call last): 2024-04-23T03:02:59.780650321Z File "/usr/lib/python3/dist-packages/osm_mon/dashboarder/dashboarder.py", line 129, in _user_msg 2024-04-23T03:02:59.780655332Z project_new_name = values["changes"]["name"] 2024-04-23T03:02:59.780660255Z KeyError: 'name' 2024-04-23T03:03:30.859418406Z 04/23/2024 03:03:30 AM - osm_mon.dashboarder.backends.grafana - INFO - Team quotas_01_proj_2 deleted in Grafana 2024-04-23T03:03:32.616357907Z 04/23/2024 03:03:32 AM - osm_mon.dashboarder.backends.grafana - INFO - User quotas_01_user deleted in Grafana 2024-04-23T03:03:34.176063236Z 04/23/2024 03:03:34 AM - osm_mon.dashboarder.backends.grafana - INFO - Team quotas_01_proj_3 deleted in Grafana 2024-04-23T03:03:52.738075495Z 04/23/2024 03:03:52 AM - osm_mon.dashboarder.backends.grafana - INFO - Dashboard admin is created in Grafana 2024-04-23T03:12:24.796386883Z 04/23/2024 03:12:24 AM - osm_mon.dashboarder.backends.grafana - INFO - Dashboard admin is created in Grafana 2024-04-23T03:26:58.084211402Z 04/23/2024 03:26:58 AM - osm_mon.dashboarder.backends.grafana - INFO - Dashboard admin is created in Grafana 2024-04-23T03:52:33.969279630Z 04/23/2024 03:52:33 AM - osm_mon.dashboarder.backends.grafana - INFO - Dashboard admin is created in Grafana