2024-04-07T01:27:35.701122582Z nc: kafka (10.106.248.92:9092): Operation timed out 2024-04-07T01:27:39.704628264Z nc: kafka (10.106.248.92:9092): Operation timed out 2024-04-07T01:27:43.708311684Z nc: kafka (10.106.248.92:9092): Operation timed out 2024-04-07T01:27:47.710719740Z nc: kafka (10.106.248.92:9092): Operation timed out 2024-04-07T01:27:51.716580424Z nc: kafka (10.106.248.92:9092): Operation timed out 2024-04-07T01:27:55.720780615Z nc: kafka (10.106.248.92:9092): Operation timed out 2024-04-07T01:27:59.729582941Z nc: kafka (10.106.248.92:9092): Operation timed out 2024-04-07T01:28:03.736096910Z nc: kafka (10.106.248.92:9092): Operation timed out 2024-04-07T01:28:07.738825655Z nc: kafka (10.106.248.92:9092): Operation timed out 2024-04-07T01:28:11.754196078Z nc: kafka (10.106.248.92:9092): Operation timed out 2024-04-07T01:28:15.762081636Z nc: kafka (10.106.248.92:9092): Operation timed out 2024-04-07T01:28:19.762413699Z nc: kafka (10.106.248.92:9092): Operation timed out 2024-04-07T01:28:23.766727584Z nc: kafka (10.106.248.92:9092): Operation timed out 2024-04-07T01:28:32.784843518Z kafka (10.106.248.92:9092) open 2024-04-07T01:28:32.786160058Z mongodb-k8s (10.244.0.21:27017) open 2024-04-07T01:30:33.361103836Z 04/07/2024 01:30:33 AM - kafka.conn - INFO - [IPv4 ('10.106.248.92', 9092)]>: connecting to kafka:9092 [('10.106.248.92', 9092) IPv4] 2024-04-07T01:30:33.362992452Z 04/07/2024 01:30:33 AM - kafka.conn - INFO - Probing node bootstrap-0 broker version 2024-04-07T01:30:33.364003018Z 04/07/2024 01:30:33 AM - kafka.conn - INFO - [IPv4 ('10.106.248.92', 9092)]>: Connection complete. 2024-04-07T01:30:33.586344642Z 04/07/2024 01:30:33 AM - kafka.conn - INFO - Broker version identified as 2.5.0 2024-04-07T01:30:33.586591595Z 04/07/2024 01:30:33 AM - kafka.conn - INFO - Set configuration api_version=(2, 5, 0) to skip auto check_version requests on startup 2024-04-07T01:30:33.621025572Z 04/07/2024 01:30:33 AM - kafka.conn - INFO - [IPv4 ('10.106.248.92', 9092)]>: connecting to kafka:9092 [('10.106.248.92', 9092) IPv4] 2024-04-07T01:30:33.621041234Z 04/07/2024 01:30:33 AM - kafka.conn - INFO - Probing node bootstrap-0 broker version 2024-04-07T01:30:33.621269170Z 04/07/2024 01:30:33 AM - kafka.conn - INFO - [IPv4 ('10.106.248.92', 9092)]>: Connection complete. 2024-04-07T01:30:33.810622946Z 04/07/2024 01:30:33 AM - kafka.conn - INFO - Broker version identified as 2.5.0 2024-04-07T01:30:33.810635259Z 04/07/2024 01:30:33 AM - kafka.conn - INFO - Set configuration api_version=(2, 5, 0) to skip auto check_version requests on startup 2024-04-07T01:30:34.013930973Z 04/07/2024 01:30:34 AM - kafka.conn - INFO - [IPv4 ('10.244.0.43', 9092)]>: connecting to kafka-controller-1.kafka-controller-headless.osm.svc.cluster.local:9092 [('10.244.0.43', 9092) IPv4] 2024-04-07T01:30:34.013958771Z 04/07/2024 01:30:34 AM - kafka.conn - INFO - [IPv4 ('10.244.0.43', 9092)]>: Connection complete. 2024-04-07T01:30:34.013965123Z 04/07/2024 01:30:34 AM - kafka.conn - INFO - [IPv4 ('10.106.248.92', 9092)]>: Closing connection. 2024-04-07T01:30:39.317221817Z 04/07/2024 01:30:39 AM - kafka.conn - INFO - [IPv4 ('10.244.0.29', 9092)]>: connecting to kafka-controller-2.kafka-controller-headless.osm.svc.cluster.local:9092 [('10.244.0.29', 9092) IPv4] 2024-04-07T01:30:39.317691380Z 04/07/2024 01:30:39 AM - kafka.conn - INFO - [IPv4 ('10.244.0.29', 9092)]>: Connection complete. 2024-04-07T01:30:39.449967333Z 04/07/2024 01:30:39 AM - kafka.cluster - ERROR - Topic users not found in cluster metadata 2024-04-07T01:30:39.557444057Z 04/07/2024 01:30:39 AM - kafka.cluster - ERROR - Topic users not found in cluster metadata 2024-04-07T01:30:39.739897368Z 04/07/2024 01:30:39 AM - kafka.cluster - ERROR - Topic users not found in cluster metadata 2024-04-07T01:30:39.965699179Z 04/07/2024 01:30:39 AM - kafka.cluster - ERROR - Topic project not found in cluster metadata 2024-04-07T01:30:40.078092687Z 04/07/2024 01:30:40 AM - kafka.cluster - ERROR - Topic project not found in cluster metadata 2024-04-07T01:30:40.195905001Z 04/07/2024 01:30:40 AM - kafka.conn - INFO - [IPv4 ('10.244.0.43', 9092)]>: Closing connection. 2024-04-07T01:30:40.199473982Z 04/07/2024 01:30:40 AM - kafka.conn - INFO - [IPv4 ('10.244.0.29', 9092)]>: Closing connection. 2024-04-07T01:30:40.199632148Z 04/07/2024 01:30:40 AM - osm_mon.cmd.mon_dashboarder - INFO - Starting MON Dashboarder... 2024-04-07T01:30:40.299344765Z 04/07/2024 01:30:40 AM - db - INFO - Connected to database osm version 1.2 2024-04-07T01:30:40.299370181Z 04/07/2024 01:30:40 AM - osm_mon.cmd.mon_dashboarder - INFO - Starting MON kafka Consumer... 2024-04-07T01:30:40.307380280Z 04/07/2024 01:30:40 AM - osm_mon.cmd.mon_dashboarder - INFO - Starting MON Dashboarder... 2024-04-07T01:30:40.393941962Z 04/07/2024 01:30:40 AM - aiokafka.consumer.subscription_state - INFO - Updating subscribed topics to: frozenset({'project', 'users'}) 2024-04-07T01:30:40.393958662Z 04/07/2024 01:30:40 AM - aiokafka.consumer.consumer - INFO - Subscribed to topic(s): {'project', 'users'} 2024-04-07T01:30:40.496475070Z 04/07/2024 01:30:40 AM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 0 for group mon-consumer 2024-04-07T01:30:40.496485208Z 04/07/2024 01:30:40 AM - aiokafka.consumer.group_coordinator - INFO - Revoking previously assigned partitions set() for group mon-consumer 2024-04-07T01:30:40.496489168Z 04/07/2024 01:30:40 AM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-04-07T01:30:40.991405111Z 04/07/2024 01:30:40 AM - osm_mon.dashboarder.backends.grafana - INFO - Dashboard admin is created in Grafana 2024-04-07T01:30:43.500711188Z 04/07/2024 01:30:43 AM - aiokafka.consumer.group_coordinator - INFO - Joined group 'mon-consumer' (generation 1) with member_id aiokafka-0.8.1-381dfe4c-d629-42c8-884a-9fc3e817a66d 2024-04-07T01:30:43.500725118Z 04/07/2024 01:30:43 AM - aiokafka.consumer.group_coordinator - INFO - Elected group leader -- performing partition assignments using roundrobin 2024-04-07T01:30:43.706076486Z 04/07/2024 01:30:43 AM - aiokafka.consumer.group_coordinator - INFO - Successfully synced group mon-consumer with generation 1 2024-04-07T01:30:43.706109468Z 04/07/2024 01:30:43 AM - aiokafka.consumer.group_coordinator - INFO - Setting newly assigned partitions {TopicPartition(topic='project', partition=0), TopicPartition(topic='users', partition=0)} for group mon-consumer 2024-04-07T01:38:13.084335808Z 04/07/2024 01:38:13 AM - osm_mon.dashboarder.backends.grafana - INFO - Datasource prom-operator-osm is created in Grafana 2024-04-07T01:38:13.084595890Z 04/07/2024 01:38:13 AM - osm_mon.dashboarder.backends.grafana - INFO - ************* response: {'_content': b'{"datasource":{"id":3,"uid":"c820e64f-4b91-4933-b0de-b049af029c2c","orgId":1,"name":"prom-operator-osm","type":"prometheus","typeLogoUrl":"public/app/plugins/datasource/prometheus/img/prometheus_logo.svg","access":"proxy","url":"http://172.21.248.125:9090","user":"","database":"","basicAuth":false,"basicAuthUser":"","withCredentials":false,"isDefault":false,"jsonData":{},"secureJsonFields":{},"version":1,"readOnly":false},"id":3,"message":"Datasource added","name":"prom-operator-osm"}', '_content_consumed': True, '_next': None, 'status_code': 200, 'headers': {'Cache-Control': 'no-store', 'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff', 'X-Frame-Options': 'deny', 'X-Xss-Protection': '1; mode=block', 'Date': 'Sun, 07 Apr 2024 01:38:13 GMT', 'Content-Length': '489'}, 'raw': , 'url': 'http://grafana:3000/api/datasources', 'encoding': 'utf-8', 'history': [], 'reason': 'OK', 'cookies': , 'elapsed': datetime.timedelta(microseconds=82245), 'request': , 'connection': } 2024-04-07T01:38:13.250750216Z 04/07/2024 01:38:13 AM - osm_mon.dashboarder.backends.grafana - INFO - Dashboard osm is created in Grafana 2024-04-07T01:44:44.804528328Z 04/07/2024 01:44:44 AM - osm_mon.dashboarder.backends.grafana - INFO - Dashboard admin is created in Grafana 2024-04-07T01:46:29.779961353Z 04/07/2024 01:46:29 AM - osm_mon.dashboarder.backends.grafana - INFO - New user basic_15_test_user created in Grafana 2024-04-07T01:46:33.060773478Z 04/07/2024 01:46:33 AM - osm_mon.dashboarder.backends.grafana - INFO - User basic_15_test_user is assigned Admin permission 2024-04-07T01:46:37.791412534Z 04/07/2024 01:46:37 AM - osm_mon.dashboarder.dashboarder - ERROR - Exception processing message: 2024-04-07T01:46:37.791434341Z Traceback (most recent call last): 2024-04-07T01:46:37.791437526Z File "/usr/lib/python3/dist-packages/osm_mon/dashboarder/dashboarder.py", line 116, in _user_msg 2024-04-07T01:46:37.791441554Z project_data = values["changes"]["add_project_role_mappings"] 2024-04-07T01:46:37.791444835Z KeyError: 'add_project_role_mappings' 2024-04-07T01:46:39.008227189Z 04/07/2024 01:46:39 AM - osm_mon.dashboarder.backends.grafana - INFO - New team basic_15_test_project created in Grafana 2024-04-07T01:46:40.624160087Z 04/07/2024 01:46:40 AM - osm_mon.dashboarder.backends.grafana - INFO - Grafana team updated {"message":"Team updated"} 2024-04-07T01:46:45.233750306Z 04/07/2024 01:46:45 AM - osm_mon.dashboarder.backends.grafana - INFO - Dashboard folder basic_15_project_test created 2024-04-07T01:46:45.427181699Z 04/07/2024 01:46:45 AM - osm_mon.dashboarder.backends.grafana - INFO - Dashboard basic_15_project_test is created in Grafana 2024-04-07T01:46:48.643323655Z 04/07/2024 01:46:48 AM - osm_mon.dashboarder.backends.grafana - INFO - User basic_15_test_user deleted in Grafana 2024-04-07T01:46:50.359123988Z 04/07/2024 01:46:50 AM - osm_mon.dashboarder.backends.grafana - INFO - Team basic_15_project_test deleted in Grafana 2024-04-07T01:47:15.726193665Z 04/07/2024 01:47:15 AM - osm_mon.dashboarder.backends.grafana - INFO - Dashboard admin is created in Grafana 2024-04-07T02:57:30.491089012Z 04/07/2024 02:57:30 AM - osm_mon.dashboarder.backends.grafana - INFO - Datasource prom-operator-k8s02 is created in Grafana 2024-04-07T02:57:30.491214411Z 04/07/2024 02:57:30 AM - osm_mon.dashboarder.backends.grafana - INFO - ************* response: {'_content': b'{"datasource":{"id":4,"uid":"eb087f94-0bdf-43a4-92e8-8b98397060d2","orgId":1,"name":"prom-operator-k8s02","type":"prometheus","typeLogoUrl":"public/app/plugins/datasource/prometheus/img/prometheus_logo.svg","access":"proxy","url":"http://172.21.248.125:9090","user":"","database":"","basicAuth":false,"basicAuthUser":"","withCredentials":false,"isDefault":false,"jsonData":{},"secureJsonFields":{},"version":1,"readOnly":false},"id":4,"message":"Datasource added","name":"prom-operator-k8s02"}', '_content_consumed': True, '_next': None, 'status_code': 200, 'headers': {'Cache-Control': 'no-store', 'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff', 'X-Frame-Options': 'deny', 'X-Xss-Protection': '1; mode=block', 'Date': 'Sun, 07 Apr 2024 02:57:30 GMT', 'Content-Length': '493'}, 'raw': , 'url': 'http://grafana:3000/api/datasources', 'encoding': 'utf-8', 'history': [], 'reason': 'OK', 'cookies': , 'elapsed': datetime.timedelta(microseconds=44015), 'request': , 'connection': } 2024-04-07T02:57:30.695214891Z 04/07/2024 02:57:30 AM - osm_mon.dashboarder.backends.grafana - INFO - Dashboard k8s02 is created in Grafana 2024-04-07T03:02:01.824065943Z 04/07/2024 03:02:01 AM - osm_mon.dashboarder.backends.grafana - INFO - Datasource prom-operator-k8s07 is created in Grafana 2024-04-07T03:02:01.824163983Z 04/07/2024 03:02:01 AM - osm_mon.dashboarder.backends.grafana - INFO - ************* response: {'_content': b'{"datasource":{"id":5,"uid":"aef25c2e-174e-4a17-8ae6-93c988e4e0ab","orgId":1,"name":"prom-operator-k8s07","type":"prometheus","typeLogoUrl":"public/app/plugins/datasource/prometheus/img/prometheus_logo.svg","access":"proxy","url":"http://172.21.248.125:9090","user":"","database":"","basicAuth":false,"basicAuthUser":"","withCredentials":false,"isDefault":false,"jsonData":{},"secureJsonFields":{},"version":1,"readOnly":false},"id":5,"message":"Datasource added","name":"prom-operator-k8s07"}', '_content_consumed': True, '_next': None, 'status_code': 200, 'headers': {'Cache-Control': 'no-store', 'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff', 'X-Frame-Options': 'deny', 'X-Xss-Protection': '1; mode=block', 'Date': 'Sun, 07 Apr 2024 03:02:01 GMT', 'Content-Length': '493'}, 'raw': , 'url': 'http://grafana:3000/api/datasources', 'encoding': 'utf-8', 'history': [], 'reason': 'OK', 'cookies': , 'elapsed': datetime.timedelta(microseconds=59752), 'request': , 'connection': } 2024-04-07T03:02:02.011467062Z 04/07/2024 03:02:02 AM - osm_mon.dashboarder.backends.grafana - INFO - Dashboard k8s07 is created in Grafana 2024-04-07T03:09:58.846267645Z 04/07/2024 03:09:58 AM - osm_mon.dashboarder.backends.grafana - INFO - New team quotas_01_proj_1 created in Grafana 2024-04-07T03:10:03.961728178Z 04/07/2024 03:10:03 AM - osm_mon.dashboarder.backends.grafana - INFO - Dashboard folder quotas_01_proj_1 created 2024-04-07T03:10:04.281551961Z 04/07/2024 03:10:04 AM - osm_mon.dashboarder.backends.grafana - INFO - Dashboard quotas_01_proj_1 is created in Grafana 2024-04-07T03:10:05.327958706Z 04/07/2024 03:10:05 AM - osm_mon.dashboarder.backends.grafana - INFO - New team quotas_01_proj_2 created in Grafana 2024-04-07T03:10:11.833603100Z 04/07/2024 03:10:11 AM - osm_mon.dashboarder.backends.grafana - INFO - New user quotas_01_user created in Grafana 2024-04-07T03:10:12.013994113Z 04/07/2024 03:10:12 AM - osm_mon.dashboarder.backends.grafana - INFO - User quotas_01_user is assigned Editor permission 2024-04-07T03:10:12.174253894Z 04/07/2024 03:10:12 AM - osm_mon.dashboarder.backends.grafana - INFO - User quotas_01_user is assigned Editor permission 2024-04-07T03:10:16.577325579Z 04/07/2024 03:10:16 AM - osm_mon.dashboarder.dashboarder - ERROR - Exception processing message: 2024-04-07T03:10:16.577345910Z Traceback (most recent call last): 2024-04-07T03:10:16.577349937Z File "/usr/lib/python3/dist-packages/osm_mon/dashboarder/dashboarder.py", line 116, in _user_msg 2024-04-07T03:10:16.577360022Z project_data = values["changes"]["add_project_role_mappings"] 2024-04-07T03:10:16.577363375Z KeyError: 'add_project_role_mappings' 2024-04-07T03:10:18.353073817Z 04/07/2024 03:10:18 AM - osm_mon.dashboarder.backends.grafana - INFO - Grafana team updated {"message":"Team updated"} 2024-04-07T03:10:28.409112949Z 04/07/2024 03:10:28 AM - osm_mon.dashboarder.dashboarder - ERROR - Exception processing message: 2024-04-07T03:10:28.409135173Z Traceback (most recent call last): 2024-04-07T03:10:28.409138862Z File "/usr/lib/python3/dist-packages/osm_mon/dashboarder/dashboarder.py", line 129, in _user_msg 2024-04-07T03:10:28.409142068Z project_new_name = values["changes"]["name"] 2024-04-07T03:10:28.409145203Z KeyError: 'name' 2024-04-07T03:10:34.437811625Z 04/07/2024 03:10:34 AM - osm_mon.dashboarder.backends.grafana - INFO - Dashboard folder quotas_01_proj_2 created 2024-04-07T03:10:34.626095157Z 04/07/2024 03:10:34 AM - osm_mon.dashboarder.backends.grafana - INFO - Dashboard quotas_01_proj_2 is created in Grafana 2024-04-07T03:10:58.629843275Z 04/07/2024 03:10:58 AM - osm_mon.dashboarder.backends.grafana - INFO - Team quotas_01_proj_2 deleted in Grafana 2024-04-07T03:11:00.440181103Z 04/07/2024 03:11:00 AM - osm_mon.dashboarder.backends.grafana - INFO - User quotas_01_user deleted in Grafana 2024-04-07T03:11:02.103192557Z 04/07/2024 03:11:02 AM - osm_mon.dashboarder.backends.grafana - INFO - Team quotas_01_proj_3 deleted in Grafana 2024-04-07T03:11:35.051934661Z 04/07/2024 03:11:35 AM - osm_mon.dashboarder.backends.grafana - INFO - Dashboard admin is created in Grafana 2024-04-07T03:16:36.328499008Z 04/07/2024 03:16:36 AM - osm_mon.dashboarder.backends.grafana - INFO - Dashboard admin is created in Grafana 2024-04-07T03:37:41.092304432Z 04/07/2024 03:37:41 AM - osm_mon.dashboarder.backends.grafana - INFO - Dashboard admin is created in Grafana 2024-04-07T04:02:46.712033099Z 04/07/2024 04:02:46 AM - osm_mon.dashboarder.backends.grafana - INFO - Dashboard admin is created in Grafana