2024-05-03T12:35:02.746332403Z nc: kafka (10.96.248.254:9092): Operation timed out 2024-05-03T12:35:06.752252165Z nc: kafka (10.96.248.254:9092): Operation timed out 2024-05-03T12:35:10.757210935Z nc: kafka (10.96.248.254:9092): Operation timed out 2024-05-03T12:35:14.764194926Z nc: kafka (10.96.248.254:9092): Operation timed out 2024-05-03T12:35:18.766679079Z nc: kafka (10.96.248.254:9092): Operation timed out 2024-05-03T12:35:22.778587548Z nc: kafka (10.96.248.254:9092): Operation timed out 2024-05-03T12:35:26.819752479Z nc: kafka (10.96.248.254:9092): Operation timed out 2024-05-03T12:36:31.058234980Z nc: kafka (10.96.248.254:9092): Operation timed out 2024-05-03T12:36:35.075083262Z nc: kafka (10.96.248.254:9092): Operation timed out 2024-05-03T12:36:39.089610498Z nc: kafka (10.96.248.254:9092): Operation timed out 2024-05-03T12:36:43.102086990Z nc: kafka (10.96.248.254:9092): Operation timed out 2024-05-03T12:36:47.110374330Z nc: kafka (10.96.248.254:9092): Operation timed out 2024-05-03T12:36:51.122038554Z nc: kafka (10.96.248.254:9092): Operation timed out 2024-05-03T12:36:57.152351164Z kafka (10.96.248.254:9092) open 2024-05-03T12:36:57.160532299Z mongodb-k8s (10.244.0.24:27017) open 2024-05-03T12:37:46.974390583Z 05/03/2024 12:37:46 PM - kafka.conn - INFO - [IPv4 ('10.96.248.254', 9092)]>: connecting to kafka:9092 [('10.96.248.254', 9092) IPv4] 2024-05-03T12:37:46.974419894Z 05/03/2024 12:37:46 PM - kafka.conn - INFO - Probing node bootstrap-0 broker version 2024-05-03T12:37:46.974423810Z 05/03/2024 12:37:46 PM - kafka.conn - INFO - [IPv4 ('10.96.248.254', 9092)]>: Connection complete. 2024-05-03T12:37:47.071793393Z 05/03/2024 12:37:47 PM - kafka.conn - INFO - Broker version identified as 2.5.0 2024-05-03T12:37:47.073522319Z 05/03/2024 12:37:47 PM - kafka.conn - INFO - Set configuration api_version=(2, 5, 0) to skip auto check_version requests on startup 2024-05-03T12:37:47.079975934Z 05/03/2024 12:37:47 PM - kafka.conn - INFO - [IPv4 ('10.96.248.254', 9092)]>: connecting to kafka:9092 [('10.96.248.254', 9092) IPv4] 2024-05-03T12:37:47.079986307Z 05/03/2024 12:37:47 PM - kafka.conn - INFO - Probing node bootstrap-0 broker version 2024-05-03T12:37:47.079989924Z 05/03/2024 12:37:47 PM - kafka.conn - INFO - [IPv4 ('10.96.248.254', 9092)]>: Connection complete. 2024-05-03T12:37:47.190447610Z 05/03/2024 12:37:47 PM - kafka.conn - INFO - Broker version identified as 2.5.0 2024-05-03T12:37:47.190475163Z 05/03/2024 12:37:47 PM - kafka.conn - INFO - Set configuration api_version=(2, 5, 0) to skip auto check_version requests on startup 2024-05-03T12:37:47.350083167Z 05/03/2024 12:37:47 PM - kafka.conn - INFO - [IPv4 ('10.244.0.26', 9092)]>: connecting to kafka-controller-0.kafka-controller-headless.osm.svc.cluster.local:9092 [('10.244.0.26', 9092) IPv4] 2024-05-03T12:37:47.350104832Z 05/03/2024 12:37:47 PM - kafka.conn - INFO - [IPv4 ('10.244.0.26', 9092)]>: Connection complete. 2024-05-03T12:37:47.350111660Z 05/03/2024 12:37:47 PM - kafka.conn - INFO - [IPv4 ('10.96.248.254', 9092)]>: Closing connection. 2024-05-03T12:37:48.002406695Z 05/03/2024 12:37:48 PM - kafka.conn - INFO - [IPv4 ('10.244.0.40', 9092)]>: connecting to kafka-controller-2.kafka-controller-headless.osm.svc.cluster.local:9092 [('10.244.0.40', 9092) IPv4] 2024-05-03T12:37:48.002634129Z 05/03/2024 12:37:48 PM - kafka.conn - INFO - [IPv4 ('10.244.0.40', 9092)]>: Connection complete. 2024-05-03T12:37:48.110187058Z 05/03/2024 12:37:48 PM - kafka.cluster - ERROR - Topic users not found in cluster metadata 2024-05-03T12:37:48.209167329Z 05/03/2024 12:37:48 PM - kafka.cluster - ERROR - Topic users not found in cluster metadata 2024-05-03T12:37:48.314161659Z 05/03/2024 12:37:48 PM - kafka.cluster - ERROR - Topic users not found in cluster metadata 2024-05-03T12:37:48.530061898Z 05/03/2024 12:37:48 PM - kafka.cluster - ERROR - Topic project not found in cluster metadata 2024-05-03T12:37:48.635885528Z 05/03/2024 12:37:48 PM - kafka.cluster - ERROR - Topic project not found in cluster metadata 2024-05-03T12:37:48.746851987Z 05/03/2024 12:37:48 PM - kafka.cluster - ERROR - Topic project not found in cluster metadata 2024-05-03T12:37:48.858769216Z 05/03/2024 12:37:48 PM - kafka.cluster - ERROR - Topic project not found in cluster metadata 2024-05-03T12:37:48.965743563Z 05/03/2024 12:37:48 PM - kafka.conn - INFO - [IPv4 ('10.244.0.36', 9092)]>: connecting to kafka-controller-1.kafka-controller-headless.osm.svc.cluster.local:9092 [('10.244.0.36', 9092) IPv4] 2024-05-03T12:37:48.966005195Z 05/03/2024 12:37:48 PM - kafka.conn - INFO - [IPv4 ('10.244.0.36', 9092)]>: Connection complete. 2024-05-03T12:37:48.973954761Z 05/03/2024 12:37:48 PM - kafka.conn - INFO - [IPv4 ('10.244.0.26', 9092)]>: Closing connection. 2024-05-03T12:37:48.973964190Z 05/03/2024 12:37:48 PM - kafka.conn - INFO - [IPv4 ('10.244.0.40', 9092)]>: Closing connection. 2024-05-03T12:37:48.973967539Z 05/03/2024 12:37:48 PM - kafka.conn - INFO - [IPv4 ('10.244.0.36', 9092)]>: Closing connection. 2024-05-03T12:37:48.973971398Z 05/03/2024 12:37:48 PM - osm_mon.cmd.mon_dashboarder - INFO - Starting MON Dashboarder... 2024-05-03T12:37:49.014032832Z 05/03/2024 12:37:49 PM - db - INFO - Connected to database osm version 1.2 2024-05-03T12:37:49.014048502Z 05/03/2024 12:37:49 PM - osm_mon.cmd.mon_dashboarder - INFO - Starting MON kafka Consumer... 2024-05-03T12:37:49.020038741Z 05/03/2024 12:37:49 PM - osm_mon.cmd.mon_dashboarder - INFO - Starting MON Dashboarder... 2024-05-03T12:37:49.080324128Z 05/03/2024 12:37:49 PM - aiokafka.consumer.subscription_state - INFO - Updating subscribed topics to: frozenset({'users', 'project'}) 2024-05-03T12:37:49.082139406Z 05/03/2024 12:37:49 PM - aiokafka.consumer.consumer - INFO - Subscribed to topic(s): {'users', 'project'} 2024-05-03T12:37:49.100807250Z 05/03/2024 12:37:49 PM - aiokafka.consumer.group_coordinator - ERROR - Group Coordinator Request failed: [Error 15] CoordinatorNotAvailableError 2024-05-03T12:37:49.227988641Z 05/03/2024 12:37:49 PM - aiokafka.consumer.group_coordinator - ERROR - Group Coordinator Request failed: [Error 15] CoordinatorNotAvailableError 2024-05-03T12:37:49.343218036Z 05/03/2024 12:37:49 PM - aiokafka.consumer.group_coordinator - ERROR - Group Coordinator Request failed: [Error 15] CoordinatorNotAvailableError 2024-05-03T12:37:49.443130159Z 05/03/2024 12:37:49 PM - osm_mon.dashboarder.backends.grafana - INFO - Dashboard admin is created in Grafana 2024-05-03T12:37:49.520777673Z 05/03/2024 12:37:49 PM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 2 for group mon-consumer 2024-05-03T12:37:49.520791290Z 05/03/2024 12:37:49 PM - aiokafka.consumer.group_coordinator - INFO - Revoking previously assigned partitions set() for group mon-consumer 2024-05-03T12:37:49.520857043Z 05/03/2024 12:37:49 PM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-05-03T12:37:49.527564879Z 05/03/2024 12:37:49 PM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 2)for group mon-consumer. 2024-05-03T12:37:49.634819082Z 05/03/2024 12:37:49 PM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 2 for group mon-consumer 2024-05-03T12:37:49.634831195Z 05/03/2024 12:37:49 PM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-05-03T12:37:49.637940630Z 05/03/2024 12:37:49 PM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 2)for group mon-consumer. 2024-05-03T12:37:49.742836027Z 05/03/2024 12:37:49 PM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 2 for group mon-consumer 2024-05-03T12:37:49.742858116Z 05/03/2024 12:37:49 PM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-05-03T12:37:49.744663918Z 05/03/2024 12:37:49 PM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 2)for group mon-consumer. 2024-05-03T12:37:49.912716417Z 05/03/2024 12:37:49 PM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 2 for group mon-consumer 2024-05-03T12:37:49.912727374Z 05/03/2024 12:37:49 PM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-05-03T12:37:49.920861536Z 05/03/2024 12:37:49 PM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 2)for group mon-consumer. 2024-05-03T12:37:50.047243992Z 05/03/2024 12:37:50 PM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 2 for group mon-consumer 2024-05-03T12:37:50.047256946Z 05/03/2024 12:37:50 PM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-05-03T12:37:50.051098930Z 05/03/2024 12:37:50 PM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 2)for group mon-consumer. 2024-05-03T12:37:50.167939169Z 05/03/2024 12:37:50 PM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 2 for group mon-consumer 2024-05-03T12:37:50.167954955Z 05/03/2024 12:37:50 PM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-05-03T12:37:50.192373425Z 05/03/2024 12:37:50 PM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 2)for group mon-consumer. 2024-05-03T12:37:50.300646172Z 05/03/2024 12:37:50 PM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 2 for group mon-consumer 2024-05-03T12:37:50.300903159Z 05/03/2024 12:37:50 PM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-05-03T12:37:50.325811093Z 05/03/2024 12:37:50 PM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 2)for group mon-consumer. 2024-05-03T12:37:50.440022455Z 05/03/2024 12:37:50 PM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 2 for group mon-consumer 2024-05-03T12:37:50.440073060Z 05/03/2024 12:37:50 PM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-05-03T12:37:50.448044845Z 05/03/2024 12:37:50 PM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 2)for group mon-consumer. 2024-05-03T12:37:50.551705715Z 05/03/2024 12:37:50 PM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 2 for group mon-consumer 2024-05-03T12:37:50.551747707Z 05/03/2024 12:37:50 PM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-05-03T12:37:50.560881113Z 05/03/2024 12:37:50 PM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 2)for group mon-consumer. 2024-05-03T12:37:50.676082206Z 05/03/2024 12:37:50 PM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 2 for group mon-consumer 2024-05-03T12:37:50.676094911Z 05/03/2024 12:37:50 PM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-05-03T12:37:50.697510340Z 05/03/2024 12:37:50 PM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 2)for group mon-consumer. 2024-05-03T12:37:50.801328434Z 05/03/2024 12:37:50 PM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 2 for group mon-consumer 2024-05-03T12:37:50.804473178Z 05/03/2024 12:37:50 PM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-05-03T12:37:50.811995262Z 05/03/2024 12:37:50 PM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 2)for group mon-consumer. 2024-05-03T12:37:50.926173842Z 05/03/2024 12:37:50 PM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 2 for group mon-consumer 2024-05-03T12:37:50.926196234Z 05/03/2024 12:37:50 PM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-05-03T12:37:50.937086714Z 05/03/2024 12:37:50 PM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 2)for group mon-consumer. 2024-05-03T12:37:51.059878527Z 05/03/2024 12:37:51 PM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 2 for group mon-consumer 2024-05-03T12:37:51.060130140Z 05/03/2024 12:37:51 PM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-05-03T12:37:51.072423331Z 05/03/2024 12:37:51 PM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 2)for group mon-consumer. 2024-05-03T12:37:51.182043037Z 05/03/2024 12:37:51 PM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 2 for group mon-consumer 2024-05-03T12:37:51.182071579Z 05/03/2024 12:37:51 PM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-05-03T12:37:51.182076191Z 05/03/2024 12:37:51 PM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 2)for group mon-consumer. 2024-05-03T12:37:51.300848841Z 05/03/2024 12:37:51 PM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 2 for group mon-consumer 2024-05-03T12:37:51.300862681Z 05/03/2024 12:37:51 PM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-05-03T12:37:51.302843967Z 05/03/2024 12:37:51 PM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 2)for group mon-consumer. 2024-05-03T12:37:51.423979491Z 05/03/2024 12:37:51 PM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 2 for group mon-consumer 2024-05-03T12:37:51.424105883Z 05/03/2024 12:37:51 PM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-05-03T12:37:51.432632023Z 05/03/2024 12:37:51 PM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 2)for group mon-consumer. 2024-05-03T12:37:51.545461936Z 05/03/2024 12:37:51 PM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 2 for group mon-consumer 2024-05-03T12:37:51.545588145Z 05/03/2024 12:37:51 PM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-05-03T12:37:51.550443318Z 05/03/2024 12:37:51 PM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 2)for group mon-consumer. 2024-05-03T12:37:51.665191092Z 05/03/2024 12:37:51 PM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 2 for group mon-consumer 2024-05-03T12:37:51.665250831Z 05/03/2024 12:37:51 PM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-05-03T12:37:54.722027886Z 05/03/2024 12:37:54 PM - aiokafka.consumer.group_coordinator - INFO - Joined group 'mon-consumer' (generation 1) with member_id aiokafka-0.8.1-847fd01f-5278-4408-8188-d71425ee41a6 2024-05-03T12:37:54.722069252Z 05/03/2024 12:37:54 PM - aiokafka.consumer.group_coordinator - INFO - Elected group leader -- performing partition assignments using roundrobin 2024-05-03T12:37:54.818454793Z 05/03/2024 12:37:54 PM - aiokafka.consumer.group_coordinator - INFO - Successfully synced group mon-consumer with generation 1 2024-05-03T12:37:54.818660020Z 05/03/2024 12:37:54 PM - aiokafka.consumer.group_coordinator - INFO - Setting newly assigned partitions {TopicPartition(topic='users', partition=0), TopicPartition(topic='project', partition=0)} for group mon-consumer 2024-05-03T12:44:20.930678067Z 05/03/2024 12:44:20 PM - osm_mon.dashboarder.backends.grafana - INFO - Datasource prom-operator-osm is created in Grafana 2024-05-03T12:44:20.930785083Z 05/03/2024 12:44:20 PM - osm_mon.dashboarder.backends.grafana - INFO - ************* response: {'_content': b'{"datasource":{"id":3,"uid":"e37f316c-4fa5-4df9-88ab-6e207e1285d0","orgId":1,"name":"prom-operator-osm","type":"prometheus","typeLogoUrl":"public/app/plugins/datasource/prometheus/img/prometheus_logo.svg","access":"proxy","url":"http://172.21.248.125:9090","user":"","database":"","basicAuth":false,"basicAuthUser":"","withCredentials":false,"isDefault":false,"jsonData":{},"secureJsonFields":{},"version":1,"readOnly":false},"id":3,"message":"Datasource added","name":"prom-operator-osm"}', '_content_consumed': True, '_next': None, 'status_code': 200, 'headers': {'Cache-Control': 'no-store', 'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff', 'X-Frame-Options': 'deny', 'X-Xss-Protection': '1; mode=block', 'Date': 'Fri, 03 May 2024 12:44:20 GMT', 'Content-Length': '489'}, 'raw': , 'url': 'http://grafana:3000/api/datasources', 'encoding': 'utf-8', 'history': [], 'reason': 'OK', 'cookies': , 'elapsed': datetime.timedelta(microseconds=47869), 'request': , 'connection': } 2024-05-03T12:44:21.102535172Z 05/03/2024 12:44:21 PM - osm_mon.dashboarder.backends.grafana - INFO - Dashboard osm is created in Grafana 2024-05-03T12:51:52.964996760Z 05/03/2024 12:51:52 PM - osm_mon.dashboarder.backends.grafana - INFO - Dashboard admin is created in Grafana