2024-04-22T14:33:35.491145677Z nc: kafka (10.99.196.83:9092): Operation timed out 2024-04-22T14:33:39.496157203Z nc: kafka (10.99.196.83:9092): Operation timed out 2024-04-22T14:33:43.503379558Z nc: kafka (10.99.196.83:9092): Operation timed out 2024-04-22T14:33:47.511216253Z nc: kafka (10.99.196.83:9092): Operation timed out 2024-04-22T14:33:51.523235745Z nc: kafka (10.99.196.83:9092): Operation timed out 2024-04-22T14:33:55.529221861Z nc: kafka (10.99.196.83:9092): Operation timed out 2024-04-22T14:33:59.532207944Z nc: kafka (10.99.196.83:9092): Operation timed out 2024-04-22T14:34:03.541008183Z nc: kafka (10.99.196.83:9092): Operation timed out 2024-04-22T14:34:07.545839218Z nc: kafka (10.99.196.83:9092): Operation timed out 2024-04-22T14:34:11.554697892Z nc: kafka (10.99.196.83:9092): Operation timed out 2024-04-22T14:34:15.556003536Z nc: kafka (10.99.196.83:9092): Operation timed out 2024-04-22T14:34:19.562055278Z nc: kafka (10.99.196.83:9092): Operation timed out 2024-04-22T14:34:23.567478194Z nc: kafka (10.99.196.83:9092): Operation timed out 2024-04-22T14:34:27.570032898Z nc: kafka (10.99.196.83:9092): Operation timed out 2024-04-22T14:34:31.578351272Z nc: kafka (10.99.196.83:9092): Operation timed out 2024-04-22T14:34:35.582861666Z nc: kafka (10.99.196.83:9092): Operation timed out 2024-04-22T14:34:39.587964705Z nc: kafka (10.99.196.83:9092): Operation timed out 2024-04-22T14:34:43.600992748Z nc: kafka (10.99.196.83:9092): Operation timed out 2024-04-22T14:34:47.615528494Z nc: kafka (10.99.196.83:9092): Operation timed out 2024-04-22T14:34:51.617943138Z nc: kafka (10.99.196.83:9092): Operation timed out 2024-04-22T14:34:55.620504526Z nc: kafka (10.99.196.83:9092): Operation timed out 2024-04-22T14:34:59.630172505Z nc: kafka (10.99.196.83:9092): Operation timed out 2024-04-22T14:35:03.637826292Z nc: kafka (10.99.196.83:9092): Operation timed out 2024-04-22T14:35:07.650715117Z nc: kafka (10.99.196.83:9092): Operation timed out 2024-04-22T14:35:11.667946988Z nc: kafka (10.99.196.83:9092): Operation timed out 2024-04-22T14:35:17.695361907Z kafka (10.99.196.83:9092) open 2024-04-22T14:35:17.703053659Z mongodb-k8s (10.244.0.46:27017) open 2024-04-22T14:36:52.854651509Z 04/22/2024 02:36:52 PM - kafka.conn - INFO - [IPv4 ('10.99.196.83', 9092)]>: connecting to kafka:9092 [('10.99.196.83', 9092) IPv4] 2024-04-22T14:36:52.855086951Z 04/22/2024 02:36:52 PM - kafka.conn - INFO - Probing node bootstrap-0 broker version 2024-04-22T14:36:52.876106013Z 04/22/2024 02:36:52 PM - kafka.conn - INFO - [IPv4 ('10.99.196.83', 9092)]>: Connection complete. 2024-04-22T14:36:52.996730357Z 04/22/2024 02:36:52 PM - kafka.conn - INFO - Broker version identified as 2.5.0 2024-04-22T14:36:52.996896229Z 04/22/2024 02:36:52 PM - kafka.conn - INFO - Set configuration api_version=(2, 5, 0) to skip auto check_version requests on startup 2024-04-22T14:36:53.020654777Z 04/22/2024 02:36:53 PM - kafka.conn - INFO - [IPv4 ('10.99.196.83', 9092)]>: connecting to kafka:9092 [('10.99.196.83', 9092) IPv4] 2024-04-22T14:36:53.020790490Z 04/22/2024 02:36:53 PM - kafka.conn - INFO - Probing node bootstrap-0 broker version 2024-04-22T14:36:53.022859168Z 04/22/2024 02:36:53 PM - kafka.conn - INFO - [IPv4 ('10.99.196.83', 9092)]>: Connection complete. 2024-04-22T14:36:53.133159151Z 04/22/2024 02:36:53 PM - kafka.conn - INFO - Broker version identified as 2.5.0 2024-04-22T14:36:53.133176296Z 04/22/2024 02:36:53 PM - kafka.conn - INFO - Set configuration api_version=(2, 5, 0) to skip auto check_version requests on startup 2024-04-22T14:36:53.299838378Z 04/22/2024 02:36:53 PM - kafka.conn - INFO - [IPv4 ('10.244.0.47', 9092)]>: connecting to kafka-controller-0.kafka-controller-headless.osm.svc.cluster.local:9092 [('10.244.0.47', 9092) IPv4] 2024-04-22T14:36:53.299870792Z 04/22/2024 02:36:53 PM - kafka.conn - INFO - [IPv4 ('10.244.0.47', 9092)]>: Connection complete. 2024-04-22T14:36:53.299875045Z 04/22/2024 02:36:53 PM - kafka.conn - INFO - [IPv4 ('10.99.196.83', 9092)]>: Closing connection. 2024-04-22T14:36:54.033682962Z 04/22/2024 02:36:54 PM - kafka.conn - INFO - [IPv4 ('10.244.0.35', 9092)]>: connecting to kafka-controller-1.kafka-controller-headless.osm.svc.cluster.local:9092 [('10.244.0.35', 9092) IPv4] 2024-04-22T14:36:54.033694841Z 04/22/2024 02:36:54 PM - kafka.conn - INFO - [IPv4 ('10.244.0.35', 9092)]>: Connection complete. 2024-04-22T14:36:54.130739399Z 04/22/2024 02:36:54 PM - kafka.cluster - ERROR - Topic users not found in cluster metadata 2024-04-22T14:36:54.272214879Z 04/22/2024 02:36:54 PM - kafka.cluster - ERROR - Topic users not found in cluster metadata 2024-04-22T14:36:54.618066689Z 04/22/2024 02:36:54 PM - kafka.cluster - ERROR - Topic users not found in cluster metadata 2024-04-22T14:36:54.918686554Z 04/22/2024 02:36:54 PM - kafka.cluster - ERROR - Topic project not found in cluster metadata 2024-04-22T14:36:55.125692469Z 04/22/2024 02:36:55 PM - kafka.cluster - ERROR - Topic project not found in cluster metadata 2024-04-22T14:36:55.349729528Z 04/22/2024 02:36:55 PM - kafka.cluster - ERROR - Topic project not found in cluster metadata 2024-04-22T14:36:55.465918903Z 04/22/2024 02:36:55 PM - kafka.cluster - ERROR - Topic project not found in cluster metadata 2024-04-22T14:36:55.640844209Z 04/22/2024 02:36:55 PM - kafka.cluster - ERROR - Topic project not found in cluster metadata 2024-04-22T14:36:55.864552014Z 04/22/2024 02:36:55 PM - kafka.cluster - ERROR - Topic project not found in cluster metadata 2024-04-22T14:36:55.982185600Z 04/22/2024 02:36:55 PM - kafka.cluster - ERROR - Topic project not found in cluster metadata 2024-04-22T14:36:56.095758514Z 04/22/2024 02:36:56 PM - kafka.cluster - ERROR - Topic project not found in cluster metadata 2024-04-22T14:36:56.305396365Z 04/22/2024 02:36:56 PM - kafka.cluster - ERROR - Topic project not found in cluster metadata 2024-04-22T14:36:56.438990870Z 04/22/2024 02:36:56 PM - kafka.cluster - ERROR - Topic project not found in cluster metadata 2024-04-22T14:36:56.634245998Z 04/22/2024 02:36:56 PM - kafka.cluster - ERROR - Topic project not found in cluster metadata 2024-04-22T14:36:56.780673762Z 04/22/2024 02:36:56 PM - kafka.cluster - ERROR - Topic project not found in cluster metadata 2024-04-22T14:36:57.135724734Z 04/22/2024 02:36:57 PM - kafka.cluster - ERROR - Topic project not found in cluster metadata 2024-04-22T14:36:57.333824735Z 04/22/2024 02:36:57 PM - kafka.cluster - ERROR - Topic project not found in cluster metadata 2024-04-22T14:36:57.483908116Z 04/22/2024 02:36:57 PM - kafka.cluster - ERROR - Topic project not found in cluster metadata 2024-04-22T14:36:57.687659971Z 04/22/2024 02:36:57 PM - kafka.cluster - ERROR - Topic project not found in cluster metadata 2024-04-22T14:36:57.930977332Z 04/22/2024 02:36:57 PM - kafka.cluster - ERROR - Topic project not found in cluster metadata 2024-04-22T14:36:58.167276266Z 04/22/2024 02:36:58 PM - kafka.cluster - ERROR - Topic project not found in cluster metadata 2024-04-22T14:36:58.460306178Z 04/22/2024 02:36:58 PM - kafka.cluster - ERROR - Topic project not found in cluster metadata 2024-04-22T14:36:58.565498885Z 04/22/2024 02:36:58 PM - kafka.cluster - ERROR - Topic project not found in cluster metadata 2024-04-22T14:36:58.694984674Z 04/22/2024 02:36:58 PM - kafka.cluster - ERROR - Topic project not found in cluster metadata 2024-04-22T14:36:58.800921907Z 04/22/2024 02:36:58 PM - kafka.cluster - ERROR - Topic project not found in cluster metadata 2024-04-22T14:36:58.922728552Z 04/22/2024 02:36:58 PM - kafka.cluster - ERROR - Topic project not found in cluster metadata 2024-04-22T14:36:59.417813711Z 04/22/2024 02:36:59 PM - kafka.cluster - ERROR - Topic project not found in cluster metadata 2024-04-22T14:36:59.627947014Z 04/22/2024 02:36:59 PM - kafka.cluster - ERROR - Topic project not found in cluster metadata 2024-04-22T14:36:59.935408747Z 04/22/2024 02:36:59 PM - kafka.cluster - ERROR - Topic project not found in cluster metadata 2024-04-22T14:37:00.048679720Z 04/22/2024 02:37:00 PM - kafka.cluster - ERROR - Topic project not found in cluster metadata 2024-04-22T14:37:00.215424965Z 04/22/2024 02:37:00 PM - kafka.cluster - ERROR - Topic project not found in cluster metadata 2024-04-22T14:37:00.374240428Z 04/22/2024 02:37:00 PM - kafka.cluster - ERROR - Topic project not found in cluster metadata 2024-04-22T14:37:00.668572842Z 04/22/2024 02:37:00 PM - kafka.cluster - ERROR - Topic project not found in cluster metadata 2024-04-22T14:37:00.929155066Z 04/22/2024 02:37:00 PM - kafka.cluster - ERROR - Topic project not found in cluster metadata 2024-04-22T14:37:01.084625908Z 04/22/2024 02:37:01 PM - kafka.cluster - ERROR - Topic project not found in cluster metadata 2024-04-22T14:37:01.293819193Z 04/22/2024 02:37:01 PM - kafka.cluster - ERROR - Topic project not found in cluster metadata 2024-04-22T14:37:01.477916106Z 04/22/2024 02:37:01 PM - kafka.cluster - ERROR - Topic project not found in cluster metadata 2024-04-22T14:37:01.601492987Z 04/22/2024 02:37:01 PM - kafka.cluster - ERROR - Topic project not found in cluster metadata 2024-04-22T14:37:01.749423636Z 04/22/2024 02:37:01 PM - kafka.cluster - ERROR - Topic project not found in cluster metadata 2024-04-22T14:37:02.009028529Z 04/22/2024 02:37:02 PM - kafka.cluster - ERROR - Topic project not found in cluster metadata 2024-04-22T14:37:02.167530211Z 04/22/2024 02:37:02 PM - kafka.cluster - ERROR - Topic project not found in cluster metadata 2024-04-22T14:37:02.330579267Z 04/22/2024 02:37:02 PM - kafka.cluster - ERROR - Topic project not found in cluster metadata 2024-04-22T14:37:02.458162827Z 04/22/2024 02:37:02 PM - kafka.cluster - ERROR - Topic project not found in cluster metadata 2024-04-22T14:37:02.579452733Z 04/22/2024 02:37:02 PM - kafka.cluster - ERROR - Topic project not found in cluster metadata 2024-04-22T14:37:02.695003599Z 04/22/2024 02:37:02 PM - kafka.cluster - ERROR - Topic project not found in cluster metadata 2024-04-22T14:37:02.821574412Z 04/22/2024 02:37:02 PM - kafka.producer.record_accumulator - WARNING - Produced messages to topic-partition TopicPartition(topic='project', partition=0) with base offset -1 log start offset -1 and error None. 2024-04-22T14:37:02.821584210Z 04/22/2024 02:37:02 PM - kafka.conn - INFO - [IPv4 ('10.244.0.47', 9092)]>: Closing connection. 2024-04-22T14:37:02.821587821Z 04/22/2024 02:37:02 PM - kafka.conn - INFO - [IPv4 ('10.244.0.35', 9092)]>: Closing connection. 2024-04-22T14:37:02.821592119Z 04/22/2024 02:37:02 PM - osm_mon.cmd.mon_dashboarder - INFO - Starting MON Dashboarder... 2024-04-22T14:37:02.866412653Z 04/22/2024 02:37:02 PM - db - INFO - Connected to database osm version 1.2 2024-04-22T14:37:02.866913585Z 04/22/2024 02:37:02 PM - osm_mon.cmd.mon_dashboarder - INFO - Starting MON kafka Consumer... 2024-04-22T14:37:02.867685575Z 04/22/2024 02:37:02 PM - osm_mon.cmd.mon_dashboarder - INFO - Starting MON Dashboarder... 2024-04-22T14:37:03.005004695Z 04/22/2024 02:37:02 PM - aiokafka.consumer.subscription_state - INFO - Updating subscribed topics to: frozenset({'project', 'users'}) 2024-04-22T14:37:03.005024873Z 04/22/2024 02:37:02 PM - aiokafka.consumer.consumer - INFO - Subscribed to topic(s): {'project', 'users'} 2024-04-22T14:37:03.182010361Z 04/22/2024 02:37:03 PM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 0 for group mon-consumer 2024-04-22T14:37:03.183032090Z 04/22/2024 02:37:03 PM - aiokafka.consumer.group_coordinator - INFO - Revoking previously assigned partitions set() for group mon-consumer 2024-04-22T14:37:03.183044030Z 04/22/2024 02:37:03 PM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-04-22T14:37:03.183047758Z 04/22/2024 02:37:03 PM - aiokafka.cluster - ERROR - Topic project not found in cluster metadata 2024-04-22T14:37:03.227135986Z 04/22/2024 02:37:03 PM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 0)for group mon-consumer. 2024-04-22T14:37:03.372998281Z 04/22/2024 02:37:03 PM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 0 for group mon-consumer 2024-04-22T14:37:03.373018275Z 04/22/2024 02:37:03 PM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-04-22T14:37:03.393058822Z 04/22/2024 02:37:03 PM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 0)for group mon-consumer. 2024-04-22T14:37:03.601874091Z 04/22/2024 02:37:03 PM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 0 for group mon-consumer 2024-04-22T14:37:03.602154369Z 04/22/2024 02:37:03 PM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-04-22T14:37:03.715060319Z 04/22/2024 02:37:03 PM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 0)for group mon-consumer. 2024-04-22T14:37:03.820570000Z 04/22/2024 02:37:03 PM - osm_mon.dashboarder.backends.grafana - INFO - Dashboard admin is created in Grafana 2024-04-22T14:37:03.880102308Z 04/22/2024 02:37:03 PM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 0 for group mon-consumer 2024-04-22T14:37:03.880125310Z 04/22/2024 02:37:03 PM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-04-22T14:37:03.882133922Z 04/22/2024 02:37:03 PM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 0)for group mon-consumer. 2024-04-22T14:37:03.987548538Z 04/22/2024 02:37:03 PM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 0 for group mon-consumer 2024-04-22T14:37:03.988516200Z 04/22/2024 02:37:03 PM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-04-22T14:37:04.091938528Z 04/22/2024 02:37:04 PM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 0)for group mon-consumer. 2024-04-22T14:37:04.202130499Z 04/22/2024 02:37:04 PM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 0 for group mon-consumer 2024-04-22T14:37:04.202170080Z 04/22/2024 02:37:04 PM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-04-22T14:37:04.353780493Z 04/22/2024 02:37:04 PM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 0)for group mon-consumer. 2024-04-22T14:37:04.545465200Z 04/22/2024 02:37:04 PM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 0 for group mon-consumer 2024-04-22T14:37:04.545501612Z 04/22/2024 02:37:04 PM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-04-22T14:37:04.666918082Z 04/22/2024 02:37:04 PM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 0)for group mon-consumer. 2024-04-22T14:37:04.771716138Z 04/22/2024 02:37:04 PM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 0 for group mon-consumer 2024-04-22T14:37:04.771750057Z 04/22/2024 02:37:04 PM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-04-22T14:37:04.827280519Z 04/22/2024 02:37:04 PM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 0)for group mon-consumer. 2024-04-22T14:37:04.946169961Z 04/22/2024 02:37:04 PM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 0 for group mon-consumer 2024-04-22T14:37:04.946192528Z 04/22/2024 02:37:04 PM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-04-22T14:37:04.956920011Z 04/22/2024 02:37:04 PM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 0)for group mon-consumer. 2024-04-22T14:37:05.477728723Z 04/22/2024 02:37:05 PM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 0 for group mon-consumer 2024-04-22T14:37:05.478232708Z 04/22/2024 02:37:05 PM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-04-22T14:37:05.593320248Z 04/22/2024 02:37:05 PM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 0)for group mon-consumer. 2024-04-22T14:37:05.721757523Z 04/22/2024 02:37:05 PM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 0 for group mon-consumer 2024-04-22T14:37:05.721767824Z 04/22/2024 02:37:05 PM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-04-22T14:37:09.499310687Z 04/22/2024 02:37:09 PM - aiokafka.consumer.group_coordinator - INFO - Joined group 'mon-consumer' (generation 1) with member_id aiokafka-0.8.1-15f09abd-66ac-4b08-b621-f9491c18f694 2024-04-22T14:37:09.499340923Z 04/22/2024 02:37:09 PM - aiokafka.consumer.group_coordinator - INFO - Elected group leader -- performing partition assignments using roundrobin 2024-04-22T14:37:09.499345349Z 04/22/2024 02:37:09 PM - kafka.coordinator.assignors.roundrobin - WARNING - No partition metadata for topic project 2024-04-22T14:37:10.605791659Z 04/22/2024 02:37:10 PM - aiokafka.consumer.group_coordinator - INFO - Successfully synced group mon-consumer with generation 1 2024-04-22T14:37:10.607248381Z 04/22/2024 02:37:10 PM - aiokafka.consumer.group_coordinator - INFO - Setting newly assigned partitions {TopicPartition(topic='users', partition=0)} for group mon-consumer 2024-04-22T14:42:03.502171389Z 04/22/2024 02:42:03 PM - aiokafka.consumer.group_coordinator - INFO - Metadata for topic has changed from {'project': 0, 'users': 1} to {'project': 1, 'users': 1}. 2024-04-22T14:42:03.515046230Z 04/22/2024 02:42:03 PM - aiokafka.consumer.group_coordinator - INFO - Revoking previously assigned partitions frozenset({TopicPartition(topic='users', partition=0)}) for group mon-consumer 2024-04-22T14:42:03.515060568Z 04/22/2024 02:42:03 PM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-04-22T14:42:03.539978760Z 04/22/2024 02:42:03 PM - aiokafka.consumer.group_coordinator - INFO - Joined group 'mon-consumer' (generation 2) with member_id aiokafka-0.8.1-15f09abd-66ac-4b08-b621-f9491c18f694 2024-04-22T14:42:03.540032810Z 04/22/2024 02:42:03 PM - aiokafka.consumer.group_coordinator - INFO - Elected group leader -- performing partition assignments using roundrobin 2024-04-22T14:42:03.551522731Z 04/22/2024 02:42:03 PM - aiokafka.consumer.group_coordinator - INFO - Successfully synced group mon-consumer with generation 2 2024-04-22T14:42:03.552140218Z 04/22/2024 02:42:03 PM - aiokafka.consumer.group_coordinator - INFO - Setting newly assigned partitions {TopicPartition(topic='project', partition=0), TopicPartition(topic='users', partition=0)} for group mon-consumer