2024-04-25T17:36:02.723917651Z 04/25/2024 05:36:02 PM - kafka.conn - INFO - [IPv4 ('10.103.177.36', 9092)]>: connecting to kafka:9092 [('10.103.177.36', 9092) IPv4] 2024-04-25T17:36:02.723949332Z 04/25/2024 05:36:02 PM - kafka.conn - INFO - Probing node bootstrap-0 broker version 2024-04-25T17:36:02.723956109Z 04/25/2024 05:36:02 PM - kafka.conn - INFO - [IPv4 ('10.103.177.36', 9092)]>: Connection complete. 2024-04-25T17:36:02.840367134Z 04/25/2024 05:36:02 PM - kafka.conn - INFO - Broker version identified as 2.5.0 2024-04-25T17:36:02.840381953Z 04/25/2024 05:36:02 PM - kafka.conn - INFO - Set configuration api_version=(2, 5, 0) to skip auto check_version requests on startup 2024-04-25T17:36:02.852416257Z 04/25/2024 05:36:02 PM - kafka.conn - INFO - [IPv4 ('10.103.177.36', 9092)]>: connecting to kafka:9092 [('10.103.177.36', 9092) IPv4] 2024-04-25T17:36:02.852429586Z 04/25/2024 05:36:02 PM - kafka.conn - INFO - Probing node bootstrap-0 broker version 2024-04-25T17:36:02.853018634Z 04/25/2024 05:36:02 PM - kafka.conn - INFO - [IPv4 ('10.103.177.36', 9092)]>: Connection complete. 2024-04-25T17:36:02.958836635Z 04/25/2024 05:36:02 PM - kafka.conn - INFO - Broker version identified as 2.5.0 2024-04-25T17:36:02.958856890Z 04/25/2024 05:36:02 PM - kafka.conn - INFO - Set configuration api_version=(2, 5, 0) to skip auto check_version requests on startup 2024-04-25T17:36:03.120906861Z 04/25/2024 05:36:03 PM - kafka.conn - INFO - [IPv4 ('10.244.0.32', 9092)]>: connecting to kafka-controller-1.kafka-controller-headless.osm.svc.cluster.local:9092 [('10.244.0.32', 9092) IPv4] 2024-04-25T17:36:03.121600124Z 04/25/2024 05:36:03 PM - kafka.conn - INFO - [IPv4 ('10.244.0.32', 9092)]>: Connection complete. 2024-04-25T17:36:03.121609161Z 04/25/2024 05:36:03 PM - kafka.conn - INFO - [IPv4 ('10.103.177.36', 9092)]>: Closing connection. 2024-04-25T17:36:03.651066710Z 04/25/2024 05:36:03 PM - kafka.conn - INFO - [IPv4 ('10.244.0.34', 9092)]>: connecting to kafka-controller-2.kafka-controller-headless.osm.svc.cluster.local:9092 [('10.244.0.34', 9092) IPv4] 2024-04-25T17:36:03.651430358Z 04/25/2024 05:36:03 PM - kafka.conn - INFO - [IPv4 ('10.244.0.34', 9092)]>: Connection complete. 2024-04-25T17:36:03.776456084Z 04/25/2024 05:36:03 PM - kafka.cluster - ERROR - Topic users not found in cluster metadata 2024-04-25T17:36:03.924945869Z 04/25/2024 05:36:03 PM - kafka.conn - INFO - [IPv4 ('10.244.0.45', 9092)]>: connecting to kafka-controller-0.kafka-controller-headless.osm.svc.cluster.local:9092 [('10.244.0.45', 9092) IPv4] 2024-04-25T17:36:03.925238747Z 04/25/2024 05:36:03 PM - kafka.conn - INFO - [IPv4 ('10.244.0.45', 9092)]>: Connection complete. 2024-04-25T17:36:03.959566012Z 04/25/2024 05:36:03 PM - kafka.producer.record_accumulator - WARNING - Produced messages to topic-partition TopicPartition(topic='users', partition=0) with base offset -1 log start offset -1 and error None. 2024-04-25T17:36:04.044313191Z 04/25/2024 05:36:04 PM - kafka.cluster - ERROR - Topic project not found in cluster metadata 2024-04-25T17:36:04.171591969Z 04/25/2024 05:36:04 PM - kafka.producer.record_accumulator - WARNING - Produced messages to topic-partition TopicPartition(topic='project', partition=0) with base offset -1 log start offset -1 and error None. 2024-04-25T17:36:04.171610160Z 04/25/2024 05:36:04 PM - kafka.conn - INFO - [IPv4 ('10.244.0.32', 9092)]>: Closing connection. 2024-04-25T17:36:04.171613926Z 04/25/2024 05:36:04 PM - kafka.conn - INFO - [IPv4 ('10.244.0.34', 9092)]>: Closing connection. 2024-04-25T17:36:04.172331789Z 04/25/2024 05:36:04 PM - kafka.conn - INFO - [IPv4 ('10.244.0.45', 9092)]>: Closing connection. 2024-04-25T17:36:04.180351872Z 04/25/2024 05:36:04 PM - osm_mon.cmd.mon_dashboarder - INFO - Starting MON Dashboarder... 2024-04-25T17:36:04.231598128Z 04/25/2024 05:36:04 PM - db - INFO - Connected to database osm version None 2024-04-25T17:36:04.231612009Z 04/25/2024 05:36:04 PM - osm_mon.cmd.mon_dashboarder - INFO - Starting MON kafka Consumer... 2024-04-25T17:36:04.231615716Z 04/25/2024 05:36:04 PM - osm_mon.cmd.mon_dashboarder - INFO - Starting MON Dashboarder... 2024-04-25T17:36:04.290712177Z 04/25/2024 05:36:04 PM - aiokafka.consumer.subscription_state - INFO - Updating subscribed topics to: frozenset({'users', 'project'}) 2024-04-25T17:36:04.290733673Z 04/25/2024 05:36:04 PM - aiokafka.consumer.consumer - INFO - Subscribed to topic(s): {'users', 'project'} 2024-04-25T17:36:04.324918913Z 04/25/2024 05:36:04 PM - aiokafka.consumer.group_coordinator - ERROR - Group Coordinator Request failed: [Error 15] CoordinatorNotAvailableError 2024-04-25T17:36:04.351066101Z 04/25/2024 05:36:04 PM - aiokafka.cluster - ERROR - Topic project not found in cluster metadata 2024-04-25T17:36:04.502057384Z 04/25/2024 05:36:04 PM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 1 for group mon-consumer 2024-04-25T17:36:04.502068853Z 04/25/2024 05:36:04 PM - aiokafka.consumer.group_coordinator - INFO - Revoking previously assigned partitions set() for group mon-consumer 2024-04-25T17:36:04.502072602Z 04/25/2024 05:36:04 PM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-04-25T17:36:04.561304735Z 04/25/2024 05:36:04 PM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 1)for group mon-consumer. 2024-04-25T17:36:04.697839934Z 04/25/2024 05:36:04 PM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 1 for group mon-consumer 2024-04-25T17:36:04.697874069Z 04/25/2024 05:36:04 PM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-04-25T17:36:04.705515278Z 04/25/2024 05:36:04 PM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 1)for group mon-consumer. 2024-04-25T17:36:04.836363260Z 04/25/2024 05:36:04 PM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 1 for group mon-consumer 2024-04-25T17:36:04.836383446Z 04/25/2024 05:36:04 PM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-04-25T17:36:04.860902183Z 04/25/2024 05:36:04 PM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 1)for group mon-consumer. 2024-04-25T17:36:04.968171804Z 04/25/2024 05:36:04 PM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 1 for group mon-consumer 2024-04-25T17:36:04.973073345Z 04/25/2024 05:36:04 PM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-04-25T17:36:04.986819793Z 04/25/2024 05:36:04 PM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 1)for group mon-consumer. 2024-04-25T17:36:05.140286839Z 04/25/2024 05:36:05 PM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 1 for group mon-consumer 2024-04-25T17:36:05.140297824Z 04/25/2024 05:36:05 PM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-04-25T17:36:05.151083780Z 04/25/2024 05:36:05 PM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 1)for group mon-consumer. 2024-04-25T17:36:05.271153493Z 04/25/2024 05:36:05 PM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 1 for group mon-consumer 2024-04-25T17:36:05.271163664Z 04/25/2024 05:36:05 PM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-04-25T17:36:05.285807533Z 04/25/2024 05:36:05 PM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 1)for group mon-consumer. 2024-04-25T17:36:05.394196445Z 04/25/2024 05:36:05 PM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 1 for group mon-consumer 2024-04-25T17:36:05.394210871Z 04/25/2024 05:36:05 PM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-04-25T17:36:05.404042182Z 04/25/2024 05:36:05 PM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 1)for group mon-consumer. 2024-04-25T17:36:05.535987874Z 04/25/2024 05:36:05 PM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 1 for group mon-consumer 2024-04-25T17:36:05.535996855Z 04/25/2024 05:36:05 PM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-04-25T17:36:05.536341404Z 04/25/2024 05:36:05 PM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 1)for group mon-consumer. 2024-04-25T17:36:05.647146484Z 04/25/2024 05:36:05 PM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 1 for group mon-consumer 2024-04-25T17:36:05.647163070Z 04/25/2024 05:36:05 PM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-04-25T17:36:05.648293128Z 04/25/2024 05:36:05 PM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 1)for group mon-consumer. 2024-04-25T17:36:05.760100092Z 04/25/2024 05:36:05 PM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 1 for group mon-consumer 2024-04-25T17:36:05.760310606Z 04/25/2024 05:36:05 PM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-04-25T17:36:05.762373455Z 04/25/2024 05:36:05 PM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 1)for group mon-consumer. 2024-04-25T17:36:05.882789547Z 04/25/2024 05:36:05 PM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 1 for group mon-consumer 2024-04-25T17:36:05.882806102Z 04/25/2024 05:36:05 PM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-04-25T17:36:05.892755145Z 04/25/2024 05:36:05 PM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 1)for group mon-consumer. 2024-04-25T17:36:06.001310916Z 04/25/2024 05:36:06 PM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 1 for group mon-consumer 2024-04-25T17:36:06.001328638Z 04/25/2024 05:36:06 PM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-04-25T17:36:06.004503216Z 04/25/2024 05:36:06 PM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 1)for group mon-consumer. 2024-04-25T17:36:06.108453983Z 04/25/2024 05:36:06 PM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 1 for group mon-consumer 2024-04-25T17:36:06.108485439Z 04/25/2024 05:36:06 PM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-04-25T17:36:06.157295606Z 04/25/2024 05:36:06 PM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 1)for group mon-consumer. 2024-04-25T17:36:06.296123389Z 04/25/2024 05:36:06 PM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 1 for group mon-consumer 2024-04-25T17:36:06.296139158Z 04/25/2024 05:36:06 PM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-04-25T17:36:06.304330060Z 04/25/2024 05:36:06 PM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 1)for group mon-consumer. 2024-04-25T17:36:06.411949692Z 04/25/2024 05:36:06 PM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 1 for group mon-consumer 2024-04-25T17:36:06.411959862Z 04/25/2024 05:36:06 PM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-04-25T17:36:06.419696134Z 04/25/2024 05:36:06 PM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 1)for group mon-consumer. 2024-04-25T17:36:06.540171810Z 04/25/2024 05:36:06 PM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 1 for group mon-consumer 2024-04-25T17:36:06.540184411Z 04/25/2024 05:36:06 PM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-04-25T17:36:09.719272320Z 04/25/2024 05:36:09 PM - aiokafka.consumer.group_coordinator - INFO - Joined group 'mon-consumer' (generation 1) with member_id aiokafka-0.8.1-917f6d5d-75fc-4396-b7a5-1bf583428a38 2024-04-25T17:36:09.719291040Z 04/25/2024 05:36:09 PM - aiokafka.consumer.group_coordinator - INFO - Elected group leader -- performing partition assignments using roundrobin 2024-04-25T17:36:09.719295000Z 04/25/2024 05:36:09 PM - kafka.coordinator.assignors.roundrobin - WARNING - No partition metadata for topic project 2024-04-25T17:36:10.109852947Z 04/25/2024 05:36:10 PM - aiokafka.consumer.group_coordinator - INFO - Successfully synced group mon-consumer with generation 1 2024-04-25T17:36:10.109875129Z 04/25/2024 05:36:10 PM - aiokafka.consumer.group_coordinator - INFO - Setting newly assigned partitions {TopicPartition(topic='users', partition=0)} for group mon-consumer 2024-04-25T17:36:35.662997862Z 04/25/2024 05:36:35 PM - osm_mon.dashboarder.backends.grafana - INFO - Dashboard admin is created in Grafana 2024-04-25T17:41:04.542102555Z 04/25/2024 05:41:04 PM - aiokafka.consumer.group_coordinator - INFO - Metadata for topic has changed from {'users': 1, 'project': 0} to {'users': 1, 'project': 1}. 2024-04-25T17:41:04.551438576Z 04/25/2024 05:41:04 PM - aiokafka.consumer.group_coordinator - INFO - Revoking previously assigned partitions frozenset({TopicPartition(topic='users', partition=0)}) for group mon-consumer 2024-04-25T17:41:04.551581705Z 04/25/2024 05:41:04 PM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-04-25T17:41:04.565488016Z 04/25/2024 05:41:04 PM - aiokafka.consumer.group_coordinator - INFO - Joined group 'mon-consumer' (generation 2) with member_id aiokafka-0.8.1-917f6d5d-75fc-4396-b7a5-1bf583428a38 2024-04-25T17:41:04.565500113Z 04/25/2024 05:41:04 PM - aiokafka.consumer.group_coordinator - INFO - Elected group leader -- performing partition assignments using roundrobin 2024-04-25T17:41:04.587739575Z 04/25/2024 05:41:04 PM - aiokafka.consumer.group_coordinator - INFO - Successfully synced group mon-consumer with generation 2 2024-04-25T17:41:04.588287097Z 04/25/2024 05:41:04 PM - aiokafka.consumer.group_coordinator - INFO - Setting newly assigned partitions {TopicPartition(topic='project', partition=0), TopicPartition(topic='users', partition=0)} for group mon-consumer 2024-04-25T17:43:07.016067244Z 04/25/2024 05:43:07 PM - osm_mon.dashboarder.backends.grafana - INFO - Datasource prom-operator-osm is created in Grafana 2024-04-25T17:43:07.016835518Z 04/25/2024 05:43:07 PM - osm_mon.dashboarder.backends.grafana - INFO - ************* response: {'_content': b'{"datasource":{"id":3,"uid":"c30a23f3-bd94-4ada-9944-75626851f907","orgId":1,"name":"prom-operator-osm","type":"prometheus","typeLogoUrl":"public/app/plugins/datasource/prometheus/img/prometheus_logo.svg","access":"proxy","url":"http://172.21.248.125:9090","user":"","database":"","basicAuth":false,"basicAuthUser":"","withCredentials":false,"isDefault":false,"jsonData":{},"secureJsonFields":{},"version":1,"readOnly":false},"id":3,"message":"Datasource added","name":"prom-operator-osm"}', '_content_consumed': True, '_next': None, 'status_code': 200, 'headers': {'Cache-Control': 'no-store', 'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff', 'X-Frame-Options': 'deny', 'X-Xss-Protection': '1; mode=block', 'Date': 'Thu, 25 Apr 2024 17:43:07 GMT', 'Content-Length': '489'}, 'raw': , 'url': 'http://grafana:3000/api/datasources', 'encoding': 'utf-8', 'history': [], 'reason': 'OK', 'cookies': , 'elapsed': datetime.timedelta(microseconds=45274), 'request': , 'connection': } 2024-04-25T17:43:07.201155066Z 04/25/2024 05:43:07 PM - osm_mon.dashboarder.backends.grafana - INFO - Dashboard osm is created in Grafana 2024-04-25T17:51:39.219903938Z 04/25/2024 05:51:39 PM - osm_mon.dashboarder.backends.grafana - INFO - Dashboard admin is created in Grafana 2024-04-25T17:33:04.378374879Z nc: kafka (10.103.177.36:9092): Operation timed out 2024-04-25T17:33:08.404323897Z nc: kafka (10.103.177.36:9092): Operation timed out 2024-04-25T17:33:12.404033386Z nc: kafka (10.103.177.36:9092): Operation timed out 2024-04-25T17:33:16.416629842Z nc: kafka (10.103.177.36:9092): Operation timed out 2024-04-25T17:33:20.417410194Z nc: kafka (10.103.177.36:9092): Operation timed out 2024-04-25T17:33:24.422817712Z nc: kafka (10.103.177.36:9092): Operation timed out 2024-04-25T17:33:28.428288199Z nc: kafka (10.103.177.36:9092): Operation timed out 2024-04-25T17:33:32.432574131Z nc: kafka (10.103.177.36:9092): Operation timed out 2024-04-25T17:33:36.448454567Z nc: kafka (10.103.177.36:9092): Operation timed out 2024-04-25T17:33:40.440351051Z nc: kafka (10.103.177.36:9092): Operation timed out 2024-04-25T17:33:44.445708713Z nc: kafka (10.103.177.36:9092): Operation timed out 2024-04-25T17:33:48.450186926Z nc: kafka (10.103.177.36:9092): Operation timed out 2024-04-25T17:33:52.457445776Z nc: kafka (10.103.177.36:9092): Operation timed out 2024-04-25T17:33:56.461779782Z nc: kafka (10.103.177.36:9092): Operation timed out 2024-04-25T17:34:00.464572981Z nc: kafka (10.103.177.36:9092): Operation timed out 2024-04-25T17:34:04.481919705Z nc: kafka (10.103.177.36:9092): Operation timed out 2024-04-25T17:34:08.489493258Z nc: kafka (10.103.177.36:9092): Operation timed out 2024-04-25T17:34:12.497015673Z nc: kafka (10.103.177.36:9092): Operation timed out 2024-04-25T17:34:16.504621315Z nc: kafka (10.103.177.36:9092): Operation timed out 2024-04-25T17:34:20.514963351Z nc: kafka (10.103.177.36:9092): Operation timed out 2024-04-25T17:34:24.520867979Z nc: kafka (10.103.177.36:9092): Operation timed out 2024-04-25T17:34:28.524692041Z nc: kafka (10.103.177.36:9092): Operation timed out 2024-04-25T17:34:32.550025038Z nc: kafka (10.103.177.36:9092): Operation timed out 2024-04-25T17:34:36.547218375Z nc: kafka (10.103.177.36:9092): Operation timed out 2024-04-25T17:34:51.589057696Z kafka (10.103.177.36:9092) open 2024-04-25T17:34:51.590346171Z mongodb-k8s (10.244.0.30:27017) open