2024-05-09T20:42:33.816028692Z nc: kafka (10.97.109.127:9092): Operation timed out 2024-05-09T20:42:37.827249603Z nc: kafka (10.97.109.127:9092): Operation timed out 2024-05-09T20:42:41.834565744Z nc: kafka (10.97.109.127:9092): Operation timed out 2024-05-09T20:42:45.840119913Z nc: kafka (10.97.109.127:9092): Operation timed out 2024-05-09T20:42:49.842396054Z nc: kafka (10.97.109.127:9092): Operation timed out 2024-05-09T20:42:53.845204452Z nc: kafka (10.97.109.127:9092): Operation timed out 2024-05-09T20:42:57.865112025Z nc: kafka (10.97.109.127:9092): Operation timed out 2024-05-09T20:43:01.869007334Z nc: kafka (10.97.109.127:9092): Operation timed out 2024-05-09T20:43:05.877542419Z nc: kafka (10.97.109.127:9092): Operation timed out 2024-05-09T20:43:09.884011214Z nc: kafka (10.97.109.127:9092): Operation timed out 2024-05-09T20:43:13.887437496Z nc: kafka (10.97.109.127:9092): Operation timed out 2024-05-09T20:43:17.890739019Z nc: kafka (10.97.109.127:9092): Operation timed out 2024-05-09T20:43:21.895237926Z nc: kafka (10.97.109.127:9092): Operation timed out 2024-05-09T20:43:25.898607281Z nc: kafka (10.97.109.127:9092): Operation timed out 2024-05-09T20:43:29.903169079Z nc: kafka (10.97.109.127:9092): Operation timed out 2024-05-09T20:43:33.906558172Z nc: kafka (10.97.109.127:9092): Operation timed out 2024-05-09T20:43:37.910897424Z nc: kafka (10.97.109.127:9092): Operation timed out 2024-05-09T20:43:41.914794089Z nc: kafka (10.97.109.127:9092): Operation timed out 2024-05-09T20:43:45.918753505Z nc: kafka (10.97.109.127:9092): Operation timed out 2024-05-09T20:43:49.926544255Z nc: kafka (10.97.109.127:9092): Operation timed out 2024-05-09T20:43:53.935879296Z nc: kafka (10.97.109.127:9092): Operation timed out 2024-05-09T20:43:57.944099333Z nc: kafka (10.97.109.127:9092): Operation timed out 2024-05-09T20:44:01.944219393Z nc: kafka (10.97.109.127:9092): Operation timed out 2024-05-09T20:44:05.951715587Z nc: kafka (10.97.109.127:9092): Operation timed out 2024-05-09T20:44:09.959713600Z nc: kafka (10.97.109.127:9092): Operation timed out 2024-05-09T20:44:28.036016893Z kafka (10.97.109.127:9092) open 2024-05-09T20:44:28.036042767Z mongodb-k8s (10.244.0.42:27017) open 2024-05-09T20:46:04.056184483Z 05/09/2024 08:46:04 PM - kafka.conn - INFO - [IPv4 ('10.97.109.127', 9092)]>: connecting to kafka:9092 [('10.97.109.127', 9092) IPv4] 2024-05-09T20:46:04.056537507Z 05/09/2024 08:46:04 PM - kafka.conn - INFO - Probing node bootstrap-0 broker version 2024-05-09T20:46:04.059364086Z 05/09/2024 08:46:04 PM - kafka.conn - INFO - [IPv4 ('10.97.109.127', 9092)]>: Connection complete. 2024-05-09T20:46:04.346271565Z 05/09/2024 08:46:04 PM - kafka.conn - INFO - Broker version identified as 2.5.0 2024-05-09T20:46:04.346288947Z 05/09/2024 08:46:04 PM - kafka.conn - INFO - Set configuration api_version=(2, 5, 0) to skip auto check_version requests on startup 2024-05-09T20:46:04.382310355Z 05/09/2024 08:46:04 PM - kafka.conn - INFO - [IPv4 ('10.97.109.127', 9092)]>: connecting to kafka:9092 [('10.97.109.127', 9092) IPv4] 2024-05-09T20:46:04.382466597Z 05/09/2024 08:46:04 PM - kafka.conn - INFO - Probing node bootstrap-0 broker version 2024-05-09T20:46:04.388788717Z 05/09/2024 08:46:04 PM - kafka.conn - INFO - [IPv4 ('10.97.109.127', 9092)]>: Connection complete. 2024-05-09T20:46:04.635865779Z 05/09/2024 08:46:04 PM - kafka.conn - INFO - Broker version identified as 2.5.0 2024-05-09T20:46:04.635881293Z 05/09/2024 08:46:04 PM - kafka.conn - INFO - Set configuration api_version=(2, 5, 0) to skip auto check_version requests on startup 2024-05-09T20:46:04.899834166Z 05/09/2024 08:46:04 PM - kafka.conn - INFO - [IPv4 ('10.244.0.32', 9092)]>: connecting to kafka-controller-1.kafka-controller-headless.osm.svc.cluster.local:9092 [('10.244.0.32', 9092) IPv4] 2024-05-09T20:46:04.899884683Z 05/09/2024 08:46:04 PM - kafka.conn - INFO - [IPv4 ('10.244.0.32', 9092)]>: Connection complete. 2024-05-09T20:46:04.899889676Z 05/09/2024 08:46:04 PM - kafka.conn - INFO - [IPv4 ('10.97.109.127', 9092)]>: Closing connection. 2024-05-09T20:46:05.843986596Z 05/09/2024 08:46:05 PM - kafka.cluster - ERROR - Topic users not found in cluster metadata 2024-05-09T20:46:05.961363015Z 05/09/2024 08:46:05 PM - kafka.conn - INFO - [IPv4 ('10.244.0.43', 9092)]>: connecting to kafka-controller-0.kafka-controller-headless.osm.svc.cluster.local:9092 [('10.244.0.43', 9092) IPv4] 2024-05-09T20:46:05.961396692Z 05/09/2024 08:46:05 PM - kafka.conn - INFO - [IPv4 ('10.244.0.43', 9092)]>: Connection complete. 2024-05-09T20:46:05.973925774Z 05/09/2024 08:46:05 PM - kafka.producer.record_accumulator - WARNING - Produced messages to topic-partition TopicPartition(topic='users', partition=0) with base offset -1 log start offset -1 and error None. 2024-05-09T20:46:06.076553068Z 05/09/2024 08:46:06 PM - kafka.cluster - ERROR - Topic project not found in cluster metadata 2024-05-09T20:46:06.215351517Z 05/09/2024 08:46:06 PM - kafka.cluster - ERROR - Topic project not found in cluster metadata 2024-05-09T20:46:06.346506454Z 05/09/2024 08:46:06 PM - kafka.cluster - ERROR - Topic project not found in cluster metadata 2024-05-09T20:46:06.467480797Z 05/09/2024 08:46:06 PM - kafka.cluster - ERROR - Topic project not found in cluster metadata 2024-05-09T20:46:06.605501346Z 05/09/2024 08:46:06 PM - kafka.conn - INFO - [IPv4 ('10.244.0.32', 9092)]>: Closing connection. 2024-05-09T20:46:06.605527187Z 05/09/2024 08:46:06 PM - kafka.conn - INFO - [IPv4 ('10.244.0.43', 9092)]>: Closing connection. 2024-05-09T20:46:06.605531226Z 05/09/2024 08:46:06 PM - osm_mon.cmd.mon_dashboarder - INFO - Starting MON Dashboarder... 2024-05-09T20:46:06.771710028Z 05/09/2024 08:46:06 PM - db - INFO - Connected to database osm version 1.2 2024-05-09T20:46:06.771816567Z 05/09/2024 08:46:06 PM - osm_mon.cmd.mon_dashboarder - INFO - Starting MON kafka Consumer... 2024-05-09T20:46:06.771821363Z 05/09/2024 08:46:06 PM - osm_mon.cmd.mon_dashboarder - INFO - Starting MON Dashboarder... 2024-05-09T20:46:06.976388793Z 05/09/2024 08:46:06 PM - aiokafka.consumer.subscription_state - INFO - Updating subscribed topics to: frozenset({'project', 'users'}) 2024-05-09T20:46:06.976715421Z 05/09/2024 08:46:06 PM - aiokafka.consumer.consumer - INFO - Subscribed to topic(s): {'project', 'users'} 2024-05-09T20:46:07.119646330Z 05/09/2024 08:46:07 PM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 1 for group mon-consumer 2024-05-09T20:46:07.119664647Z 05/09/2024 08:46:07 PM - aiokafka.consumer.group_coordinator - INFO - Revoking previously assigned partitions set() for group mon-consumer 2024-05-09T20:46:07.119670801Z 05/09/2024 08:46:07 PM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-05-09T20:46:08.124282057Z 05/09/2024 08:46:08 PM - osm_mon.dashboarder.backends.grafana - INFO - Dashboard admin is created in Grafana 2024-05-09T20:46:10.420527120Z 05/09/2024 08:46:10 PM - aiokafka.consumer.group_coordinator - INFO - Joined group 'mon-consumer' (generation 1) with member_id aiokafka-0.8.1-495e7b8e-4a1d-417d-8cdc-d478f4a3db97 2024-05-09T20:46:10.420542754Z 05/09/2024 08:46:10 PM - aiokafka.consumer.group_coordinator - INFO - Elected group leader -- performing partition assignments using roundrobin 2024-05-09T20:46:10.904465429Z 05/09/2024 08:46:10 PM - aiokafka.consumer.group_coordinator - INFO - Successfully synced group mon-consumer with generation 1 2024-05-09T20:46:10.906030883Z 05/09/2024 08:46:10 PM - aiokafka.consumer.group_coordinator - INFO - Setting newly assigned partitions {TopicPartition(topic='project', partition=0), TopicPartition(topic='users', partition=0)} for group mon-consumer 2024-05-09T20:52:39.609231367Z 05/09/2024 08:52:39 PM - osm_mon.dashboarder.backends.grafana - INFO - Datasource prom-operator-osm is created in Grafana 2024-05-09T20:52:39.609278649Z 05/09/2024 08:52:39 PM - osm_mon.dashboarder.backends.grafana - INFO - ************* response: {'_content': b'{"datasource":{"id":3,"uid":"db324398-8eec-4e84-b019-79c126cd6d99","orgId":1,"name":"prom-operator-osm","type":"prometheus","typeLogoUrl":"public/app/plugins/datasource/prometheus/img/prometheus_logo.svg","access":"proxy","url":"http://172.21.248.125:9090","user":"","database":"","basicAuth":false,"basicAuthUser":"","withCredentials":false,"isDefault":false,"jsonData":{},"secureJsonFields":{},"version":1,"readOnly":false},"id":3,"message":"Datasource added","name":"prom-operator-osm"}', '_content_consumed': True, '_next': None, 'status_code': 200, 'headers': {'Cache-Control': 'no-store', 'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff', 'X-Frame-Options': 'deny', 'X-Xss-Protection': '1; mode=block', 'Date': 'Thu, 09 May 2024 20:52:39 GMT', 'Content-Length': '489'}, 'raw': , 'url': 'http://grafana:3000/api/datasources', 'encoding': 'utf-8', 'history': [], 'reason': 'OK', 'cookies': , 'elapsed': datetime.timedelta(microseconds=96164), 'request': , 'connection': } 2024-05-09T20:52:39.909176007Z 05/09/2024 08:52:39 PM - osm_mon.dashboarder.backends.grafana - INFO - Dashboard osm is created in Grafana 2024-05-09T21:02:42.241463433Z 05/09/2024 09:02:42 PM - osm_mon.dashboarder.backends.grafana - INFO - Dashboard admin is created in Grafana