nc: kafka (10.98.143.236:9092): Operation timed out
nc: kafka (10.98.143.236:9092): Operation timed out
nc: kafka (10.98.143.236:9092): Operation timed out
nc: kafka (10.98.143.236:9092): Operation timed out
nc: kafka (10.98.143.236:9092): Operation timed out
nc: kafka (10.98.143.236:9092): Operation timed out
nc: kafka (10.98.143.236:9092): Operation timed out
nc: kafka (10.98.143.236:9092): Operation timed out
nc: kafka (10.98.143.236:9092): Operation timed out
nc: kafka (10.98.143.236:9092): Operation timed out
nc: kafka (10.98.143.236:9092): Operation timed out
nc: kafka (10.98.143.236:9092): Operation timed out
nc: kafka (10.98.143.236:9092): Operation timed out
nc: kafka (10.98.143.236:9092): Operation timed out
nc: kafka (10.98.143.236:9092): Operation timed out
nc: kafka (10.98.143.236:9092): Operation timed out
nc: kafka (10.98.143.236:9092): Operation timed out
nc: kafka (10.98.143.236:9092): Operation timed out
nc: kafka (10.98.143.236:9092): Operation timed out
nc: kafka (10.98.143.236:9092): Operation timed out
nc: kafka (10.98.143.236:9092): Operation timed out
nc: kafka (10.98.143.236:9092): Operation timed out
nc: kafka (10.98.143.236:9092): Operation timed out
nc: kafka (10.98.143.236:9092): Operation timed out
nc: kafka (10.98.143.236:9092): Operation timed out
kafka (10.98.143.236:9092) open
mongodb-k8s (10.244.0.31:27017) open
02/11/2024 12:26:59 PM - kafka.conn - INFO - <BrokerConnection node_id=bootstrap-0 host=kafka:9092 <connecting> [IPv4 ('10.98.143.236', 9092)]>: connecting to kafka:9092 [('10.98.143.236', 9092) IPv4]
02/11/2024 12:26:59 PM - kafka.conn - INFO - Probing node bootstrap-0 broker version
02/11/2024 12:26:59 PM - kafka.conn - INFO - <BrokerConnection node_id=bootstrap-0 host=kafka:9092 <connecting> [IPv4 ('10.98.143.236', 9092)]>: Connection complete.
02/11/2024 12:27:00 PM - kafka.conn - INFO - Broker version identified as 2.5.0
02/11/2024 12:27:00 PM - kafka.conn - INFO - Set configuration api_version=(2, 5, 0) to skip auto check_version requests on startup
02/11/2024 12:27:00 PM - kafka.conn - INFO - <BrokerConnection node_id=bootstrap-0 host=kafka:9092 <connecting> [IPv4 ('10.98.143.236', 9092)]>: connecting to kafka:9092 [('10.98.143.236', 9092) IPv4]
02/11/2024 12:27:00 PM - kafka.conn - INFO - Probing node bootstrap-0 broker version
02/11/2024 12:27:00 PM - kafka.conn - INFO - <BrokerConnection node_id=bootstrap-0 host=kafka:9092 <connecting> [IPv4 ('10.98.143.236', 9092)]>: Connection complete.
02/11/2024 12:27:00 PM - kafka.conn - INFO - Broker version identified as 2.5.0
02/11/2024 12:27:00 PM - kafka.conn - INFO - Set configuration api_version=(2, 5, 0) to skip auto check_version requests on startup
02/11/2024 12:27:00 PM - kafka.conn - INFO - <BrokerConnection node_id=2 host=kafka-controller-2.kafka-controller-headless.osm.svc.cluster.local:9092 <connecting> [IPv4 ('10.244.0.35', 9092)]>: connecting to kafka-controller-2.kafka-controller-headless.osm.svc.cluster.local:9092 [('10.244.0.35', 9092) IPv4]
02/11/2024 12:27:00 PM - kafka.conn - INFO - <BrokerConnection node_id=2 host=kafka-controller-2.kafka-controller-headless.osm.svc.cluster.local:9092 <connecting> [IPv4 ('10.244.0.35', 9092)]>: Connection complete.
02/11/2024 12:27:00 PM - kafka.conn - INFO - <BrokerConnection node_id=bootstrap-0 host=kafka:9092 <connected> [IPv4 ('10.98.143.236', 9092)]>: Closing connection. 
02/11/2024 12:27:00 PM - kafka.cluster - ERROR - Topic users not found in cluster metadata
02/11/2024 12:27:01 PM - kafka.cluster - ERROR - Topic users not found in cluster metadata
02/11/2024 12:27:01 PM - kafka.cluster - ERROR - Topic users not found in cluster metadata
02/11/2024 12:27:01 PM - kafka.conn - INFO - <BrokerConnection node_id=0 host=kafka-controller-0.kafka-controller-headless.osm.svc.cluster.local:9092 <connecting> [IPv4 ('10.244.0.45', 9092)]>: connecting to kafka-controller-0.kafka-controller-headless.osm.svc.cluster.local:9092 [('10.244.0.45', 9092) IPv4]
02/11/2024 12:27:01 PM - kafka.conn - INFO - <BrokerConnection node_id=0 host=kafka-controller-0.kafka-controller-headless.osm.svc.cluster.local:9092 <connecting> [IPv4 ('10.244.0.45', 9092)]>: Connection complete.
02/11/2024 12:27:01 PM - kafka.cluster - ERROR - Topic project not found in cluster metadata
02/11/2024 12:27:01 PM - kafka.cluster - ERROR - Topic project not found in cluster metadata
02/11/2024 12:27:01 PM - kafka.producer.record_accumulator - WARNING - Produced messages to topic-partition TopicPartition(topic='project', partition=0) with base offset -1 log start offset -1 and error None.
02/11/2024 12:27:01 PM - kafka.conn - INFO - <BrokerConnection node_id=2 host=kafka-controller-2.kafka-controller-headless.osm.svc.cluster.local:9092 <connected> [IPv4 ('10.244.0.35', 9092)]>: Closing connection. 
02/11/2024 12:27:01 PM - kafka.conn - INFO - <BrokerConnection node_id=0 host=kafka-controller-0.kafka-controller-headless.osm.svc.cluster.local:9092 <connected> [IPv4 ('10.244.0.45', 9092)]>: Closing connection. 
02/11/2024 12:27:01 PM - osm_mon.cmd.mon_dashboarder - INFO - Starting MON Dashboarder...
02/11/2024 12:27:01 PM - db - INFO - Connected to database osm version 1.2
02/11/2024 12:27:01 PM - osm_mon.cmd.mon_dashboarder - INFO - Starting MON kafka Consumer...
02/11/2024 12:27:01 PM - osm_mon.cmd.mon_dashboarder - INFO - Starting MON Dashboarder...
02/11/2024 12:27:01 PM - aiokafka.consumer.subscription_state - INFO - Updating subscribed topics to: frozenset({'users', 'project'})
02/11/2024 12:27:01 PM - aiokafka.consumer.consumer - INFO - Subscribed to topic(s): {'users', 'project'}
02/11/2024 12:27:01 PM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 0 for group mon-consumer
02/11/2024 12:27:01 PM - aiokafka.consumer.group_coordinator - INFO - Revoking previously assigned partitions set() for group mon-consumer
02/11/2024 12:27:01 PM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer
02/11/2024 12:27:01 PM - aiokafka.cluster - ERROR - Topic project not found in cluster metadata
02/11/2024 12:27:02 PM - osm_mon.dashboarder.backends.grafana - INFO - Dashboard admin is created in Grafana
02/11/2024 12:27:04 PM - aiokafka.consumer.group_coordinator - INFO - Joined group 'mon-consumer' (generation 1) with member_id aiokafka-0.8.1-53085e07-ab76-4bf5-975c-99cbcc4f09a3
02/11/2024 12:27:04 PM - aiokafka.consumer.group_coordinator - INFO - Elected group leader -- performing partition assignments using roundrobin
02/11/2024 12:27:04 PM - kafka.coordinator.assignors.roundrobin - WARNING - No partition metadata for topic project
02/11/2024 12:27:04 PM - aiokafka.consumer.group_coordinator - INFO - Successfully synced group mon-consumer with generation 1
02/11/2024 12:27:04 PM - aiokafka.consumer.group_coordinator - INFO - Setting newly assigned partitions {TopicPartition(topic='users', partition=0)} for group mon-consumer
02/11/2024 12:30:02 PM - osm_mon.dashboarder.backends.grafana - INFO - Datasource prom-operator-k8stest202402111208 is created in Grafana
02/11/2024 12:30:02 PM - osm_mon.dashboarder.backends.grafana - INFO - ************* response: {'_content': b'{"datasource":{"id":3,"uid":"a1beae4c-8eda-4ff8-8c7a-f4716ae55246","orgId":1,"name":"prom-operator-k8stest202402111208","type":"prometheus","typeLogoUrl":"public/app/plugins/datasource/prometheus/img/prometheus_logo.svg","access":"proxy","url":"http://172.21.23.12:9090","user":"","database":"","basicAuth":false,"basicAuthUser":"","withCredentials":false,"isDefault":false,"jsonData":{},"secureJsonFields":{},"version":1,"readOnly":false},"id":3,"message":"Datasource added","name":"prom-operator-k8stest202402111208"}', '_content_consumed': True, '_next': None, 'status_code': 200, 'headers': {'Cache-Control': 'no-store', 'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff', 'X-Frame-Options': 'deny', 'X-Xss-Protection': '1; mode=block', 'Date': 'Sun, 11 Feb 2024 12:30:02 GMT', 'Content-Length': '519'}, 'raw': <urllib3.response.HTTPResponse object at 0x7fae202fc3a0>, 'url': 'http://grafana:3000/api/datasources', 'encoding': 'utf-8', 'history': [], 'reason': 'OK', 'cookies': <RequestsCookieJar[]>, 'elapsed': datetime.timedelta(microseconds=47210), 'request': <PreparedRequest [POST]>, 'connection': <requests.adapters.HTTPAdapter object at 0x7fae202930a0>}
02/11/2024 12:30:02 PM - osm_mon.dashboarder.backends.grafana - INFO - Dashboard k8stest202402111208 is created in Grafana
02/11/2024 12:32:01 PM - aiokafka.consumer.group_coordinator - INFO - Metadata for topic has changed from {'users': 1, 'project': 0} to {'users': 1, 'project': 1}. 
02/11/2024 12:32:01 PM - aiokafka.consumer.group_coordinator - INFO - Revoking previously assigned partitions frozenset({TopicPartition(topic='users', partition=0)}) for group mon-consumer
02/11/2024 12:32:01 PM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer
02/11/2024 12:32:01 PM - aiokafka.consumer.group_coordinator - INFO - Joined group 'mon-consumer' (generation 2) with member_id aiokafka-0.8.1-53085e07-ab76-4bf5-975c-99cbcc4f09a3
02/11/2024 12:32:01 PM - aiokafka.consumer.group_coordinator - INFO - Elected group leader -- performing partition assignments using roundrobin
02/11/2024 12:32:01 PM - aiokafka.consumer.group_coordinator - INFO - Successfully synced group mon-consumer with generation 2
02/11/2024 12:32:01 PM - aiokafka.consumer.group_coordinator - INFO - Setting newly assigned partitions {TopicPartition(topic='users', partition=0), TopicPartition(topic='project', partition=0)} for group mon-consumer
02/11/2024 01:49:14 PM - osm_mon.dashboarder.backends.grafana - INFO - Dashboard admin is created in Grafana
02/11/2024 04:22:20 PM - osm_mon.dashboarder.backends.grafana - INFO - New user basic_15_test_user created in Grafana
02/11/2024 04:22:23 PM - osm_mon.dashboarder.backends.grafana - INFO - User basic_15_test_user is assigned Admin permission
02/11/2024 04:22:28 PM - osm_mon.dashboarder.dashboarder - ERROR - Exception processing message: 
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/osm_mon/dashboarder/dashboarder.py", line 116, in _user_msg
    project_data = values["changes"]["add_project_role_mappings"]
KeyError: 'add_project_role_mappings'
02/11/2024 04:22:30 PM - osm_mon.dashboarder.backends.grafana - INFO - New team basic_15_test_project created in Grafana
02/11/2024 04:22:32 PM - osm_mon.dashboarder.backends.grafana - INFO - Grafana team updated {"message":"Team updated"}
02/11/2024 04:22:37 PM - osm_mon.dashboarder.backends.grafana - INFO - Dashboard folder basic_15_project_test created
02/11/2024 04:22:37 PM - osm_mon.dashboarder.backends.grafana - INFO - Dashboard basic_15_project_test is created in Grafana
02/11/2024 04:22:41 PM - osm_mon.dashboarder.backends.grafana - INFO - User basic_15_test_user deleted in Grafana
02/11/2024 04:22:43 PM - osm_mon.dashboarder.backends.grafana - INFO - Team basic_15_project_test deleted in Grafana
02/11/2024 04:23:08 PM - osm_mon.dashboarder.backends.grafana - INFO - Dashboard admin is created in Grafana
02/11/2024 05:53:22 PM - osm_mon.dashboarder.backends.grafana - INFO - Datasource prom-operator-k8s02 is created in Grafana
02/11/2024 05:53:22 PM - osm_mon.dashboarder.backends.grafana - INFO - ************* response: {'_content': b'{"datasource":{"id":4,"uid":"b938c609-f338-4b70-8647-2ef648518013","orgId":1,"name":"prom-operator-k8s02","type":"prometheus","typeLogoUrl":"public/app/plugins/datasource/prometheus/img/prometheus_logo.svg","access":"proxy","url":"http://172.21.23.12:9090","user":"","database":"","basicAuth":false,"basicAuthUser":"","withCredentials":false,"isDefault":false,"jsonData":{},"secureJsonFields":{},"version":1,"readOnly":false},"id":4,"message":"Datasource added","name":"prom-operator-k8s02"}', '_content_consumed': True, '_next': None, 'status_code': 200, 'headers': {'Cache-Control': 'no-store', 'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff', 'X-Frame-Options': 'deny', 'X-Xss-Protection': '1; mode=block', 'Date': 'Sun, 11 Feb 2024 17:53:22 GMT', 'Content-Length': '491'}, 'raw': <urllib3.response.HTTPResponse object at 0x7fae2027b0d0>, 'url': 'http://grafana:3000/api/datasources', 'encoding': 'utf-8', 'history': [], 'reason': 'OK', 'cookies': <RequestsCookieJar[]>, 'elapsed': datetime.timedelta(microseconds=91920), 'request': <PreparedRequest [POST]>, 'connection': <requests.adapters.HTTPAdapter object at 0x7fae2027b070>}
02/11/2024 05:53:22 PM - osm_mon.dashboarder.backends.grafana - INFO - Dashboard k8s02 is created in Grafana
02/11/2024 07:10:05 PM - osm_mon.dashboarder.backends.grafana - INFO - Datasource prom-operator-k8s07 is created in Grafana
02/11/2024 07:10:05 PM - osm_mon.dashboarder.backends.grafana - INFO - ************* response: {'_content': b'{"datasource":{"id":5,"uid":"ebf66d7b-1df3-4f0a-8edd-c4f7cbb2ca7b","orgId":1,"name":"prom-operator-k8s07","type":"prometheus","typeLogoUrl":"public/app/plugins/datasource/prometheus/img/prometheus_logo.svg","access":"proxy","url":"http://172.21.23.12:9090","user":"","database":"","basicAuth":false,"basicAuthUser":"","withCredentials":false,"isDefault":false,"jsonData":{},"secureJsonFields":{},"version":1,"readOnly":false},"id":5,"message":"Datasource added","name":"prom-operator-k8s07"}', '_content_consumed': True, '_next': None, 'status_code': 200, 'headers': {'Cache-Control': 'no-store', 'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff', 'X-Frame-Options': 'deny', 'X-Xss-Protection': '1; mode=block', 'Date': 'Sun, 11 Feb 2024 19:10:05 GMT', 'Content-Length': '491'}, 'raw': <urllib3.response.HTTPResponse object at 0x7fae20279e70>, 'url': 'http://grafana:3000/api/datasources', 'encoding': 'utf-8', 'history': [], 'reason': 'OK', 'cookies': <RequestsCookieJar[]>, 'elapsed': datetime.timedelta(microseconds=35885), 'request': <PreparedRequest [POST]>, 'connection': <requests.adapters.HTTPAdapter object at 0x7fae202791b0>}
02/11/2024 07:10:05 PM - osm_mon.dashboarder.backends.grafana - INFO - Dashboard k8s07 is created in Grafana
02/11/2024 07:49:41 PM - osm_mon.dashboarder.backends.grafana - INFO - Datasource prom-operator-k8s12 is created in Grafana
02/11/2024 07:49:41 PM - osm_mon.dashboarder.backends.grafana - INFO - ************* response: {'_content': b'{"datasource":{"id":6,"uid":"a5e691b5-ce77-4f27-8af3-d72058fd8c80","orgId":1,"name":"prom-operator-k8s12","type":"prometheus","typeLogoUrl":"public/app/plugins/datasource/prometheus/img/prometheus_logo.svg","access":"proxy","url":"http://172.21.23.12:9090","user":"","database":"","basicAuth":false,"basicAuthUser":"","withCredentials":false,"isDefault":false,"jsonData":{},"secureJsonFields":{},"version":1,"readOnly":false},"id":6,"message":"Datasource added","name":"prom-operator-k8s12"}', '_content_consumed': True, '_next': None, 'status_code': 200, 'headers': {'Cache-Control': 'no-store', 'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff', 'X-Frame-Options': 'deny', 'X-Xss-Protection': '1; mode=block', 'Date': 'Sun, 11 Feb 2024 19:49:41 GMT', 'Content-Length': '491'}, 'raw': <urllib3.response.HTTPResponse object at 0x7fae20279c60>, 'url': 'http://grafana:3000/api/datasources', 'encoding': 'utf-8', 'history': [], 'reason': 'OK', 'cookies': <RequestsCookieJar[]>, 'elapsed': datetime.timedelta(microseconds=41868), 'request': <PreparedRequest [POST]>, 'connection': <requests.adapters.HTTPAdapter object at 0x7fae20278fd0>}
02/11/2024 07:49:41 PM - osm_mon.dashboarder.backends.grafana - INFO - Dashboard k8s12 is created in Grafana
02/11/2024 08:30:18 PM - osm_mon.dashboarder.backends.grafana - INFO - Dashboard admin is created in Grafana