2024-04-28T01:28:15.628768123Z nc: kafka (10.99.104.130:9092): Operation timed out 2024-04-28T01:28:19.634751161Z nc: kafka (10.99.104.130:9092): Operation timed out 2024-04-28T01:28:23.635137877Z nc: kafka (10.99.104.130:9092): Operation timed out 2024-04-28T01:28:27.637852294Z nc: kafka (10.99.104.130:9092): Operation timed out 2024-04-28T01:28:31.644533103Z nc: kafka (10.99.104.130:9092): Operation timed out 2024-04-28T01:28:35.647020344Z nc: kafka (10.99.104.130:9092): Operation timed out 2024-04-28T01:28:39.651409619Z nc: kafka (10.99.104.130:9092): Operation timed out 2024-04-28T01:28:43.653668418Z nc: kafka (10.99.104.130:9092): Operation timed out 2024-04-28T01:28:47.660784706Z nc: kafka (10.99.104.130:9092): Operation timed out 2024-04-28T01:28:51.676960507Z nc: kafka (10.99.104.130:9092): Operation timed out 2024-04-28T01:28:55.681539526Z nc: kafka (10.99.104.130:9092): Operation timed out 2024-04-28T01:28:59.684561607Z nc: kafka (10.99.104.130:9092): Operation timed out 2024-04-28T01:29:03.687662706Z nc: kafka (10.99.104.130:9092): Operation timed out 2024-04-28T01:29:07.691059651Z nc: kafka (10.99.104.130:9092): Operation timed out 2024-04-28T01:29:11.693596233Z nc: kafka (10.99.104.130:9092): Operation timed out 2024-04-28T01:29:15.729434509Z nc: kafka (10.99.104.130:9092): Operation timed out 2024-04-28T01:29:19.738474475Z nc: kafka (10.99.104.130:9092): Operation timed out 2024-04-28T01:29:23.746473212Z nc: kafka (10.99.104.130:9092): Operation timed out 2024-04-28T01:29:27.752280829Z nc: kafka (10.99.104.130:9092): Operation timed out 2024-04-28T01:29:36.803510626Z kafka (10.99.104.130:9092) open 2024-04-28T01:29:36.803532888Z mongodb-k8s (10.244.0.30:27017) open 2024-04-28T01:31:00.625432046Z 04/28/2024 01:31:00 AM - kafka.conn - INFO - [IPv4 ('10.99.104.130', 9092)]>: connecting to kafka:9092 [('10.99.104.130', 9092) IPv4] 2024-04-28T01:31:00.627745508Z 04/28/2024 01:31:00 AM - kafka.conn - INFO - Probing node bootstrap-0 broker version 2024-04-28T01:31:00.630995402Z 04/28/2024 01:31:00 AM - kafka.conn - INFO - [IPv4 ('10.99.104.130', 9092)]>: Connection complete. 2024-04-28T01:31:00.743248710Z 04/28/2024 01:31:00 AM - kafka.conn - INFO - Broker version identified as 2.5.0 2024-04-28T01:31:00.743330746Z 04/28/2024 01:31:00 AM - kafka.conn - INFO - Set configuration api_version=(2, 5, 0) to skip auto check_version requests on startup 2024-04-28T01:31:00.765045273Z 04/28/2024 01:31:00 AM - kafka.conn - INFO - [IPv4 ('10.99.104.130', 9092)]>: connecting to kafka:9092 [('10.99.104.130', 9092) IPv4] 2024-04-28T01:31:00.765063028Z 04/28/2024 01:31:00 AM - kafka.conn - INFO - Probing node bootstrap-0 broker version 2024-04-28T01:31:00.769884913Z 04/28/2024 01:31:00 AM - kafka.conn - INFO - [IPv4 ('10.99.104.130', 9092)]>: Connection complete. 2024-04-28T01:31:00.951244472Z 04/28/2024 01:31:00 AM - kafka.conn - INFO - Broker version identified as 2.5.0 2024-04-28T01:31:00.951285597Z 04/28/2024 01:31:00 AM - kafka.conn - INFO - Set configuration api_version=(2, 5, 0) to skip auto check_version requests on startup 2024-04-28T01:31:01.136386476Z 04/28/2024 01:31:01 AM - kafka.conn - INFO - [IPv4 ('10.244.0.32', 9092)]>: connecting to kafka-controller-2.kafka-controller-headless.osm.svc.cluster.local:9092 [('10.244.0.32', 9092) IPv4] 2024-04-28T01:31:01.143100856Z 04/28/2024 01:31:01 AM - kafka.conn - INFO - [IPv4 ('10.244.0.32', 9092)]>: Connection complete. 2024-04-28T01:31:01.143386513Z 04/28/2024 01:31:01 AM - kafka.conn - INFO - [IPv4 ('10.99.104.130', 9092)]>: Closing connection. 2024-04-28T01:31:01.369488309Z 04/28/2024 01:31:01 AM - kafka.conn - INFO - [IPv4 ('10.244.0.37', 9092)]>: connecting to kafka-controller-0.kafka-controller-headless.osm.svc.cluster.local:9092 [('10.244.0.37', 9092) IPv4] 2024-04-28T01:31:01.369510915Z 04/28/2024 01:31:01 AM - kafka.conn - INFO - [IPv4 ('10.244.0.37', 9092)]>: Connection complete. 2024-04-28T01:31:01.407694730Z 04/28/2024 01:31:01 AM - kafka.producer.record_accumulator - WARNING - Produced messages to topic-partition TopicPartition(topic='alarm_request', partition=0) with base offset -1 log start offset -1 and error None. 2024-04-28T01:31:01.540255938Z 04/28/2024 01:31:01 AM - kafka.cluster - ERROR - Topic users not found in cluster metadata 2024-04-28T01:31:01.540274337Z 04/28/2024 01:31:01 AM - kafka.cluster - ERROR - Topic alarm_request not found in cluster metadata 2024-04-28T01:31:01.655228561Z 04/28/2024 01:31:01 AM - kafka.cluster - ERROR - Topic users not found in cluster metadata 2024-04-28T01:31:01.830985713Z 04/28/2024 01:31:01 AM - kafka.cluster - ERROR - Topic users not found in cluster metadata 2024-04-28T01:31:01.942174844Z 04/28/2024 01:31:01 AM - kafka.cluster - ERROR - Topic users not found in cluster metadata 2024-04-28T01:31:02.110883644Z 04/28/2024 01:31:02 AM - kafka.producer.record_accumulator - WARNING - Produced messages to topic-partition TopicPartition(topic='users', partition=0) with base offset -1 log start offset -1 and error None. 2024-04-28T01:31:02.226472008Z 04/28/2024 01:31:02 AM - kafka.cluster - ERROR - Topic project not found in cluster metadata 2024-04-28T01:31:02.419506088Z 04/28/2024 01:31:02 AM - kafka.conn - INFO - [IPv4 ('10.244.0.32', 9092)]>: Closing connection. 2024-04-28T01:31:02.419542398Z 04/28/2024 01:31:02 AM - kafka.conn - INFO - [IPv4 ('10.244.0.37', 9092)]>: Closing connection. 2024-04-28T01:31:02.420003836Z 04/28/2024 01:31:02 AM - osm_mon.cmd.mon_dashboarder - INFO - Starting MON Dashboarder... 2024-04-28T01:31:02.502800468Z 04/28/2024 01:31:02 AM - db - INFO - Connected to database osm version 1.2 2024-04-28T01:31:02.502913027Z 04/28/2024 01:31:02 AM - osm_mon.cmd.mon_dashboarder - INFO - Starting MON kafka Consumer... 2024-04-28T01:31:02.512387868Z 04/28/2024 01:31:02 AM - osm_mon.cmd.mon_dashboarder - INFO - Starting MON Dashboarder... 2024-04-28T01:31:02.625488115Z 04/28/2024 01:31:02 AM - aiokafka.consumer.subscription_state - INFO - Updating subscribed topics to: frozenset({'project', 'users'}) 2024-04-28T01:31:02.627639082Z 04/28/2024 01:31:02 AM - aiokafka.consumer.consumer - INFO - Subscribed to topic(s): {'project', 'users'} 2024-04-28T01:31:02.750311488Z 04/28/2024 01:31:02 AM - aiokafka.consumer.group_coordinator - ERROR - Group Coordinator Request failed: [Error 15] CoordinatorNotAvailableError 2024-04-28T01:31:02.910617778Z 04/28/2024 01:31:02 AM - aiokafka.consumer.group_coordinator - ERROR - Group Coordinator Request failed: [Error 15] CoordinatorNotAvailableError 2024-04-28T01:31:03.062468189Z 04/28/2024 01:31:03 AM - aiokafka.consumer.group_coordinator - ERROR - Group Coordinator Request failed: [Error 15] CoordinatorNotAvailableError 2024-04-28T01:31:03.194138289Z 04/28/2024 01:31:03 AM - osm_mon.dashboarder.backends.grafana - INFO - Dashboard admin is created in Grafana 2024-04-28T01:31:03.222481337Z 04/28/2024 01:31:03 AM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 0 for group mon-consumer 2024-04-28T01:31:03.222499707Z 04/28/2024 01:31:03 AM - aiokafka.consumer.group_coordinator - INFO - Revoking previously assigned partitions set() for group mon-consumer 2024-04-28T01:31:03.222504408Z 04/28/2024 01:31:03 AM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-04-28T01:31:03.238285591Z 04/28/2024 01:31:03 AM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 0)for group mon-consumer. 2024-04-28T01:31:03.388573161Z 04/28/2024 01:31:03 AM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 0 for group mon-consumer 2024-04-28T01:31:03.388607515Z 04/28/2024 01:31:03 AM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-04-28T01:31:03.398441557Z 04/28/2024 01:31:03 AM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 0)for group mon-consumer. 2024-04-28T01:31:03.554502934Z 04/28/2024 01:31:03 AM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 0 for group mon-consumer 2024-04-28T01:31:03.554521404Z 04/28/2024 01:31:03 AM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-04-28T01:31:03.606231109Z 04/28/2024 01:31:03 AM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 0)for group mon-consumer. 2024-04-28T01:31:03.762460309Z 04/28/2024 01:31:03 AM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 0 for group mon-consumer 2024-04-28T01:31:03.762476325Z 04/28/2024 01:31:03 AM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-04-28T01:31:03.778648747Z 04/28/2024 01:31:03 AM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 0)for group mon-consumer. 2024-04-28T01:31:03.895592774Z 04/28/2024 01:31:03 AM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 0 for group mon-consumer 2024-04-28T01:31:03.895601346Z 04/28/2024 01:31:03 AM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-04-28T01:31:03.915149095Z 04/28/2024 01:31:03 AM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 0)for group mon-consumer. 2024-04-28T01:31:04.042516184Z 04/28/2024 01:31:04 AM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 0 for group mon-consumer 2024-04-28T01:31:04.045000993Z 04/28/2024 01:31:04 AM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-04-28T01:31:04.052929738Z 04/28/2024 01:31:04 AM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 0)for group mon-consumer. 2024-04-28T01:31:04.170434011Z 04/28/2024 01:31:04 AM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 0 for group mon-consumer 2024-04-28T01:31:04.170450610Z 04/28/2024 01:31:04 AM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-04-28T01:31:04.178456134Z 04/28/2024 01:31:04 AM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 0)for group mon-consumer. 2024-04-28T01:31:04.285110478Z 04/28/2024 01:31:04 AM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 0 for group mon-consumer 2024-04-28T01:31:04.307427045Z 04/28/2024 01:31:04 AM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-04-28T01:31:04.307441476Z 04/28/2024 01:31:04 AM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 0)for group mon-consumer. 2024-04-28T01:31:04.397490357Z 04/28/2024 01:31:04 AM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 0 for group mon-consumer 2024-04-28T01:31:04.397512686Z 04/28/2024 01:31:04 AM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-04-28T01:31:04.411010655Z 04/28/2024 01:31:04 AM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 0)for group mon-consumer. 2024-04-28T01:31:04.549035250Z 04/28/2024 01:31:04 AM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 0 for group mon-consumer 2024-04-28T01:31:04.549053406Z 04/28/2024 01:31:04 AM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-04-28T01:31:04.549058071Z 04/28/2024 01:31:04 AM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 0)for group mon-consumer. 2024-04-28T01:31:04.654184856Z 04/28/2024 01:31:04 AM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 0 for group mon-consumer 2024-04-28T01:31:04.654206675Z 04/28/2024 01:31:04 AM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-04-28T01:31:04.670722284Z 04/28/2024 01:31:04 AM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 0)for group mon-consumer. 2024-04-28T01:31:04.771820151Z 04/28/2024 01:31:04 AM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 0 for group mon-consumer 2024-04-28T01:31:04.771835662Z 04/28/2024 01:31:04 AM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-04-28T01:31:04.788789522Z 04/28/2024 01:31:04 AM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 0)for group mon-consumer. 2024-04-28T01:31:04.907901065Z 04/28/2024 01:31:04 AM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 0 for group mon-consumer 2024-04-28T01:31:04.907914724Z 04/28/2024 01:31:04 AM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-04-28T01:31:04.912803347Z 04/28/2024 01:31:04 AM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 0)for group mon-consumer. 2024-04-28T01:31:05.052427122Z 04/28/2024 01:31:05 AM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 0 for group mon-consumer 2024-04-28T01:31:05.052451133Z 04/28/2024 01:31:05 AM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-04-28T01:31:05.059132355Z 04/28/2024 01:31:05 AM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 0)for group mon-consumer. 2024-04-28T01:31:05.170079380Z 04/28/2024 01:31:05 AM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 0 for group mon-consumer 2024-04-28T01:31:05.170096125Z 04/28/2024 01:31:05 AM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-04-28T01:31:05.174896542Z 04/28/2024 01:31:05 AM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 0)for group mon-consumer. 2024-04-28T01:31:05.317235622Z 04/28/2024 01:31:05 AM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 0 for group mon-consumer 2024-04-28T01:31:05.317257903Z 04/28/2024 01:31:05 AM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-04-28T01:31:05.409582944Z 04/28/2024 01:31:05 AM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 0)for group mon-consumer. 2024-04-28T01:31:05.517232253Z 04/28/2024 01:31:05 AM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 0 for group mon-consumer 2024-04-28T01:31:05.517304093Z 04/28/2024 01:31:05 AM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-04-28T01:31:05.519738092Z 04/28/2024 01:31:05 AM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 0)for group mon-consumer. 2024-04-28T01:31:05.627852813Z 04/28/2024 01:31:05 AM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 0 for group mon-consumer 2024-04-28T01:31:05.627885730Z 04/28/2024 01:31:05 AM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-04-28T01:31:05.660463186Z 04/28/2024 01:31:05 AM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 0)for group mon-consumer. 2024-04-28T01:31:05.782815305Z 04/28/2024 01:31:05 AM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 0 for group mon-consumer 2024-04-28T01:31:05.782889162Z 04/28/2024 01:31:05 AM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-04-28T01:31:05.806757775Z 04/28/2024 01:31:05 AM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 0)for group mon-consumer. 2024-04-28T01:31:05.927788434Z 04/28/2024 01:31:05 AM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 0 for group mon-consumer 2024-04-28T01:31:05.927802063Z 04/28/2024 01:31:05 AM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-04-28T01:31:05.933359805Z 04/28/2024 01:31:05 AM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 0)for group mon-consumer. 2024-04-28T01:31:06.038480016Z 04/28/2024 01:31:06 AM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 0 for group mon-consumer 2024-04-28T01:31:06.038498822Z 04/28/2024 01:31:06 AM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-04-28T01:31:06.042055655Z 04/28/2024 01:31:06 AM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 0)for group mon-consumer. 2024-04-28T01:31:06.165775444Z 04/28/2024 01:31:06 AM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 0 for group mon-consumer 2024-04-28T01:31:06.165792250Z 04/28/2024 01:31:06 AM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-04-28T01:31:06.167746280Z 04/28/2024 01:31:06 AM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 0)for group mon-consumer. 2024-04-28T01:31:06.271061331Z 04/28/2024 01:31:06 AM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 0 for group mon-consumer 2024-04-28T01:31:06.271121328Z 04/28/2024 01:31:06 AM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-04-28T01:31:06.288356730Z 04/28/2024 01:31:06 AM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 0)for group mon-consumer. 2024-04-28T01:31:06.486119524Z 04/28/2024 01:31:06 AM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 0 for group mon-consumer 2024-04-28T01:31:06.486136058Z 04/28/2024 01:31:06 AM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-04-28T01:31:06.563874920Z 04/28/2024 01:31:06 AM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 0)for group mon-consumer. 2024-04-28T01:31:06.683713771Z 04/28/2024 01:31:06 AM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 0 for group mon-consumer 2024-04-28T01:31:06.683766693Z 04/28/2024 01:31:06 AM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-04-28T01:31:06.685475741Z 04/28/2024 01:31:06 AM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 0)for group mon-consumer. 2024-04-28T01:31:06.842455583Z 04/28/2024 01:31:06 AM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 0 for group mon-consumer 2024-04-28T01:31:06.842485560Z 04/28/2024 01:31:06 AM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-04-28T01:31:06.879775618Z 04/28/2024 01:31:06 AM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 0)for group mon-consumer. 2024-04-28T01:31:06.992613083Z 04/28/2024 01:31:06 AM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 0 for group mon-consumer 2024-04-28T01:31:06.992630481Z 04/28/2024 01:31:06 AM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-04-28T01:31:07.013486443Z 04/28/2024 01:31:07 AM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 0)for group mon-consumer. 2024-04-28T01:31:07.163662133Z 04/28/2024 01:31:07 AM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 0 for group mon-consumer 2024-04-28T01:31:07.163677357Z 04/28/2024 01:31:07 AM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-04-28T01:31:10.531671902Z 04/28/2024 01:31:10 AM - aiokafka.consumer.group_coordinator - INFO - Joined group 'mon-consumer' (generation 1) with member_id aiokafka-0.8.1-9d7e43df-2b5a-483d-9988-702a4429945e 2024-04-28T01:31:10.531690817Z 04/28/2024 01:31:10 AM - aiokafka.consumer.group_coordinator - INFO - Elected group leader -- performing partition assignments using roundrobin 2024-04-28T01:31:10.790502887Z 04/28/2024 01:31:10 AM - aiokafka.consumer.group_coordinator - INFO - Successfully synced group mon-consumer with generation 1 2024-04-28T01:31:10.790523524Z 04/28/2024 01:31:10 AM - aiokafka.consumer.group_coordinator - INFO - Setting newly assigned partitions {TopicPartition(topic='users', partition=0), TopicPartition(topic='project', partition=0)} for group mon-consumer 2024-04-28T01:38:35.157636789Z 04/28/2024 01:38:35 AM - osm_mon.dashboarder.backends.grafana - INFO - Datasource prom-operator-osm is created in Grafana 2024-04-28T01:38:35.157964698Z 04/28/2024 01:38:35 AM - osm_mon.dashboarder.backends.grafana - INFO - ************* response: {'_content': b'{"datasource":{"id":3,"uid":"f44e7327-ac6a-4a30-9259-887c7fae44c1","orgId":1,"name":"prom-operator-osm","type":"prometheus","typeLogoUrl":"public/app/plugins/datasource/prometheus/img/prometheus_logo.svg","access":"proxy","url":"http://172.21.248.125:9090","user":"","database":"","basicAuth":false,"basicAuthUser":"","withCredentials":false,"isDefault":false,"jsonData":{},"secureJsonFields":{},"version":1,"readOnly":false},"id":3,"message":"Datasource added","name":"prom-operator-osm"}', '_content_consumed': True, '_next': None, 'status_code': 200, 'headers': {'Cache-Control': 'no-store', 'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff', 'X-Frame-Options': 'deny', 'X-Xss-Protection': '1; mode=block', 'Date': 'Sun, 28 Apr 2024 01:38:35 GMT', 'Content-Length': '489'}, 'raw': , 'url': 'http://grafana:3000/api/datasources', 'encoding': 'utf-8', 'history': [], 'reason': 'OK', 'cookies': , 'elapsed': datetime.timedelta(microseconds=42326), 'request': , 'connection': } 2024-04-28T01:38:35.369614379Z 04/28/2024 01:38:35 AM - osm_mon.dashboarder.backends.grafana - INFO - Dashboard osm is created in Grafana 2024-04-28T01:45:07.057824649Z 04/28/2024 01:45:07 AM - osm_mon.dashboarder.backends.grafana - INFO - Dashboard admin is created in Grafana 2024-04-28T01:47:25.485918521Z 04/28/2024 01:47:25 AM - osm_mon.dashboarder.backends.grafana - INFO - New user basic_15_test_user created in Grafana 2024-04-28T01:47:28.720630154Z 04/28/2024 01:47:28 AM - osm_mon.dashboarder.backends.grafana - INFO - User basic_15_test_user is assigned Admin permission 2024-04-28T01:47:33.399758458Z 04/28/2024 01:47:33 AM - osm_mon.dashboarder.dashboarder - ERROR - Exception processing message: 2024-04-28T01:47:33.399794344Z Traceback (most recent call last): 2024-04-28T01:47:33.399801245Z File "/usr/lib/python3/dist-packages/osm_mon/dashboarder/dashboarder.py", line 116, in _user_msg 2024-04-28T01:47:33.399807687Z project_data = values["changes"]["add_project_role_mappings"] 2024-04-28T01:47:33.399813643Z KeyError: 'add_project_role_mappings' 2024-04-28T01:47:34.650490191Z 04/28/2024 01:47:34 AM - osm_mon.dashboarder.backends.grafana - INFO - New team basic_15_test_project created in Grafana 2024-04-28T01:47:36.274833028Z 04/28/2024 01:47:36 AM - osm_mon.dashboarder.backends.grafana - INFO - Grafana team updated {"message":"Team updated"} 2024-04-28T01:47:37.600353042Z 04/28/2024 01:47:37 AM - osm_mon.dashboarder.backends.grafana - INFO - Dashboard folder basic_15_project_test created 2024-04-28T01:47:37.773209163Z 04/28/2024 01:47:37 AM - osm_mon.dashboarder.backends.grafana - INFO - Dashboard basic_15_project_test is created in Grafana 2024-04-28T01:47:44.352344729Z 04/28/2024 01:47:44 AM - osm_mon.dashboarder.backends.grafana - INFO - User basic_15_test_user deleted in Grafana 2024-04-28T01:47:46.028283658Z 04/28/2024 01:47:46 AM - osm_mon.dashboarder.backends.grafana - INFO - Team basic_15_project_test deleted in Grafana 2024-04-28T01:48:08.065126415Z 04/28/2024 01:48:08 AM - osm_mon.dashboarder.backends.grafana - INFO - Dashboard admin is created in Grafana 2024-04-28T02:57:52.540235760Z 04/28/2024 02:57:52 AM - osm_mon.dashboarder.backends.grafana - INFO - Datasource prom-operator-k8s02 is created in Grafana 2024-04-28T02:57:52.540758452Z 04/28/2024 02:57:52 AM - osm_mon.dashboarder.backends.grafana - INFO - ************* response: {'_content': b'{"datasource":{"id":4,"uid":"bc840350-33b6-4581-8498-8c6a4cf24efd","orgId":1,"name":"prom-operator-k8s02","type":"prometheus","typeLogoUrl":"public/app/plugins/datasource/prometheus/img/prometheus_logo.svg","access":"proxy","url":"http://172.21.248.125:9090","user":"","database":"","basicAuth":false,"basicAuthUser":"","withCredentials":false,"isDefault":false,"jsonData":{},"secureJsonFields":{},"version":1,"readOnly":false},"id":4,"message":"Datasource added","name":"prom-operator-k8s02"}', '_content_consumed': True, '_next': None, 'status_code': 200, 'headers': {'Cache-Control': 'no-store', 'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff', 'X-Frame-Options': 'deny', 'X-Xss-Protection': '1; mode=block', 'Date': 'Sun, 28 Apr 2024 02:57:52 GMT', 'Content-Length': '493'}, 'raw': , 'url': 'http://grafana:3000/api/datasources', 'encoding': 'utf-8', 'history': [], 'reason': 'OK', 'cookies': , 'elapsed': datetime.timedelta(microseconds=44356), 'request': , 'connection': } 2024-04-28T02:57:52.708015445Z 04/28/2024 02:57:52 AM - osm_mon.dashboarder.backends.grafana - INFO - Dashboard k8s02 is created in Grafana 2024-04-28T03:01:53.666713638Z 04/28/2024 03:01:53 AM - osm_mon.dashboarder.backends.grafana - INFO - Datasource prom-operator-k8s07 is created in Grafana 2024-04-28T03:01:53.666770024Z 04/28/2024 03:01:53 AM - osm_mon.dashboarder.backends.grafana - INFO - ************* response: {'_content': b'{"datasource":{"id":5,"uid":"e4b68894-7c63-4c55-9993-d604c4884d95","orgId":1,"name":"prom-operator-k8s07","type":"prometheus","typeLogoUrl":"public/app/plugins/datasource/prometheus/img/prometheus_logo.svg","access":"proxy","url":"http://172.21.248.125:9090","user":"","database":"","basicAuth":false,"basicAuthUser":"","withCredentials":false,"isDefault":false,"jsonData":{},"secureJsonFields":{},"version":1,"readOnly":false},"id":5,"message":"Datasource added","name":"prom-operator-k8s07"}', '_content_consumed': True, '_next': None, 'status_code': 200, 'headers': {'Cache-Control': 'no-store', 'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff', 'X-Frame-Options': 'deny', 'X-Xss-Protection': '1; mode=block', 'Date': 'Sun, 28 Apr 2024 03:01:53 GMT', 'Content-Length': '493'}, 'raw': , 'url': 'http://grafana:3000/api/datasources', 'encoding': 'utf-8', 'history': [], 'reason': 'OK', 'cookies': , 'elapsed': datetime.timedelta(microseconds=36316), 'request': , 'connection': } 2024-04-28T03:01:53.829243768Z 04/28/2024 03:01:53 AM - osm_mon.dashboarder.backends.grafana - INFO - Dashboard k8s07 is created in Grafana 2024-04-28T03:11:02.872035072Z 04/28/2024 03:11:02 AM - osm_mon.dashboarder.backends.grafana - INFO - New team quotas_01_proj_1 created in Grafana 2024-04-28T03:11:09.315359805Z 04/28/2024 03:11:09 AM - osm_mon.dashboarder.backends.grafana - INFO - New team quotas_01_proj_2 created in Grafana 2024-04-28T03:11:15.829011321Z 04/28/2024 03:11:15 AM - osm_mon.dashboarder.backends.grafana - INFO - New user quotas_01_user created in Grafana 2024-04-28T03:11:15.944097297Z 04/28/2024 03:11:15 AM - osm_mon.dashboarder.backends.grafana - INFO - User quotas_01_user is assigned Editor permission 2024-04-28T03:11:16.101573799Z 04/28/2024 03:11:16 AM - osm_mon.dashboarder.backends.grafana - INFO - User quotas_01_user is assigned Editor permission 2024-04-28T03:11:20.499120893Z 04/28/2024 03:11:20 AM - osm_mon.dashboarder.dashboarder - ERROR - Exception processing message: 2024-04-28T03:11:20.499169705Z Traceback (most recent call last): 2024-04-28T03:11:20.499179974Z File "/usr/lib/python3/dist-packages/osm_mon/dashboarder/dashboarder.py", line 116, in _user_msg 2024-04-28T03:11:20.499215276Z project_data = values["changes"]["add_project_role_mappings"] 2024-04-28T03:11:20.499228255Z KeyError: 'add_project_role_mappings' 2024-04-28T03:11:22.257090760Z 04/28/2024 03:11:22 AM - osm_mon.dashboarder.backends.grafana - INFO - Grafana team updated {"message":"Team updated"} 2024-04-28T03:11:26.056992262Z 04/28/2024 03:11:26 AM - osm_mon.dashboarder.backends.grafana - INFO - Dashboard folder quotas_01_proj_3 created 2024-04-28T03:11:26.231547004Z 04/28/2024 03:11:26 AM - osm_mon.dashboarder.backends.grafana - INFO - Dashboard quotas_01_proj_3 is created in Grafana 2024-04-28T03:11:26.278639116Z 04/28/2024 03:11:26 AM - osm_mon.dashboarder.backends.grafana - INFO - Dashboard folder quotas_01_proj_2 created 2024-04-28T03:11:26.466951389Z 04/28/2024 03:11:26 AM - osm_mon.dashboarder.backends.grafana - INFO - Dashboard quotas_01_proj_2 is created in Grafana 2024-04-28T03:11:31.973833732Z 04/28/2024 03:11:31 AM - osm_mon.dashboarder.dashboarder - ERROR - Exception processing message: 2024-04-28T03:11:31.973873150Z Traceback (most recent call last): 2024-04-28T03:11:31.973882850Z File "/usr/lib/python3/dist-packages/osm_mon/dashboarder/dashboarder.py", line 129, in _user_msg 2024-04-28T03:11:31.973891823Z project_new_name = values["changes"]["name"] 2024-04-28T03:11:31.973900998Z KeyError: 'name' 2024-04-28T03:12:02.382127713Z 04/28/2024 03:12:02 AM - osm_mon.dashboarder.backends.grafana - INFO - Team quotas_01_proj_2 deleted in Grafana 2024-04-28T03:12:04.095433066Z 04/28/2024 03:12:04 AM - osm_mon.dashboarder.backends.grafana - INFO - User quotas_01_user deleted in Grafana 2024-04-28T03:12:05.798090113Z 04/28/2024 03:12:05 AM - osm_mon.dashboarder.backends.grafana - INFO - Team quotas_01_proj_3 deleted in Grafana 2024-04-28T03:12:26.863909597Z 04/28/2024 03:12:26 AM - osm_mon.dashboarder.backends.grafana - INFO - Dashboard admin is created in Grafana 2024-04-28T03:16:27.833147095Z 04/28/2024 03:16:27 AM - osm_mon.dashboarder.backends.grafana - INFO - Dashboard admin is created in Grafana 2024-04-28T03:32:31.282249942Z 04/28/2024 03:32:31 AM - osm_mon.dashboarder.backends.grafana - INFO - Dashboard admin is created in Grafana 2024-04-28T03:53:05.964993171Z 04/28/2024 03:53:05 AM - osm_mon.dashboarder.backends.grafana - INFO - Dashboard admin is created in Grafana