2024-04-03T01:23:18.829679997Z nc: kafka (10.101.96.56:9092): Operation timed out 2024-04-03T01:23:22.833409699Z nc: kafka (10.101.96.56:9092): Operation timed out 2024-04-03T01:23:26.840065688Z nc: kafka (10.101.96.56:9092): Operation timed out 2024-04-03T01:23:30.846234708Z nc: kafka (10.101.96.56:9092): Operation timed out 2024-04-03T01:23:34.848352553Z nc: kafka (10.101.96.56:9092): Operation timed out 2024-04-03T01:23:38.852081092Z nc: kafka (10.101.96.56:9092): Operation timed out 2024-04-03T01:23:42.855023833Z nc: kafka (10.101.96.56:9092): Operation timed out 2024-04-03T01:23:46.860306152Z nc: kafka (10.101.96.56:9092): Operation timed out 2024-04-03T01:23:50.862563451Z nc: kafka (10.101.96.56:9092): Operation timed out 2024-04-03T01:23:54.865397683Z nc: kafka (10.101.96.56:9092): Operation timed out 2024-04-03T01:23:58.870673762Z nc: kafka (10.101.96.56:9092): Operation timed out 2024-04-03T01:24:02.879579674Z nc: kafka (10.101.96.56:9092): Operation timed out 2024-04-03T01:24:06.880771434Z nc: kafka (10.101.96.56:9092): Operation timed out 2024-04-03T01:24:10.887437049Z nc: kafka (10.101.96.56:9092): Operation timed out 2024-04-03T01:24:14.894155016Z nc: kafka (10.101.96.56:9092): Operation timed out 2024-04-03T01:24:18.896811265Z nc: kafka (10.101.96.56:9092): Operation timed out 2024-04-03T01:25:29.114661312Z nc: kafka (10.101.96.56:9092): Operation timed out 2024-04-03T01:25:33.114841919Z nc: kafka (10.101.96.56:9092): Operation timed out 2024-04-03T01:25:37.125937915Z nc: kafka (10.101.96.56:9092): Operation timed out 2024-04-03T01:25:41.137207852Z nc: kafka (10.101.96.56:9092): Operation timed out 2024-04-03T01:25:45.151977671Z nc: kafka (10.101.96.56:9092): Operation timed out 2024-04-03T01:25:57.181903868Z kafka (10.101.96.56:9092) open 2024-04-03T01:25:57.199601125Z mongodb-k8s (10.244.0.48:27017) open 2024-04-03T01:26:43.322816810Z 04/03/2024 01:26:43 AM - kafka.conn - INFO - [IPv4 ('10.101.96.56', 9092)]>: connecting to kafka:9092 [('10.101.96.56', 9092) IPv4] 2024-04-03T01:26:43.323294528Z 04/03/2024 01:26:43 AM - kafka.conn - INFO - Probing node bootstrap-0 broker version 2024-04-03T01:26:43.325262561Z 04/03/2024 01:26:43 AM - kafka.conn - INFO - [IPv4 ('10.101.96.56', 9092)]>: Connection complete. 2024-04-03T01:26:43.431849937Z 04/03/2024 01:26:43 AM - kafka.conn - INFO - Broker version identified as 2.5.0 2024-04-03T01:26:43.431868103Z 04/03/2024 01:26:43 AM - kafka.conn - INFO - Set configuration api_version=(2, 5, 0) to skip auto check_version requests on startup 2024-04-03T01:26:43.473834795Z 04/03/2024 01:26:43 AM - kafka.conn - INFO - [IPv4 ('10.101.96.56', 9092)]>: connecting to kafka:9092 [('10.101.96.56', 9092) IPv4] 2024-04-03T01:26:43.473876719Z 04/03/2024 01:26:43 AM - kafka.conn - INFO - Probing node bootstrap-0 broker version 2024-04-03T01:26:43.478385898Z 04/03/2024 01:26:43 AM - kafka.conn - INFO - [IPv4 ('10.101.96.56', 9092)]>: Connection complete. 2024-04-03T01:26:43.613224011Z 04/03/2024 01:26:43 AM - kafka.conn - INFO - Broker version identified as 2.5.0 2024-04-03T01:26:43.613240915Z 04/03/2024 01:26:43 AM - kafka.conn - INFO - Set configuration api_version=(2, 5, 0) to skip auto check_version requests on startup 2024-04-03T01:26:43.826908925Z 04/03/2024 01:26:43 AM - kafka.conn - INFO - [IPv4 ('10.244.0.37', 9092)]>: connecting to kafka-controller-0.kafka-controller-headless.osm.svc.cluster.local:9092 [('10.244.0.37', 9092) IPv4] 2024-04-03T01:26:43.826925953Z 04/03/2024 01:26:43 AM - kafka.conn - INFO - [IPv4 ('10.244.0.37', 9092)]>: Connection complete. 2024-04-03T01:26:43.826936497Z 04/03/2024 01:26:43 AM - kafka.conn - INFO - [IPv4 ('10.101.96.56', 9092)]>: Closing connection. 2024-04-03T01:26:44.566736891Z 04/03/2024 01:26:44 AM - kafka.conn - INFO - [IPv4 ('10.244.0.43', 9092)]>: connecting to kafka-controller-1.kafka-controller-headless.osm.svc.cluster.local:9092 [('10.244.0.43', 9092) IPv4] 2024-04-03T01:26:44.566757655Z 04/03/2024 01:26:44 AM - kafka.conn - INFO - [IPv4 ('10.244.0.43', 9092)]>: Connection complete. 2024-04-03T01:26:44.673291229Z 04/03/2024 01:26:44 AM - kafka.cluster - ERROR - Topic users not found in cluster metadata 2024-04-03T01:26:44.821516030Z 04/03/2024 01:26:44 AM - kafka.producer.record_accumulator - WARNING - Produced messages to topic-partition TopicPartition(topic='users', partition=0) with base offset -1 log start offset -1 and error None. 2024-04-03T01:26:44.903304615Z 04/03/2024 01:26:44 AM - kafka.cluster - ERROR - Topic users not found in cluster metadata 2024-04-03T01:26:44.903319164Z 04/03/2024 01:26:44 AM - kafka.cluster - ERROR - Topic project not found in cluster metadata 2024-04-03T01:26:45.036827755Z 04/03/2024 01:26:45 AM - kafka.cluster - ERROR - Topic project not found in cluster metadata 2024-04-03T01:26:45.209832685Z 04/03/2024 01:26:45 AM - kafka.conn - INFO - [IPv4 ('10.244.0.30', 9092)]>: connecting to kafka-controller-2.kafka-controller-headless.osm.svc.cluster.local:9092 [('10.244.0.30', 9092) IPv4] 2024-04-03T01:26:45.209972908Z 04/03/2024 01:26:45 AM - kafka.conn - INFO - [IPv4 ('10.244.0.30', 9092)]>: Connection complete. 2024-04-03T01:26:45.291407163Z 04/03/2024 01:26:45 AM - kafka.conn - INFO - [IPv4 ('10.244.0.37', 9092)]>: Closing connection. 2024-04-03T01:26:45.291424627Z 04/03/2024 01:26:45 AM - kafka.conn - INFO - [IPv4 ('10.244.0.43', 9092)]>: Closing connection. 2024-04-03T01:26:45.291428315Z 04/03/2024 01:26:45 AM - kafka.conn - INFO - [IPv4 ('10.244.0.30', 9092)]>: Closing connection. 2024-04-03T01:26:45.291432783Z 04/03/2024 01:26:45 AM - osm_mon.cmd.mon_dashboarder - INFO - Starting MON Dashboarder... 2024-04-03T01:26:45.367423698Z 04/03/2024 01:26:45 AM - db - INFO - Connected to database osm version None 2024-04-03T01:26:45.367440974Z 04/03/2024 01:26:45 AM - osm_mon.cmd.mon_dashboarder - INFO - Starting MON kafka Consumer... 2024-04-03T01:26:45.372568810Z 04/03/2024 01:26:45 AM - osm_mon.cmd.mon_dashboarder - INFO - Starting MON Dashboarder... 2024-04-03T01:26:45.431964415Z 04/03/2024 01:26:45 AM - aiokafka.consumer.subscription_state - INFO - Updating subscribed topics to: frozenset({'project', 'users'}) 2024-04-03T01:26:45.431973307Z 04/03/2024 01:26:45 AM - aiokafka.consumer.consumer - INFO - Subscribed to topic(s): {'project', 'users'} 2024-04-03T01:26:45.483629997Z 04/03/2024 01:26:45 AM - aiokafka.consumer.group_coordinator - ERROR - Group Coordinator Request failed: [Error 15] CoordinatorNotAvailableError 2024-04-03T01:26:45.694646243Z 04/03/2024 01:26:45 AM - aiokafka.consumer.group_coordinator - ERROR - Group Coordinator Request failed: [Error 15] CoordinatorNotAvailableError 2024-04-03T01:26:45.907811787Z 04/03/2024 01:26:45 AM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 0 for group mon-consumer 2024-04-03T01:26:45.907828756Z 04/03/2024 01:26:45 AM - aiokafka.consumer.group_coordinator - INFO - Revoking previously assigned partitions set() for group mon-consumer 2024-04-03T01:26:45.907834802Z 04/03/2024 01:26:45 AM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-04-03T01:26:45.941298617Z 04/03/2024 01:26:45 AM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 0)for group mon-consumer. 2024-04-03T01:26:46.048930420Z 04/03/2024 01:26:46 AM - aiokafka.consumer.group_coordinator - ERROR - Group Coordinator Request failed: [Error 15] CoordinatorNotAvailableError 2024-04-03T01:26:46.179993910Z 04/03/2024 01:26:46 AM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 0 for group mon-consumer 2024-04-03T01:26:46.180034949Z 04/03/2024 01:26:46 AM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-04-03T01:26:46.189846498Z 04/03/2024 01:26:46 AM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 0)for group mon-consumer. 2024-04-03T01:26:46.297020382Z 04/03/2024 01:26:46 AM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 0 for group mon-consumer 2024-04-03T01:26:46.297195911Z 04/03/2024 01:26:46 AM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-04-03T01:26:46.300674046Z 04/03/2024 01:26:46 AM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 0)for group mon-consumer. 2024-04-03T01:26:46.483932341Z 04/03/2024 01:26:46 AM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 0 for group mon-consumer 2024-04-03T01:26:46.483951289Z 04/03/2024 01:26:46 AM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-04-03T01:26:46.496335071Z 04/03/2024 01:26:46 AM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 0)for group mon-consumer. 2024-04-03T01:26:46.618013697Z 04/03/2024 01:26:46 AM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 0 for group mon-consumer 2024-04-03T01:26:46.618048035Z 04/03/2024 01:26:46 AM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-04-03T01:26:46.634112777Z 04/03/2024 01:26:46 AM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 0)for group mon-consumer. 2024-04-03T01:26:46.743229561Z 04/03/2024 01:26:46 AM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 0 for group mon-consumer 2024-04-03T01:26:46.743291800Z 04/03/2024 01:26:46 AM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-04-03T01:26:46.749638964Z 04/03/2024 01:26:46 AM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 0)for group mon-consumer. 2024-04-03T01:26:46.974993966Z 04/03/2024 01:26:46 AM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 0 for group mon-consumer 2024-04-03T01:26:46.975046807Z 04/03/2024 01:26:46 AM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-04-03T01:26:46.985811209Z 04/03/2024 01:26:46 AM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 0)for group mon-consumer. 2024-04-03T01:26:47.207368269Z 04/03/2024 01:26:47 AM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 0 for group mon-consumer 2024-04-03T01:26:47.211186480Z 04/03/2024 01:26:47 AM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-04-03T01:26:47.334493272Z 04/03/2024 01:26:47 AM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 0)for group mon-consumer. 2024-04-03T01:26:47.556111472Z 04/03/2024 01:26:47 AM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 0 for group mon-consumer 2024-04-03T01:26:47.556130046Z 04/03/2024 01:26:47 AM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-04-03T01:26:47.582698932Z 04/03/2024 01:26:47 AM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 0)for group mon-consumer. 2024-04-03T01:26:47.834807787Z 04/03/2024 01:26:47 AM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 0 for group mon-consumer 2024-04-03T01:26:47.863681955Z 04/03/2024 01:26:47 AM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-04-03T01:26:47.966670337Z 04/03/2024 01:26:47 AM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 0)for group mon-consumer. 2024-04-03T01:26:48.057803001Z 04/03/2024 01:26:48 AM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 0 for group mon-consumer 2024-04-03T01:26:48.058017805Z 04/03/2024 01:26:48 AM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-04-03T01:26:48.060068744Z 04/03/2024 01:26:48 AM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 0)for group mon-consumer. 2024-04-03T01:26:48.190840026Z 04/03/2024 01:26:48 AM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 0 for group mon-consumer 2024-04-03T01:26:48.190857837Z 04/03/2024 01:26:48 AM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-04-03T01:26:48.211253200Z 04/03/2024 01:26:48 AM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 0)for group mon-consumer. 2024-04-03T01:26:48.491570982Z 04/03/2024 01:26:48 AM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 0 for group mon-consumer 2024-04-03T01:26:48.491593583Z 04/03/2024 01:26:48 AM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-04-03T01:26:48.505084192Z 04/03/2024 01:26:48 AM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 0)for group mon-consumer. 2024-04-03T01:26:48.608268163Z 04/03/2024 01:26:48 AM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 0 for group mon-consumer 2024-04-03T01:26:48.608289650Z 04/03/2024 01:26:48 AM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-04-03T01:26:48.609811422Z 04/03/2024 01:26:48 AM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 0)for group mon-consumer. 2024-04-03T01:26:48.723499543Z 04/03/2024 01:26:48 AM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 0 for group mon-consumer 2024-04-03T01:26:48.723594910Z 04/03/2024 01:26:48 AM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-04-03T01:26:48.745455559Z 04/03/2024 01:26:48 AM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 0)for group mon-consumer. 2024-04-03T01:26:48.843051500Z 04/03/2024 01:26:48 AM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 0 for group mon-consumer 2024-04-03T01:26:48.843076375Z 04/03/2024 01:26:48 AM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-04-03T01:26:48.868006730Z 04/03/2024 01:26:48 AM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 0)for group mon-consumer. 2024-04-03T01:26:48.970744444Z 04/03/2024 01:26:48 AM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 0 for group mon-consumer 2024-04-03T01:26:48.992256462Z 04/03/2024 01:26:48 AM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-04-03T01:26:48.992278949Z 04/03/2024 01:26:48 AM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 0)for group mon-consumer. 2024-04-03T01:26:49.082833746Z 04/03/2024 01:26:49 AM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 0 for group mon-consumer 2024-04-03T01:26:49.082850328Z 04/03/2024 01:26:49 AM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-04-03T01:26:49.099310551Z 04/03/2024 01:26:49 AM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 0)for group mon-consumer. 2024-04-03T01:26:49.209248698Z 04/03/2024 01:26:49 AM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 0 for group mon-consumer 2024-04-03T01:26:49.209264993Z 04/03/2024 01:26:49 AM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-04-03T01:26:49.214522009Z 04/03/2024 01:26:49 AM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 0)for group mon-consumer. 2024-04-03T01:26:49.334456182Z 04/03/2024 01:26:49 AM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 0 for group mon-consumer 2024-04-03T01:26:49.334469542Z 04/03/2024 01:26:49 AM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-04-03T01:26:49.341621401Z 04/03/2024 01:26:49 AM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 0)for group mon-consumer. 2024-04-03T01:26:49.456309432Z 04/03/2024 01:26:49 AM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 0 for group mon-consumer 2024-04-03T01:26:49.456801778Z 04/03/2024 01:26:49 AM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-04-03T01:26:49.463712311Z 04/03/2024 01:26:49 AM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 0)for group mon-consumer. 2024-04-03T01:26:49.586733885Z 04/03/2024 01:26:49 AM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 0 for group mon-consumer 2024-04-03T01:26:49.586744766Z 04/03/2024 01:26:49 AM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-04-03T01:26:49.675685543Z 04/03/2024 01:26:49 AM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 0)for group mon-consumer. 2024-04-03T01:26:49.821883546Z 04/03/2024 01:26:49 AM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 0 for group mon-consumer 2024-04-03T01:26:49.821901175Z 04/03/2024 01:26:49 AM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-04-03T01:26:49.824769637Z 04/03/2024 01:26:49 AM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 0)for group mon-consumer. 2024-04-03T01:26:49.942815065Z 04/03/2024 01:26:49 AM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 0 for group mon-consumer 2024-04-03T01:26:49.942833986Z 04/03/2024 01:26:49 AM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-04-03T01:26:49.949177413Z 04/03/2024 01:26:49 AM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 0)for group mon-consumer. 2024-04-03T01:26:50.094667432Z 04/03/2024 01:26:50 AM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 0 for group mon-consumer 2024-04-03T01:26:50.094685335Z 04/03/2024 01:26:50 AM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-04-03T01:26:50.105501872Z 04/03/2024 01:26:50 AM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 0)for group mon-consumer. 2024-04-03T01:26:50.266723925Z 04/03/2024 01:26:50 AM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 0 for group mon-consumer 2024-04-03T01:26:50.266740119Z 04/03/2024 01:26:50 AM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-04-03T01:26:50.268267526Z 04/03/2024 01:26:50 AM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 0)for group mon-consumer. 2024-04-03T01:26:50.436146540Z 04/03/2024 01:26:50 AM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 0 for group mon-consumer 2024-04-03T01:26:50.436167181Z 04/03/2024 01:26:50 AM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-04-03T01:26:50.441553193Z 04/03/2024 01:26:50 AM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 0)for group mon-consumer. 2024-04-03T01:26:50.562953708Z 04/03/2024 01:26:50 AM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 0 for group mon-consumer 2024-04-03T01:26:50.563018145Z 04/03/2024 01:26:50 AM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-04-03T01:26:50.570785907Z 04/03/2024 01:26:50 AM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 0)for group mon-consumer. 2024-04-03T01:26:50.671809205Z 04/03/2024 01:26:50 AM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 0 for group mon-consumer 2024-04-03T01:26:50.672043843Z 04/03/2024 01:26:50 AM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-04-03T01:26:50.675704088Z 04/03/2024 01:26:50 AM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 0)for group mon-consumer. 2024-04-03T01:26:50.784979043Z 04/03/2024 01:26:50 AM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 0 for group mon-consumer 2024-04-03T01:26:50.784993828Z 04/03/2024 01:26:50 AM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-04-03T01:26:50.793047119Z 04/03/2024 01:26:50 AM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 0)for group mon-consumer. 2024-04-03T01:26:50.906737082Z 04/03/2024 01:26:50 AM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 0 for group mon-consumer 2024-04-03T01:26:50.911944792Z 04/03/2024 01:26:50 AM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-04-03T01:26:50.919325719Z 04/03/2024 01:26:50 AM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 0)for group mon-consumer. 2024-04-03T01:26:51.096389205Z 04/03/2024 01:26:51 AM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 0 for group mon-consumer 2024-04-03T01:26:51.096436390Z 04/03/2024 01:26:51 AM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-04-03T01:26:51.134674242Z 04/03/2024 01:26:51 AM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 0)for group mon-consumer. 2024-04-03T01:26:51.261143555Z 04/03/2024 01:26:51 AM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 0 for group mon-consumer 2024-04-03T01:26:51.261160215Z 04/03/2024 01:26:51 AM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-04-03T01:26:54.787204066Z 04/03/2024 01:26:54 AM - aiokafka.consumer.group_coordinator - INFO - Joined group 'mon-consumer' (generation 1) with member_id aiokafka-0.8.1-a8ba8de2-a03e-41ab-85c6-33ea57991139 2024-04-03T01:26:54.787225701Z 04/03/2024 01:26:54 AM - aiokafka.consumer.group_coordinator - INFO - Elected group leader -- performing partition assignments using roundrobin 2024-04-03T01:26:55.055462577Z 04/03/2024 01:26:55 AM - aiokafka.consumer.group_coordinator - INFO - Successfully synced group mon-consumer with generation 1 2024-04-03T01:26:55.056021884Z 04/03/2024 01:26:55 AM - aiokafka.consumer.group_coordinator - INFO - Setting newly assigned partitions {TopicPartition(topic='users', partition=0), TopicPartition(topic='project', partition=0)} for group mon-consumer 2024-04-03T01:27:16.240343574Z 04/03/2024 01:27:16 AM - osm_mon.dashboarder.backends.grafana - INFO - Dashboard admin is created in Grafana 2024-04-03T01:33:47.797406875Z 04/03/2024 01:33:47 AM - osm_mon.dashboarder.backends.grafana - INFO - Datasource prom-operator-osm is created in Grafana 2024-04-03T01:33:47.797499270Z 04/03/2024 01:33:47 AM - osm_mon.dashboarder.backends.grafana - INFO - ************* response: {'_content': b'{"datasource":{"id":3,"uid":"d89f86ee-5237-45ee-9665-04b002c1f401","orgId":1,"name":"prom-operator-osm","type":"prometheus","typeLogoUrl":"public/app/plugins/datasource/prometheus/img/prometheus_logo.svg","access":"proxy","url":"http://172.21.248.125:9090","user":"","database":"","basicAuth":false,"basicAuthUser":"","withCredentials":false,"isDefault":false,"jsonData":{},"secureJsonFields":{},"version":1,"readOnly":false},"id":3,"message":"Datasource added","name":"prom-operator-osm"}', '_content_consumed': True, '_next': None, 'status_code': 200, 'headers': {'Cache-Control': 'no-store', 'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff', 'X-Frame-Options': 'deny', 'X-Xss-Protection': '1; mode=block', 'Date': 'Wed, 03 Apr 2024 01:33:47 GMT', 'Content-Length': '489'}, 'raw': , 'url': 'http://grafana:3000/api/datasources', 'encoding': 'utf-8', 'history': [], 'reason': 'OK', 'cookies': , 'elapsed': datetime.timedelta(microseconds=63339), 'request': , 'connection': } 2024-04-03T01:33:47.942248197Z 04/03/2024 01:33:47 AM - osm_mon.dashboarder.backends.grafana - INFO - Dashboard osm is created in Grafana 2024-04-03T01:40:19.545983737Z 04/03/2024 01:40:19 AM - osm_mon.dashboarder.backends.grafana - INFO - Dashboard admin is created in Grafana 2024-04-03T01:42:30.841574677Z 04/03/2024 01:42:30 AM - osm_mon.dashboarder.backends.grafana - INFO - New user basic_15_test_user created in Grafana 2024-04-03T01:42:34.049876369Z 04/03/2024 01:42:34 AM - osm_mon.dashboarder.backends.grafana - INFO - User basic_15_test_user is assigned Admin permission 2024-04-03T01:42:38.802693008Z 04/03/2024 01:42:38 AM - osm_mon.dashboarder.dashboarder - ERROR - Exception processing message: 2024-04-03T01:42:38.802732439Z Traceback (most recent call last): 2024-04-03T01:42:38.802742071Z File "/usr/lib/python3/dist-packages/osm_mon/dashboarder/dashboarder.py", line 116, in _user_msg 2024-04-03T01:42:38.802751464Z project_data = values["changes"]["add_project_role_mappings"] 2024-04-03T01:42:38.802760016Z KeyError: 'add_project_role_mappings' 2024-04-03T01:42:40.047474847Z 04/03/2024 01:42:40 AM - osm_mon.dashboarder.backends.grafana - INFO - New team basic_15_test_project created in Grafana 2024-04-03T01:42:41.683889691Z 04/03/2024 01:42:41 AM - osm_mon.dashboarder.backends.grafana - INFO - Grafana team updated {"message":"Team updated"} 2024-04-03T01:42:49.636186367Z 04/03/2024 01:42:49 AM - osm_mon.dashboarder.backends.grafana - INFO - User basic_15_test_user deleted in Grafana 2024-04-03T01:42:50.156576364Z 04/03/2024 01:42:50 AM - osm_mon.dashboarder.backends.grafana - INFO - Dashboard folder basic_15_project_test created 2024-04-03T01:42:50.325465261Z 04/03/2024 01:42:50 AM - osm_mon.dashboarder.backends.grafana - INFO - Dashboard basic_15_project_test is created in Grafana 2024-04-03T01:42:51.320882663Z 04/03/2024 01:42:51 AM - osm_mon.dashboarder.backends.grafana - INFO - Team basic_15_project_test deleted in Grafana 2024-04-03T01:43:20.605486058Z 04/03/2024 01:43:20 AM - osm_mon.dashboarder.backends.grafana - INFO - Dashboard admin is created in Grafana 2024-04-03T02:54:05.262614183Z 04/03/2024 02:54:05 AM - osm_mon.dashboarder.backends.grafana - INFO - Datasource prom-operator-k8s02 is created in Grafana 2024-04-03T02:54:05.264804708Z 04/03/2024 02:54:05 AM - osm_mon.dashboarder.backends.grafana - INFO - ************* response: {'_content': b'{"datasource":{"id":4,"uid":"bd0cb319-c18a-4fb1-adb0-257fdced9a67","orgId":1,"name":"prom-operator-k8s02","type":"prometheus","typeLogoUrl":"public/app/plugins/datasource/prometheus/img/prometheus_logo.svg","access":"proxy","url":"http://172.21.248.125:9090","user":"","database":"","basicAuth":false,"basicAuthUser":"","withCredentials":false,"isDefault":false,"jsonData":{},"secureJsonFields":{},"version":1,"readOnly":false},"id":4,"message":"Datasource added","name":"prom-operator-k8s02"}', '_content_consumed': True, '_next': None, 'status_code': 200, 'headers': {'Cache-Control': 'no-store', 'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff', 'X-Frame-Options': 'deny', 'X-Xss-Protection': '1; mode=block', 'Date': 'Wed, 03 Apr 2024 02:54:05 GMT', 'Content-Length': '493'}, 'raw': , 'url': 'http://grafana:3000/api/datasources', 'encoding': 'utf-8', 'history': [], 'reason': 'OK', 'cookies': , 'elapsed': datetime.timedelta(microseconds=52822), 'request': , 'connection': } 2024-04-03T02:54:05.467378937Z 04/03/2024 02:54:05 AM - osm_mon.dashboarder.backends.grafana - INFO - Dashboard k8s02 is created in Grafana 2024-04-03T03:06:38.068076475Z 04/03/2024 03:06:38 AM - osm_mon.dashboarder.backends.grafana - INFO - Datasource prom-operator-k8s07 is created in Grafana 2024-04-03T03:06:38.068427374Z 04/03/2024 03:06:38 AM - osm_mon.dashboarder.backends.grafana - INFO - ************* response: {'_content': b'{"datasource":{"id":5,"uid":"cade8875-c4ed-4ee8-a999-383fa6c00e1f","orgId":1,"name":"prom-operator-k8s07","type":"prometheus","typeLogoUrl":"public/app/plugins/datasource/prometheus/img/prometheus_logo.svg","access":"proxy","url":"http://172.21.248.125:9090","user":"","database":"","basicAuth":false,"basicAuthUser":"","withCredentials":false,"isDefault":false,"jsonData":{},"secureJsonFields":{},"version":1,"readOnly":false},"id":5,"message":"Datasource added","name":"prom-operator-k8s07"}', '_content_consumed': True, '_next': None, 'status_code': 200, 'headers': {'Cache-Control': 'no-store', 'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff', 'X-Frame-Options': 'deny', 'X-Xss-Protection': '1; mode=block', 'Date': 'Wed, 03 Apr 2024 03:06:38 GMT', 'Content-Length': '493'}, 'raw': , 'url': 'http://grafana:3000/api/datasources', 'encoding': 'utf-8', 'history': [], 'reason': 'OK', 'cookies': , 'elapsed': datetime.timedelta(microseconds=45479), 'request': , 'connection': } 2024-04-03T03:06:38.237453496Z 04/03/2024 03:06:38 AM - osm_mon.dashboarder.backends.grafana - INFO - Dashboard k8s07 is created in Grafana 2024-04-03T03:17:19.752068956Z 04/03/2024 03:17:19 AM - osm_mon.dashboarder.backends.grafana - INFO - New team quotas_01_proj_1 created in Grafana 2024-04-03T03:17:26.119877015Z 04/03/2024 03:17:26 AM - osm_mon.dashboarder.backends.grafana - INFO - New team quotas_01_proj_2 created in Grafana 2024-04-03T03:17:32.444735781Z 04/03/2024 03:17:32 AM - osm_mon.dashboarder.backends.grafana - INFO - New user quotas_01_user created in Grafana 2024-04-03T03:17:32.603073820Z 04/03/2024 03:17:32 AM - osm_mon.dashboarder.backends.grafana - INFO - User quotas_01_user is assigned Editor permission 2024-04-03T03:17:32.780927172Z 04/03/2024 03:17:32 AM - osm_mon.dashboarder.backends.grafana - INFO - User quotas_01_user is assigned Editor permission 2024-04-03T03:17:37.127513977Z 04/03/2024 03:17:37 AM - osm_mon.dashboarder.dashboarder - ERROR - Exception processing message: 2024-04-03T03:17:37.127532534Z Traceback (most recent call last): 2024-04-03T03:17:37.127536166Z File "/usr/lib/python3/dist-packages/osm_mon/dashboarder/dashboarder.py", line 116, in _user_msg 2024-04-03T03:17:37.127539979Z project_data = values["changes"]["add_project_role_mappings"] 2024-04-03T03:17:37.127543065Z KeyError: 'add_project_role_mappings' 2024-04-03T03:17:38.841907586Z 04/03/2024 03:17:38 AM - osm_mon.dashboarder.backends.grafana - INFO - Grafana team updated {"message":"Team updated"} 2024-04-03T03:17:40.539798545Z 04/03/2024 03:17:40 AM - osm_mon.dashboarder.backends.grafana - INFO - Dashboard folder quotas_01_proj_3 created 2024-04-03T03:17:40.778921813Z 04/03/2024 03:17:40 AM - osm_mon.dashboarder.backends.grafana - INFO - Dashboard quotas_01_proj_3 is created in Grafana 2024-04-03T03:17:40.841573868Z 04/03/2024 03:17:40 AM - osm_mon.dashboarder.backends.grafana - INFO - Dashboard folder quotas_01_proj_2 created 2024-04-03T03:17:41.044684178Z 04/03/2024 03:17:41 AM - osm_mon.dashboarder.backends.grafana - INFO - Dashboard quotas_01_proj_2 is created in Grafana 2024-04-03T03:17:48.367261158Z 04/03/2024 03:17:48 AM - osm_mon.dashboarder.dashboarder - ERROR - Exception processing message: 2024-04-03T03:17:48.367280320Z Traceback (most recent call last): 2024-04-03T03:17:48.367283814Z File "/usr/lib/python3/dist-packages/osm_mon/dashboarder/dashboarder.py", line 129, in _user_msg 2024-04-03T03:17:48.367286835Z project_new_name = values["changes"]["name"] 2024-04-03T03:17:48.367309490Z KeyError: 'name' 2024-04-03T03:18:18.568106622Z 04/03/2024 03:18:18 AM - osm_mon.dashboarder.backends.grafana - INFO - Team quotas_01_proj_2 deleted in Grafana 2024-04-03T03:18:20.334182129Z 04/03/2024 03:18:20 AM - osm_mon.dashboarder.backends.grafana - INFO - User quotas_01_user deleted in Grafana 2024-04-03T03:18:22.031119189Z 04/03/2024 03:18:22 AM - osm_mon.dashboarder.backends.grafana - INFO - Team quotas_01_proj_3 deleted in Grafana 2024-04-03T03:18:41.444092453Z 04/03/2024 03:18:41 AM - osm_mon.dashboarder.backends.grafana - INFO - Dashboard admin is created in Grafana 2024-04-03T03:27:13.469777224Z 04/03/2024 03:27:13 AM - osm_mon.dashboarder.backends.grafana - INFO - Dashboard admin is created in Grafana 2024-04-03T03:46:47.928795305Z 04/03/2024 03:46:47 AM - osm_mon.dashboarder.backends.grafana - INFO - Dashboard admin is created in Grafana 2024-04-03T04:13:54.083226136Z 04/03/2024 04:13:54 AM - osm_mon.dashboarder.backends.grafana - INFO - Dashboard admin is created in Grafana