2022-09-21T16:39:12.263543897Z level=info ts=2022-09-21T16:39:12.263263562Z caller=main.go:238 msg="Starting Prometheus" version="(version=2.4.3, branch=HEAD, revision=167a4b4e73a8eca8df648d2d2043e21bdb9a7449)"
2022-09-21T16:39:12.263634396Z level=info ts=2022-09-21T16:39:12.263417742Z caller=main.go:239 build_context="(go=go1.11.1, user=root@1e42b46043e9, date=20181004-08:42:02)"
2022-09-21T16:39:12.263766969Z level=info ts=2022-09-21T16:39:12.263617896Z caller=main.go:240 host_details="(Linux 5.4.0-107-generic #121-Ubuntu SMP Thu Mar 24 16:04:27 UTC 2022 x86_64 prometheus-0 (none))"
2022-09-21T16:39:12.264008747Z level=info ts=2022-09-21T16:39:12.263840791Z caller=main.go:241 fd_limits="(soft=1048576, hard=1048576)"
2022-09-21T16:39:12.264201474Z level=info ts=2022-09-21T16:39:12.264033434Z caller=main.go:242 vm_limits="(soft=unlimited, hard=unlimited)"
2022-09-21T16:39:12.266138906Z level=info ts=2022-09-21T16:39:12.266048266Z caller=web.go:397 component=web msg="Start listening for connections" address=0.0.0.0:9090
2022-09-21T16:39:12.267677285Z level=info ts=2022-09-21T16:39:12.26589695Z caller=main.go:554 msg="Starting TSDB ..."
2022-09-21T16:39:12.280278079Z level=info ts=2022-09-21T16:39:12.280148151Z caller=main.go:564 msg="TSDB started"
2022-09-21T16:39:12.280468674Z level=info ts=2022-09-21T16:39:12.280413144Z caller=main.go:624 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml
2022-09-21T16:39:12.281740787Z level=info ts=2022-09-21T16:39:12.281673769Z caller=main.go:650 msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml
2022-09-21T16:39:12.281841390Z level=info ts=2022-09-21T16:39:12.281806699Z caller=main.go:523 msg="Server is ready to receive web requests."
2022-09-21T16:43:12.895003629Z level=info ts=2022-09-21T16:43:12.894759876Z caller=main.go:624 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml
2022-09-21T16:43:12.896372974Z level=info ts=2022-09-21T16:43:12.896273447Z caller=main.go:650 msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml
2022-09-21T16:39:57.433441308Z Connected to MongoDB!
2022-09-21T16:39:57.433523018Z Refreshing prometheus config file for first time
2022-09-21T16:40:27.881402273Z Error in first configuration attempt!
2022-09-21T16:40:27.881444259Z No replica set members match selector "<function readable_server_selector at 0x7f4cdcf54670>", Timeout: 30s, Topology Description: <TopologyDescription id: 632b3e5d84681002fb4bad64, topology_type: ReplicaSetNoPrimary, servers: [<ServerDescription ('mongodb-k8s', 27017) server_type: RSGhost, rtt: 0.002624815999979546>]>
2022-09-21T16:40:57.912157725Z No replica set members match selector "<function readable_server_selector at 0x7f4cdcf54670>", Timeout: 30s, Topology Description: <TopologyDescription id: 632b3e5d84681002fb4bad64, topology_type: ReplicaSetNoPrimary, servers: [<ServerDescription ('mongodb-k8s', 27017) server_type: RSGhost, rtt: 0.0029999235519881044>]>
2022-09-21T16:40:57.912265016Z Detected failure while listening to prometheus jobs collection, retrying...
2022-09-21T16:41:32.950149748Z No replica set members match selector "<function readable_server_selector at 0x7f4cdcf54670>", Timeout: 30s, Topology Description: <TopologyDescription id: 632b3e5d84681002fb4bad64, topology_type: ReplicaSetNoPrimary, servers: [<ServerDescription ('mongodb-k8s', 27017) server_type: RSGhost, rtt: 0.0021969076986262737>]>
2022-09-21T16:41:32.950281450Z Detected failure while listening to prometheus jobs collection, retrying...
2022-09-21T16:42:07.980811553Z No replica set members match selector "<function readable_server_selector at 0x7f4cdcf54670>", Timeout: 30s, Topology Description: <TopologyDescription id: 632b3e5d84681002fb4bad64, topology_type: ReplicaSetNoPrimary, servers: [<ServerDescription ('mongodb-k8s', 27017) server_type: RSGhost, rtt: 0.005668491718169995>]>
2022-09-21T16:42:07.980874337Z Detected failure while listening to prometheus jobs collection, retrying...
2022-09-21T16:42:43.014049422Z No replica set members match selector "<function readable_server_selector at 0x7f4cdcf54670>", Timeout: 30s, Topology Description: <TopologyDescription id: 632b3e5d84681002fb4bad64, topology_type: ReplicaSetNoPrimary, servers: [<ServerDescription ('mongodb-k8s', 27017) server_type: RSGhost, rtt: 0.0034715708717157146>]>
2022-09-21T16:42:43.014114207Z Detected failure while listening to prometheus jobs collection, retrying...
2022-09-21T16:43:11.558863126Z Listening to changes in prometheus jobs collection
2022-09-21T16:43:12.870851344Z Change detected, updating prometheus config
2022-09-21T16:43:12.877958727Z Jobs detected : 1
2022-09-21T16:43:12.882438959Z Writing new config file to /etc/prometheus/prometheus.yml
2022-09-21T16:43:12.884880236Z New config written, updating prometheus
2022-09-21T16:43:17.924971145Z Prometheus config update successful
2022-09-21T16:43:17.928358447Z