2024-03-23T22:29:00.434410127Z Connected to MongoDB!
2024-03-23T22:29:00.434469792Z Refreshing prometheus config file for first time
2024-03-23T22:29:30.464873686Z Error in first configuration attempt!
2024-03-23T22:29:30.464904769Z No replica set members match selector "<function readable_server_selector at 0x7f05384524c0>", Timeout: 30s, Topology Description: <TopologyDescription id: 65ff57ac4b08de4eb4394738, topology_type: ReplicaSetNoPrimary, servers: [<ServerDescription ('mongodb-k8s', 27017) server_type: RSGhost, rtt: 0.0022739652799909893>]>
2024-03-23T22:30:00.481886219Z No replica set members match selector "<function readable_server_selector at 0x7f05384524c0>", Timeout: 30s, Topology Description: <TopologyDescription id: 65ff57ac4b08de4eb4394738, topology_type: ReplicaSetNoPrimary, servers: [<ServerDescription ('mongodb-k8s', 27017) server_type: RSGhost, rtt: 0.0016893156873471601>]>
2024-03-23T22:30:00.481961524Z Detected failure while listening to prometheus jobs collection, retrying...
2024-03-23T22:30:35.528872682Z No replica set members match selector "<function readable_server_selector at 0x7f05384524c0>", Timeout: 30s, Topology Description: <TopologyDescription id: 65ff57ac4b08de4eb4394738, topology_type: ReplicaSetNoPrimary, servers: [<ServerDescription ('mongodb-k8s', 27017) server_type: RSGhost, rtt: 0.0012283095839011142>]>
2024-03-23T22:30:35.529079673Z Detected failure while listening to prometheus jobs collection, retrying...
2024-03-23T22:31:10.550413619Z No replica set members match selector "<function readable_server_selector at 0x7f05384524c0>", Timeout: 30s, Topology Description: <TopologyDescription id: 65ff57ac4b08de4eb4394738, topology_type: ReplicaSetNoPrimary, servers: [<ServerDescription ('mongodb-k8s', 27017) server_type: RSGhost, rtt: 0.001039697645572475>]>
2024-03-23T22:31:10.550451228Z Detected failure while listening to prometheus jobs collection, retrying...
2024-03-23T22:31:45.593061197Z No replica set members match selector "<function readable_server_selector at 0x7f05384524c0>", Timeout: 30s, Topology Description: <TopologyDescription id: 65ff57ac4b08de4eb4394738, topology_type: ReplicaSetNoPrimary, servers: [<ServerDescription ('mongodb-k8s', 27017) server_type: RSGhost, rtt: 0.0009329625305356537>]>
2024-03-23T22:31:45.593233300Z Detected failure while listening to prometheus jobs collection, retrying...
2024-03-23T22:32:20.616232160Z No replica set members match selector "<function readable_server_selector at 0x7f05384524c0>", Timeout: 30s, Topology Description: <TopologyDescription id: 65ff57ac4b08de4eb4394738, topology_type: ReplicaSetNoPrimary, servers: [<ServerDescription ('mongodb-k8s', 27017) server_type: RSGhost, rtt: 0.0008150456316169545>]>
2024-03-23T22:32:20.616274446Z Detected failure while listening to prometheus jobs collection, retrying...
2024-03-23T22:32:34.370268979Z Listening to changes in prometheus jobs collection
2024-03-23T22:32:35.698635109Z Change detected, updating prometheus config
2024-03-23T22:32:35.700047019Z Jobs detected : 1
2024-03-23T22:32:35.704621881Z Writing new config file to /etc/prometheus/prometheus.yml
2024-03-23T22:32:35.706745852Z New config written, updating prometheus
2024-03-23T22:32:40.731771700Z Prometheus config update successful
2024-03-23T22:32:40.733773153Z 
2024-03-23T22:28:33.133330336Z level=info ts=2024-03-23T22:28:33.132Z caller=main.go:389 msg="No time or size retention was set so using the default time retention" duration=15d
2024-03-23T22:28:33.133378954Z level=info ts=2024-03-23T22:28:33.133Z caller=main.go:443 msg="Starting Prometheus" version="(version=2.28.1, branch=HEAD, revision=b0944590a1c9a6b35dc5a696869f75f422b107a1)"
2024-03-23T22:28:33.133387679Z level=info ts=2024-03-23T22:28:33.133Z caller=main.go:448 build_context="(go=go1.16.5, user=root@2915dd495090, date=20210701-15:20:10)"
2024-03-23T22:28:33.133392654Z level=info ts=2024-03-23T22:28:33.133Z caller=main.go:449 host_details="(Linux 5.4.0-107-generic #121-Ubuntu SMP Thu Mar 24 16:04:27 UTC 2022 x86_64 prometheus-0 (none))"
2024-03-23T22:28:33.133513900Z level=info ts=2024-03-23T22:28:33.133Z caller=main.go:450 fd_limits="(soft=1048576, hard=1048576)"
2024-03-23T22:28:33.133520827Z level=info ts=2024-03-23T22:28:33.133Z caller=main.go:451 vm_limits="(soft=unlimited, hard=unlimited)"
2024-03-23T22:28:33.135998328Z level=info ts=2024-03-23T22:28:33.135Z caller=web.go:541 component=web msg="Start listening for connections" address=0.0.0.0:9090
2024-03-23T22:28:33.137762934Z level=info ts=2024-03-23T22:28:33.137Z caller=main.go:824 msg="Starting TSDB ..."
2024-03-23T22:28:33.147752003Z level=info ts=2024-03-23T22:28:33.147Z caller=tls_config.go:191 component=web msg="TLS is disabled." http2=false
2024-03-23T22:28:33.150926815Z level=info ts=2024-03-23T22:28:33.150Z caller=head.go:780 component=tsdb msg="Replaying on-disk memory mappable chunks if any"
2024-03-23T22:28:33.150969755Z level=info ts=2024-03-23T22:28:33.150Z caller=head.go:794 component=tsdb msg="On-disk memory mappable chunks replay completed" duration=9.888µs
2024-03-23T22:28:33.150978398Z level=info ts=2024-03-23T22:28:33.150Z caller=head.go:800 component=tsdb msg="Replaying WAL, this may take a while"
2024-03-23T22:28:33.151897898Z level=info ts=2024-03-23T22:28:33.151Z caller=head.go:854 component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0
2024-03-23T22:28:33.151979065Z level=info ts=2024-03-23T22:28:33.151Z caller=head.go:860 component=tsdb msg="WAL replay completed" checkpoint_replay_duration=85.684µs wal_replay_duration=768.653µs total_replay_duration=1.021897ms
2024-03-23T22:28:33.155431990Z level=info ts=2024-03-23T22:28:33.155Z caller=main.go:851 fs_type=EXT4_SUPER_MAGIC
2024-03-23T22:28:33.155449805Z level=info ts=2024-03-23T22:28:33.155Z caller=main.go:854 msg="TSDB started"
2024-03-23T22:28:33.155559227Z level=info ts=2024-03-23T22:28:33.155Z caller=main.go:981 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml
2024-03-23T22:28:33.157258218Z level=info ts=2024-03-23T22:28:33.157Z caller=main.go:1012 msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=1.621262ms remote_storage=12.068µs web_handler=1.056µs query_engine=7.134µs scrape=526.413µs scrape_sd=65.249µs notify=48.645µs notify_sd=25.022µs rules=10.259µs
2024-03-23T22:28:33.157282138Z level=info ts=2024-03-23T22:28:33.157Z caller=main.go:796 msg="Server is ready to receive web requests."
2024-03-23T22:32:35.712404215Z level=info ts=2024-03-23T22:32:35.712Z caller=main.go:981 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml
2024-03-23T22:32:35.713254119Z level=info ts=2024-03-23T22:32:35.713Z caller=main.go:1012 msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=944.081µs remote_storage=6.977µs web_handler=463ns query_engine=7.415µs scrape=129.302µs scrape_sd=36.232µs notify=42.856µs notify_sd=10.253µs rules=6.724µs