2024-02-02T01:27:49.231220888Z Connected to MongoDB!
2024-02-02T01:27:49.231263757Z Refreshing prometheus config file for first time
2024-02-02T01:28:19.576061244Z Error in first configuration attempt!
2024-02-02T01:28:19.576085464Z No replica set members match selector "<function readable_server_selector at 0x7fa0501ef550>", Timeout: 30s, Topology Description: <TopologyDescription id: 65bc4515f31c3dbb38ee2a20, topology_type: ReplicaSetNoPrimary, servers: [<ServerDescription ('mongodb-k8s', 27017) server_type: RSGhost, rtt: 0.0021337470000162284>]>
2024-02-02T01:28:49.614462630Z No replica set members match selector "<function readable_server_selector at 0x7fa0501ef550>", Timeout: 30s, Topology Description: <TopologyDescription id: 65bc4515f31c3dbb38ee2a20, topology_type: ReplicaSetNoPrimary, servers: [<ServerDescription ('mongodb-k8s', 27017) server_type: RSGhost, rtt: 0.0025298359840244303>]>
2024-02-02T01:28:49.614527020Z Detected failure while listening to prometheus jobs collection, retrying...
2024-02-02T01:29:24.630085263Z No replica set members match selector "<function readable_server_selector at 0x7fa0501ef550>", Timeout: 30s, Topology Description: <TopologyDescription id: 65bc4515f31c3dbb38ee2a20, topology_type: ReplicaSetNoPrimary, servers: [<ServerDescription ('mongodb-k8s', 27017) server_type: RSGhost, rtt: 0.001655093998257953>]>
2024-02-02T01:29:24.630156757Z Detected failure while listening to prometheus jobs collection, retrying...
2024-02-02T01:29:59.661936116Z No replica set members match selector "<function readable_server_selector at 0x7fa0501ef550>", Timeout: 30s, Topology Description: <TopologyDescription id: 65bc4515f31c3dbb38ee2a20, topology_type: ReplicaSetNoPrimary, servers: [<ServerDescription ('mongodb-k8s', 27017) server_type: RSGhost, rtt: 0.001341974215103808>]>
2024-02-02T01:29:59.661996084Z Detected failure while listening to prometheus jobs collection, retrying...
2024-02-02T01:30:34.693819246Z No replica set members match selector "<function readable_server_selector at 0x7fa0501ef550>", Timeout: 30s, Topology Description: <TopologyDescription id: 65bc4515f31c3dbb38ee2a20, topology_type: ReplicaSetNoPrimary, servers: [<ServerDescription ('mongodb-k8s', 27017) server_type: RSGhost, rtt: 0.0011408680781308162>]>
2024-02-02T01:30:34.694253104Z Detected failure while listening to prometheus jobs collection, retrying...
2024-02-02T01:31:09.727771820Z No replica set members match selector "<function readable_server_selector at 0x7fa0501ef550>", Timeout: 30s, Topology Description: <TopologyDescription id: 65bc4515f31c3dbb38ee2a20, topology_type: ReplicaSetNoPrimary, servers: [<ServerDescription ('mongodb-k8s', 27017) server_type: RSGhost, rtt: 0.0010757630240029212>]>
2024-02-02T01:31:09.727866106Z Detected failure while listening to prometheus jobs collection, retrying...
2024-02-02T01:31:44.758286721Z No replica set members match selector "<function readable_server_selector at 0x7fa0501ef550>", Timeout: 30s, Topology Description: <TopologyDescription id: 65bc4515f31c3dbb38ee2a20, topology_type: ReplicaSetNoPrimary, servers: [<ServerDescription ('mongodb-k8s', 27017) server_type: RSGhost, rtt: 0.0009914444418186136>]>
2024-02-02T01:31:44.758351139Z Detected failure while listening to prometheus jobs collection, retrying...
2024-02-02T01:32:17.629747004Z Listening to changes in prometheus jobs collection
2024-02-02T01:32:21.640443319Z Change detected, updating prometheus config
2024-02-02T01:32:21.644797684Z Jobs detected : 1
2024-02-02T01:32:21.655539699Z Writing new config file to /etc/prometheus/prometheus.yml
2024-02-02T01:32:21.657605366Z New config written, updating prometheus
2024-02-02T01:32:26.683059930Z Prometheus config update successful
2024-02-02T01:32:26.685056782Z 
2024-02-02T04:52:38.706850347Z Change detected, updating prometheus config
2024-02-02T04:52:38.709259851Z Jobs detected : 2
2024-02-02T04:52:38.713451781Z Writing new config file to /etc/prometheus/prometheus.yml
2024-02-02T04:52:38.717652111Z New config written, updating prometheus
2024-02-02T04:52:43.750389732Z Prometheus config update successful
2024-02-02T04:52:43.754094129Z 
2024-02-02T04:53:48.079545837Z Change detected, updating prometheus config
2024-02-02T04:53:48.081135793Z Jobs detected : 1
2024-02-02T04:53:48.088248054Z Writing new config file to /etc/prometheus/prometheus.yml
2024-02-02T04:53:48.091780269Z New config written, updating prometheus
2024-02-02T04:53:53.124118517Z Prometheus config update successful
2024-02-02T04:53:53.125775011Z 
2024-02-02T01:27:17.086603307Z level=info ts=2024-02-02T01:27:17.086Z caller=main.go:389 msg="No time or size retention was set so using the default time retention" duration=15d
2024-02-02T01:27:17.086654313Z level=info ts=2024-02-02T01:27:17.086Z caller=main.go:443 msg="Starting Prometheus" version="(version=2.28.1, branch=HEAD, revision=b0944590a1c9a6b35dc5a696869f75f422b107a1)"
2024-02-02T01:27:17.086658921Z level=info ts=2024-02-02T01:27:17.086Z caller=main.go:448 build_context="(go=go1.16.5, user=root@2915dd495090, date=20210701-15:20:10)"
2024-02-02T01:27:17.086662432Z level=info ts=2024-02-02T01:27:17.086Z caller=main.go:449 host_details="(Linux 5.4.0-107-generic #121-Ubuntu SMP Thu Mar 24 16:04:27 UTC 2022 x86_64 prometheus-0 (none))"
2024-02-02T01:27:17.086665646Z level=info ts=2024-02-02T01:27:17.086Z caller=main.go:450 fd_limits="(soft=1048576, hard=1048576)"
2024-02-02T01:27:17.086668625Z level=info ts=2024-02-02T01:27:17.086Z caller=main.go:451 vm_limits="(soft=unlimited, hard=unlimited)"
2024-02-02T01:27:17.088505542Z level=info ts=2024-02-02T01:27:17.088Z caller=web.go:541 component=web msg="Start listening for connections" address=0.0.0.0:9090
2024-02-02T01:27:17.089683322Z level=info ts=2024-02-02T01:27:17.089Z caller=main.go:824 msg="Starting TSDB ..."
2024-02-02T01:27:17.096282811Z level=info ts=2024-02-02T01:27:17.096Z caller=tls_config.go:191 component=web msg="TLS is disabled." http2=false
2024-02-02T01:27:17.097756071Z level=info ts=2024-02-02T01:27:17.097Z caller=head.go:780 component=tsdb msg="Replaying on-disk memory mappable chunks if any"
2024-02-02T01:27:17.097769712Z level=info ts=2024-02-02T01:27:17.097Z caller=head.go:794 component=tsdb msg="On-disk memory mappable chunks replay completed" duration=5.261µs
2024-02-02T01:27:17.099304318Z level=info ts=2024-02-02T01:27:17.099Z caller=head.go:800 component=tsdb msg="Replaying WAL, this may take a while"
2024-02-02T01:27:17.099622769Z level=info ts=2024-02-02T01:27:17.099Z caller=head.go:854 component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0
2024-02-02T01:27:17.099655021Z level=info ts=2024-02-02T01:27:17.099Z caller=head.go:860 component=tsdb msg="WAL replay completed" checkpoint_replay_duration=106.689µs wal_replay_duration=311.056µs total_replay_duration=1.897159ms
2024-02-02T01:27:17.101196706Z level=info ts=2024-02-02T01:27:17.101Z caller=main.go:851 fs_type=EXT4_SUPER_MAGIC
2024-02-02T01:27:17.101210013Z level=info ts=2024-02-02T01:27:17.101Z caller=main.go:854 msg="TSDB started"
2024-02-02T01:27:17.102016219Z level=info ts=2024-02-02T01:27:17.101Z caller=main.go:981 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml
2024-02-02T01:27:17.106046853Z level=info ts=2024-02-02T01:27:17.105Z caller=main.go:1012 msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=3.992974ms remote_storage=6.017µs web_handler=656ns query_engine=4.077µs scrape=3.280112ms scrape_sd=41.326µs notify=51.256µs notify_sd=52.087µs rules=4.409µs
2024-02-02T01:27:17.106084999Z level=info ts=2024-02-02T01:27:17.105Z caller=main.go:796 msg="Server is ready to receive web requests."
2024-02-02T01:32:21.663848393Z level=info ts=2024-02-02T01:32:21.663Z caller=main.go:981 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml
2024-02-02T01:32:21.665765957Z level=info ts=2024-02-02T01:32:21.665Z caller=main.go:1012 msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=1.984215ms remote_storage=4.847µs web_handler=3.998µs query_engine=6.701µs scrape=142.421µs scrape_sd=36.922µs notify=32.13µs notify_sd=10.696µs rules=22.56µs
2024-02-02T04:27:39.176711128Z level=info ts=2024-02-02T04:27:39.176Z caller=compact.go:518 component=tsdb msg="write block" mint=1706837244123 maxt=1706839200000 ulid=01HNM14ZR1D9JDBH246S3R0YSV duration=38.369799ms
2024-02-02T04:27:39.178488223Z level=info ts=2024-02-02T04:27:39.178Z caller=head.go:967 component=tsdb msg="Head GC completed" duration=1.209838ms
2024-02-02T04:52:38.723264461Z level=info ts=2024-02-02T04:52:38.722Z caller=main.go:981 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml
2024-02-02T04:52:38.727052798Z level=info ts=2024-02-02T04:52:38.726Z caller=main.go:1012 msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=4.170128ms remote_storage=14.399µs web_handler=14.019µs query_engine=29.976µs scrape=397.278µs scrape_sd=197.479µs notify=67.758µs notify_sd=46.167µs rules=14.614µs
2024-02-02T04:53:48.101040172Z level=info ts=2024-02-02T04:53:48.100Z caller=main.go:981 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml
2024-02-02T04:53:48.101392129Z level=info ts=2024-02-02T04:53:48.101Z caller=main.go:1012 msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=527.97µs remote_storage=1.947µs web_handler=671ns query_engine=1.071µs scrape=195.442µs scrape_sd=27.202µs notify=13.62µs notify_sd=10.975µs rules=1.247µs
2024-02-02T05:00:24.200037847Z level=info ts=2024-02-02T05:00:24.199Z caller=compact.go:518 component=tsdb msg="write block" mint=1706839209123 maxt=1706846400000 ulid=01HNM30YP5K88GRTPWF8WTNPVT duration=65.95824ms
2024-02-02T05:00:24.202008907Z level=info ts=2024-02-02T05:00:24.201Z caller=head.go:967 component=tsdb msg="Head GC completed" duration=1.573841ms