all the time Prometheus Make complaints about the most. HA And clustering solutions .

Think before , We use top matching machines , But there will be disks , network IO, Query timeout and other issues , Mainly because the amount of data collected is too large , Beyond the range of single machine .

In the past , We have , department , All that can be split into different ones Prometheus It's gone , But even so , There are still some problems :

*
Some businesses cannot be split , Data volume is still very large .

*
What data collection and query do is completely independent , But they often compete with each other to occupy resources .

Today we are going to explore new ways to solve this problem , I.e remote_read realization Prometheus Cluster scheme of data read write separation .

remote_read brief introduction

as everyone knows , from Prometheus 1.8 start , Added a name remote_read Configuration of , Details are as follows :

*
# The URL of the endpoint to query from.

*
url: <string>

*


*
# Timeout for requests to the remote read endpoint.

*
[ remote_timeout: <duration> | default = 30s ]

*


*
# Sets the `Authorization` header on every remote read request with the

*
# configured username and password.

*
basic_auth:

*
 [ username: <string> ]

*
 [ password: <string> ]

*


*
# Sets the `Authorization` header on every remote read request with

*
# the configured bearer token. It is mutually exclusive with
`bearer_token_file`.

*
[ bearer_token: <string> ]

*


*
# Sets the `Authorization` header on every remote read request with the bearer
token

*
# read from the configured file. It is mutually exclusive with `bearer_token`.

*
[ bearer_token_file: /path/to/bearer/token/file ]

*


*
# Configures the remote read request's TLS settings.

*
tls_config:

*
 [ <tls_config> ]

*


*
# Optional proxy URL.

*
[ proxy_url: <string> ]

remote_read Parameters are used to read data remotely , use http agreement .

Of course, being read as a data source needs to support its remote storage reader Interface of , This is designed to decouple in a new generation of storage architecture , It's easy to do :

*
Prometheus The reading and writing of can be in different Prometheus Server conduct , It's a Prometheus Read other Prometheus Data for .

*
Prometheus Can be read and written in more storage engines , That is, you can use InfluxDB Store data as a database .

Deployment architecture



Architecture description :

*
Server A'  express Server A  Image of , Same functions and data (B,C Homology ).

*
Data flow adopts pull mode .

Not hard to find , In architecture Prometheus There are two main categories , For data collection ( for example B) and For data query ( for example A).

Where data query Prometheus The data is read from the nodes collected to the data , Please note that , It only does real-time queries and memory operations , No data storage .

Through such an architecture , We can easily separate the whole monitoring data collection and query , It's also easier to achieve high availability .

How to configure

The deployment architecture we discussed above , How to configure and implement this deployment scheme ?

Now I'll show you how to use the  remote_read realization Prometheus Data reading process between .

Experimental environment :

*
Prometheus edition : prometheus-1.8.2.darwin-amd64

*
NodeExporter edition :node_exporter-0.12.0.darwin-amd64

Experiment content :

Run three locally Prometheus Server, They run in 9090, 9091, 9092 port . among 9091 and 9092 Mainly used for collection
node_exporter data ,9090 For reading 9091, 9092 Data collected .

The configuration information is as follows :

*
# Data read 9090:

*
remote_read:

*
 - url: 'http://localhost:9091/api/v1/read'

*
   remote_timeout: 8s

*
 - url: 'http://localhost:9092/api/v1/read'

*
   remote_timeout: 8s

*


*
# data collection 9091

*
- job_name: 'node'

*
 static_configs:

*
   - targets: ["localhost:9100"]

*


*
# data collection 9092

*
- job_name: 'node'

*
 static_configs:

*
   - targets: ["localhost:9100"]

Complete configuration , After successfully starting these three instances , We can verify them in their own interfaces .



9091 Data collected



9092 Data collected








9090 Read data

Support aggregate operation query :



okay , So far, we have successfully passed remote_read Configuration implementation from different Prometheus Server
Read data , It means that the deployment architecture we discussed earlier is completely feasible .

Write here , The focus of this article is basically completed , Here is to adjust your deployment architecture ?

Event promotion :

GopherChina 2018 Early bird ticket ends at 1 month 31 day ( Last three days ), click   Read the original   Get to know first ~