[Kubernetes] Install Promtail(v2.9.4) Using Helm Chart
Promtail #
Loki가 로그를 저장하는 역할을 하면 Promtail은 application에서 로그를 전달하는 agent의 역할을 한다.
Promtail 이외에도 Bit, Fluentd, LogStash 등을 사용할 수 있다.
kubernetes는 node 별로
/var/log/pods아래에 모든 Pod의 로그가 기록된다.daemonset으로 설정하고 node별로 로그를 수집하도록 처리를 하면 된다.
설치 방식은 sidecar, daemonset 방식이 있는데 daemonset 방식을 추천한다고 한다.
- daemonset : 각 Node마다 promtail Pod가 실행되어 해당 Node 장비에서 실행 중인 Pod의 로그를 추적
- sidecar : 각 Pod에 container로 추가되어 실행, 해당 Pod 내부에서 로그 파일을 읽어서 Loki로 전송
Pod마다 agent 형태로 설정하는 것보다 daemonset을 하나 띄워 해당 node의 Pod들을 찾아 로그를 수집하는 것이 훨씬 편한 것 같다.
Prometheus가 저장소와 polling 역할을 같이 담당하는 반면 Promtail은 저장소의 역할은 하지 않고 로그를 찾아 저장소로 push 하는 역할을 한다.
하지만 설정 방식이나 문법은 크게 차이가 없다.
Helm 설치 및 설명 참고 {: .prompt-info }
Install the Promtail Helm charts #
1helm repo add grafana https://grafana.github.io/helm-charts
2helm repo update
3helm install promtail grafana/promtail --namespace [NAMESPACE NAME]Promtail 설치 참고
- https://grafana.com/docs/loki/latest/clients/promtail/installation/ {: .prompt-info }
Customize Default Configuration #
values.yaml 수정
최상위 values.yaml을 수정하면 하위 폴더 values.yaml을 override 한다. {: .prompt-info }
Setting Volumes #
1# -- Default volumes that are mounted into pods. In most cases, these should not be changed.
2# Use `extraVolumes`/`extraVolumeMounts` for additional custom volumes.
3# @default -- See `values.yaml`
4defaultVolumes:
5 - name: run
6 hostPath:
7 path: /run/promtail
8 - name: containers
9 hostPath:
10 path: /var/lib/docker/containers
11 - name: pods
12 hostPath:
13 path: /var/log/pods
14 - name: syslogs
15 hostPath:
16 path: /var/log
17
18# -- Default volume mounts. Corresponds to `volumes`.
19# @default -- See `values.yaml`
20defaultVolumeMounts:
21 - name: run
22 mountPath: /run/promtail
23 - name: containers
24 mountPath: /var/lib/docker/containers
25 readOnly: true
26 - name: pods
27 mountPath: /var/log/pods
28 readOnly: true
29 - name: syslogs
30 mountPath: /var/log
31 readOnly: trueSetting Config #
1config:
2 # -- Enable Promtail config from Helm chart
3 # Set `configmap.enabled: true` and this to `false` to manage your own Promtail config
4 # See default config in `values.yaml`
5 enabled: true
6 # -- The log level of the Promtail server
7 # Must be reference in `config.file` to configure `server.log_level`
8 # See default config in `values.yaml`
9 logLevel: info
10 # -- The log format of the Promtail server
11 # Must be reference in `config.file` to configure `server.log_format`
12 # Valid formats: `logfmt, json`
13 # See default config in `values.yaml`
14 logFormat: logfmt
15 # -- The port of the Promtail server
16 # Must be reference in `config.file` to configure `server.http_listen_port`
17 # See default config in `values.yaml`
18 serverPort: 3101
19 # -- The config of clients of the Promtail server
20 # Must be reference in `config.file` to configure `clients`
21 # @default -- See `values.yaml`
22 clients:
23 - url: http://loki-gateway/loki/api/v1/push
24 - basic_auth:
25 password: ${LOKI_BASIC_AUTH_PW}
26 username: ${LOKI_BASIC_AUTH_USER}
27 external_labels:
28 cluster: ${CLUSTER}
29 url: ${LOKI_URL}
30 # -- Configures where Promtail will save it's positions file, to resume reading after restarts.
31 # Must be referenced in `config.file` to configure `positions`
32
33...✂...
34
35 snippets:
36 pipelineStages:
37 - cri: {}
38 common:
39 - action: replace
40 source_labels:
41 - __meta_kubernetes_pod_node_name
42 target_label: node_name
43 - action: replace
44 source_labels:
45 - __meta_kubernetes_namespace
46 target_label: namespace
47 - action: replace
48 replacement: $1
49 separator: /
50 source_labels:
51 - namespace
52 - app
53 target_label: job
54 - action: replace
55 source_labels:
56 - __meta_kubernetes_pod_name
57 target_label: pod
58 - action: replace
59 source_labels:
60 - __meta_kubernetes_pod_container_name
61 target_label: container
62 - action: replace
63 replacement: /var/log/pods/*$1/*.log
64 separator: /
65 source_labels:
66 - __meta_kubernetes_pod_uid
67 - __meta_kubernetes_pod_container_name
68 target_label: __path__
69 - action: replace
70 replacement: /var/log/pods/*$1/*.log
71 regex: true/(.*)
72 separator: /
73 source_labels:
74 - __meta_kubernetes_pod_annotationpresent_kubernetes_io_config_hash
75 - __meta_kubernetes_pod_annotation_kubernetes_io_config_hash
76 - __meta_kubernetes_pod_container_name
77 target_label: __path__
78
79...✂...
80
81 scrapeConfigs: |
82 - job_name: syslog
83 static_configs:
84 - targets:
85 - localhost
86 labels:
87 job: syslog
88 __path__: /var/log/syslog
89 pipeline_stages:
90 - regex:
91 expression: '^(?P<time>[^ ]* {1,2}[^ ]* [^ ]*) (?P<hostname>[^ ]*) (?P<daemon>[^ :\[]*)(?:\[(?P<pid>[0-9]+)\])?(?:[^\:]*\:)? *(?P<message>.*)$'
92 - labels:
93 time:
94 hostname:
95 daemon:
96 pid:
97 message:
98 - match:
99 selector: '{daemon=~"kubelet|kernel"}'
100 stages:
101 - regex:
102 expression: '^(?P<time>[^ ]* {1,2}[^ ]* [^ ]*) (?P<hostname>[^ ]*) (?P<daemon>[^ :\[]*)(?:\[(?P<pid>[0-9]+)\])?(?:[^\:]*\:)? *(?P<message>.*)$'
103 - labels:
104 time:
105 hostname:
106 daemon:
107 pid:
108 message:
109 - timestamp:
110 source: time
111 format: %b %d %H:%M:%S
112 - job_name: custom-log
113 static_configs:
114 - targets:
115 - localhost
116 labels:
117 job: custom-log
118 __path__: /appdata/applog
119 pipeline_stages:
120 - regex:
121 expression: '^(?P<log_type>[^ ]*) '
122 - labels:
123 log_type:
124 - match:
125 selector: '{log_type="type1"}'
126 stages:
127 - regex:
128 expression: '^(?P<log_type>[^ ]*) (?P<log_level>[^ ]*) ~'
129 - template:
130 source: log_level
131 template: 'warning'
132 - labels:
133 log_type:
134 - match:
135 selector: '{log_type="type2"}'
136 stages:
137 - regex:
138 expression: '^(?P<log_type>[^ ]*) (?P<log_level>[^ ]*) (?P<message>.*) ~'
139 - template:
140 source: log_type
141 template: 'API'
142 - labels:
143 log_type:
144 log_level:
145 message:
146 - match:
147 selector: '{log_level="I"}'
148 stages:
149 - regex:
150 expression: '^(?P<log_type>[^ ]*) (?P<log_level>[^ ]*) (?P<message>.*) ~'
151 - template:
152 source: log_level
153 template: 'INFO'
154 - labels:
155 log_type:
156 log_level:
157 message:
158 - match:
159 selector: '{log_level=~"W|E"}'
160 stages:
161 - regex:
162 expression: '^(?P<log_type>[^ ]*) (?P<log_level>[^ ]*) (?P<message>.*) ~'
163 - template:
164 source: log_level
165 template: '{{ if eq .Value "W" }}{{ Replace .Value "W" "WARNING" -1 }}{{ else if eq .Value "E" }}{{ Replace .Value "E" "ERROR" -1 }}{{ else }}{{ .Value }}{{ end }}'
166 - labels:
167 log_type:
168 log_level:
169 message:
170
171 # See also https://github.com/grafana/loki/blob/master/production/ksonnet/promtail/scrape_config.libsonnet for reference
172 - job_name: kubernetes-pods
173 pipeline_stages:
174 {{- toYaml .Values.config.snippets.pipelineStages | nindent 4 }}
175 kubernetes_sd_configs:
176 - role: pod
177 namespaces:
178 names:
179 - kube-system
180 - ...✂...
181 - ...✂...
182 relabel_configs:
183 - source_labels:
184 - __meta_kubernetes_pod_controller_name
185 regex: ([0-9a-z-.]+?)(-[0-9a-f]{8,10})?
186 action: replace
187 target_label: __tmp_controller_name
188 - source_labels:
189 - __meta_kubernetes_pod_label_app_kubernetes_io_name
190 - __meta_kubernetes_pod_label_app
191 - __tmp_controller_name
192 - __meta_kubernetes_pod_name
193 regex: ^;*([^;]+)(;.*)?$
194 action: replace
195 target_label: app
196 - source_labels:
197 - __meta_kubernetes_pod_label_app_kubernetes_io_instance
198 - __meta_kubernetes_pod_label_instance
199 regex: ^;*([^;]+)(;.*)?$
200 action: replace
201 target_label: instance
202 - source_labels:
203 - __meta_kubernetes_pod_label_app_kubernetes_io_component
204 - __meta_kubernetes_pod_label_component
205 regex: ^;*([^;]+)(;.*)?$
206 action: replace
207 target_label: component
208 {{- if .Values.config.snippets.addScrapeJobLabel }}
209 - replacement: kubernetes-pods
210 target_label: scrape_job
211 {{- end }}
212 {{- toYaml .Values.config.snippets.common | nindent 4 }}
213 {{- with .Values.config.snippets.extraRelabelConfigs }}
214 {{- toYaml . | nindent 4 }}
215 {{- end }}
216
217 # -- Config file contents for Promtail.
218 # Must be configured as string.
219 # It is templated so it can be assembled from reusable snippets in order to avoid redundancy.
220 # @default -- See `values.yaml`
221 file: |
222 server:
223 log_level: {{ .Values.config.logLevel }}
224 log_format: {{ .Values.config.logFormat }}
225 http_listen_port: {{ .Values.config.serverPort }}
226 {{- with .Values.httpPathPrefix }}
227 http_path_prefix: {{ . }}
228 {{- end }}
229 {{- tpl .Values.config.snippets.extraServerConfigs . | nindent 2 }}
230
231 clients:
232 {{- tpl (toYaml .Values.config.clients) . | nindent 2 }}
233
234 positions:
235 {{- tpl (toYaml .Values.config.positions) . | nindent 2 }}
236
237 scrape_configs:
238 {{- tpl .Values.config.snippets.scrapeConfigs . | nindent 2 }}
239 {{- tpl .Values.config.snippets.extraScrapeConfigs . | nindent 2 }}
240
241 limits_config:
242 {{- tpl .Values.config.snippets.extraLimitsConfig . | nindent 2 }}
243
244 tracing:
245 enabled: {{ .Values.config.enableTracing }}syslog regex #
^(?P<time>[^ ]* {1,2}[^ ]* [^ ]*) (?P<hostname>[^ ]*) (?P<daemon>[^ :\[]*)(?:\[(?P<pid>[0-9]+)\])?(?:[^\:]*\:)? *(?P<message>.*)$Install Customize Default Configuration #
1helm install [RELEASE NAME] [Chart.yaml 경로] -f [YAML 파일 또는 URL에 값 지정 (여러 개를 지정가능)] -n [NAMESPACE NAME]1helm install promtail grafana/promtail -f override-values.yaml -n [NAMESPACE NAME]Uninstall the Chart #
1helm uninstall [RELEASE NAME] -n [NAMESPACE NAME]Advertisement