[Kubernetes] Install Promtail(v2.9.4) Using Helm Chart

Promtail #

Loki가 로그를 저장하는 역할을 하면 Promtail은 application에서 로그를 전달하는 agent의 역할을 한다.
Promtail 이외에도 Bit, Fluentd, LogStash 등을 사용할 수 있다.
kubernetes는 node 별로 /var/log/pods 아래에 모든 Pod의 로그가 기록된다.
daemonset으로 설정하고 node별로 로그를 수집하도록 처리를 하면 된다.
설치 방식은 sidecar, daemonset 방식이 있는데 daemonset 방식을 추천한다고 한다.
- daemonset : 각 Node마다 promtail Pod가 실행되어 해당 Node 장비에서 실행 중인 Pod의 로그를 추적
- sidecar : 각 Pod에 container로 추가되어 실행, 해당 Pod 내부에서 로그 파일을 읽어서 Loki로 전송
Pod마다 agent 형태로 설정하는 것보다 daemonset을 하나 띄워 해당 node의 Pod들을 찾아 로그를 수집하는 것이 훨씬 편한 것 같다.
Prometheus가 저장소와 polling 역할을 같이 담당하는 반면 Promtail은 저장소의 역할은 하지 않고 로그를 찾아 저장소로 push 하는 역할을 한다.
하지만 설정 방식이나 문법은 크게 차이가 없다.

💡 Helm 설치 및 설명 참고

Install the Promtail Helm charts #

1helm repo add grafana https://grafana.github.io/helm-charts
2helm repo update
3helm install promtail grafana/promtail --namespace [NAMESPACE NAME]

Promtail 설치 참고

https://grafana.com/docs/loki/latest/clients/promtail/installation/

Customize Default Configuration #

values.yaml 수정
💡 최상위 values.yaml을 수정하면 하위 폴더 values.yaml을 override 한다.
- Chart
  - https://github.com/grafana/helm-charts/tree/main/charts/promtail
- Release file (.tgz)
  - https://github.com/grafana/helm-charts/releases

Setting Volumes #

 1# -- Default volumes that are mounted into pods. In most cases, these should not be changed.
 2# Use `extraVolumes`/`extraVolumeMounts` for additional custom volumes.
 3# @default -- See `values.yaml`
 4defaultVolumes:
 5  - name: run
 6    hostPath:
 7      path: /run/promtail
 8  - name: containers
 9    hostPath:
10      path: /var/lib/docker/containers
11  - name: pods
12    hostPath:
13      path: /var/log/pods
14  - name: syslogs
15    hostPath:
16      path: /var/log
17
18# -- Default volume mounts. Corresponds to `volumes`.
19# @default -- See `values.yaml`
20defaultVolumeMounts:
21  - name: run
22    mountPath: /run/promtail
23  - name: containers
24    mountPath: /var/lib/docker/containers
25    readOnly: true
26  - name: pods
27    mountPath: /var/log/pods
28    readOnly: true
29  - name: syslogs
30    mountPath: /var/log
31    readOnly: true

Setting Config #

  1config:
  2  # -- Enable Promtail config from Helm chart
  3  # Set `configmap.enabled: true` and this to `false` to manage your own Promtail config
  4  # See default config in `values.yaml`
  5  enabled: true
  6  # -- The log level of the Promtail server
  7  # Must be reference in `config.file` to configure `server.log_level`
  8  # See default config in `values.yaml`
  9  logLevel: info
 10  # -- The log format of the Promtail server
 11  # Must be reference in `config.file` to configure `server.log_format`
 12  # Valid formats: `logfmt, json`
 13  # See default config in `values.yaml`
 14  logFormat: logfmt
 15  # -- The port of the Promtail server
 16  # Must be reference in `config.file` to configure `server.http_listen_port`
 17  # See default config in `values.yaml`
 18  serverPort: 3101
 19  # -- The config of clients of the Promtail server
 20  # Must be reference in `config.file` to configure `clients`
 21  # @default -- See `values.yaml`
 22  clients:
 23    - url: http://loki-gateway/loki/api/v1/push
 24    - basic_auth:
 25        password: ${LOKI_BASIC_AUTH_PW}
 26        username: ${LOKI_BASIC_AUTH_USER}
 27      external_labels:
 28        cluster: ${CLUSTER}
 29      url: ${LOKI_URL}
 30  # -- Configures where Promtail will save it's positions file, to resume reading after restarts.
 31  # Must be referenced in `config.file` to configure `positions`
 32
 33...✂...
 34
 35  snippets:
 36    pipelineStages:
 37      - cri: {}
 38    common:
 39      - action: replace
 40        source_labels:
 41          - __meta_kubernetes_pod_node_name
 42        target_label: node_name
 43      - action: replace
 44        source_labels:
 45          - __meta_kubernetes_namespace
 46        target_label: namespace
 47      - action: replace
 48        replacement: $1
 49        separator: /
 50        source_labels:
 51          - namespace
 52          - app
 53        target_label: job
 54      - action: replace
 55        source_labels:
 56          - __meta_kubernetes_pod_name
 57        target_label: pod
 58      - action: replace
 59        source_labels:
 60          - __meta_kubernetes_pod_container_name
 61        target_label: container
 62      - action: replace
 63        replacement: /var/log/pods/*$1/*.log
 64        separator: /
 65        source_labels:
 66          - __meta_kubernetes_pod_uid
 67          - __meta_kubernetes_pod_container_name
 68        target_label: __path__
 69      - action: replace
 70        replacement: /var/log/pods/*$1/*.log
 71        regex: true/(.*)
 72        separator: /
 73        source_labels:
 74          - __meta_kubernetes_pod_annotationpresent_kubernetes_io_config_hash
 75          - __meta_kubernetes_pod_annotation_kubernetes_io_config_hash
 76          - __meta_kubernetes_pod_container_name
 77        target_label: __path__
 78
 79...✂...
 80
 81    scrapeConfigs: |
 82      - job_name: syslog
 83        static_configs:
 84        - targets:
 85            - localhost
 86          labels:
 87            job: syslog
 88            __path__: /var/log/syslog
 89        pipeline_stages:
 90          - regex:
 91              expression: '^(?P<time>[^ ]* {1,2}[^ ]* [^ ]*) (?P<hostname>[^ ]*) (?P<daemon>[^ :\[]*)(?:\[(?P<pid>[0-9]+)\])?(?:[^\:]*\:)? *(?P<message>.*)$'
 92          - labels:
 93              time:
 94              hostname:
 95              daemon:
 96              pid:
 97              message:
 98          - match:
 99              selector: '{daemon=~"kubelet|kernel"}'
100              stages:
101              - regex:
102                  expression: '^(?P<time>[^ ]* {1,2}[^ ]* [^ ]*) (?P<hostname>[^ ]*) (?P<daemon>[^ :\[]*)(?:\[(?P<pid>[0-9]+)\])?(?:[^\:]*\:)? *(?P<message>.*)$'
103              - labels:
104                  time:
105                  hostname:
106                  daemon:
107                  pid:
108                  message:
109              - timestamp:
110                  source: time
111                  format: %b %d %H:%M:%S
112      - job_name: custom-log
113        static_configs:
114        - targets:
115            - localhost
116          labels:
117            job: custom-log
118            __path__: /appdata/applog
119        pipeline_stages:
120          - regex:
121              expression: '^(?P<log_type>[^ ]*) '
122          - labels:
123              log_type:
124          - match:
125              selector: '{log_type="type1"}'
126              stages:
127              - regex:
128                  expression: '^(?P<log_type>[^ ]*) (?P<log_level>[^ ]*) ~'
129              - template:
130                  source: log_level
131                  template: 'warning'
132              - labels:
133                  log_type:
134          - match:
135              selector: '{log_type="type2"}'
136              stages:
137              - regex:
138                  expression: '^(?P<log_type>[^ ]*) (?P<log_level>[^ ]*) (?P<message>.*) ~'
139              - template:
140                  source: log_type
141                  template: 'API'
142              - labels:
143                  log_type:
144                  log_level:
145                  message:
146              - match:
147                  selector: '{log_level="I"}'
148                  stages:
149                  - regex:
150                      expression: '^(?P<log_type>[^ ]*) (?P<log_level>[^ ]*) (?P<message>.*) ~'
151                  - template:
152                      source: log_level
153                      template: 'INFO'
154                  - labels:
155                      log_type:
156                      log_level:
157                      message:
158              - match:
159                  selector: '{log_level=~"W|E"}'
160                  stages:
161                  - regex:
162                      expression: '^(?P<log_type>[^ ]*) (?P<log_level>[^ ]*) (?P<message>.*) ~'
163                  - template:
164                      source: log_level
165                      template: '{{ if eq .Value "W" }}{{ Replace .Value "W" "WARNING" -1 }}{{ else if eq .Value "E" }}{{ Replace .Value "E" "ERROR" -1 }}{{ else }}{{ .Value }}{{ end }}'
166                  - labels:
167                      log_type:
168                      log_level:
169                      message:
170
171      # See also https://github.com/grafana/loki/blob/master/production/ksonnet/promtail/scrape_config.libsonnet for reference
172      - job_name: kubernetes-pods
173        pipeline_stages:
174          {{- toYaml .Values.config.snippets.pipelineStages | nindent 4 }}
175        kubernetes_sd_configs:
176          - role: pod
177            namespaces:
178              names:
179              - kube-system
180              - ...✂...
181              - ...✂...
182        relabel_configs:
183          - source_labels:
184              - __meta_kubernetes_pod_controller_name
185            regex: ([0-9a-z-.]+?)(-[0-9a-f]{8,10})?
186            action: replace
187            target_label: __tmp_controller_name
188          - source_labels:
189              - __meta_kubernetes_pod_label_app_kubernetes_io_name
190              - __meta_kubernetes_pod_label_app
191              - __tmp_controller_name
192              - __meta_kubernetes_pod_name
193            regex: ^;*([^;]+)(;.*)?$
194            action: replace
195            target_label: app
196          - source_labels:
197              - __meta_kubernetes_pod_label_app_kubernetes_io_instance
198              - __meta_kubernetes_pod_label_instance
199            regex: ^;*([^;]+)(;.*)?$
200            action: replace
201            target_label: instance
202          - source_labels:
203              - __meta_kubernetes_pod_label_app_kubernetes_io_component
204              - __meta_kubernetes_pod_label_component
205            regex: ^;*([^;]+)(;.*)?$
206            action: replace
207            target_label: component
208          {{- if .Values.config.snippets.addScrapeJobLabel }}
209          - replacement: kubernetes-pods
210            target_label: scrape_job
211          {{- end }}
212          {{- toYaml .Values.config.snippets.common | nindent 4 }}
213          {{- with .Values.config.snippets.extraRelabelConfigs }}
214          {{- toYaml . | nindent 4 }}
215          {{- end }}
216
217  # -- Config file contents for Promtail.
218  # Must be configured as string.
219  # It is templated so it can be assembled from reusable snippets in order to avoid redundancy.
220  # @default -- See `values.yaml`
221  file: |
222    server:
223      log_level: {{ .Values.config.logLevel }}
224      log_format: {{ .Values.config.logFormat }}
225      http_listen_port: {{ .Values.config.serverPort }}
226      {{- with .Values.httpPathPrefix }}
227      http_path_prefix: {{ . }}
228      {{- end }}
229      {{- tpl .Values.config.snippets.extraServerConfigs . | nindent 2 }}
230
231    clients:
232      {{- tpl (toYaml .Values.config.clients) . | nindent 2 }}
233
234    positions:
235      {{- tpl (toYaml .Values.config.positions) . | nindent 2 }}
236
237    scrape_configs:
238      {{- tpl .Values.config.snippets.scrapeConfigs . | nindent 2 }}
239      {{- tpl .Values.config.snippets.extraScrapeConfigs . | nindent 2 }}
240
241    limits_config:
242      {{- tpl .Values.config.snippets.extraLimitsConfig . | nindent 2 }}
243
244    tracing:
245      enabled: {{ .Values.config.enableTracing }}

syslog regex #

^(?P<time>[^ ]* {1,2}[^ ]* [^ ]*) (?P<hostname>[^ ]*) (?P<daemon>[^ :\[]*)(?:\[(?P<pid>[0-9]+)\])?(?:[^\:]*\:)? *(?P<message>.*)$

Install Customize Default Configuration #

1helm install [RELEASE NAME] [Chart.yaml 경로] -f [YAML 파일 또는 URL에 값 지정 (여러 개를 지정가능)] -n [NAMESPACE NAME]

1helm install promtail grafana/promtail -f override-values.yaml -n [NAMESPACE NAME]

Uninstall the Chart #

1helm uninstall [RELEASE NAME] -n [NAMESPACE NAME]