Skip to content

Commit a8c33bd

Browse files
aravindhpDonatoHorn
authored andcommitted
1 parent fc1203a commit a8c33bd

File tree

1 file changed

+78
-0
lines changed

1 file changed

+78
-0
lines changed
Lines changed: 78 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,78 @@
1+
---
2+
layout: blog
3+
title: "Kubernetes 1.27: Query Node Logs Using The Kubelet API"
4+
date: 2023-04-21
5+
slug: node-log-query-alpha
6+
---
7+
8+
**Author:** Aravindh Puthiyaparambil (Red Hat)
9+
10+
Kubernetes 1.27 introduced a new feature called _Node log query_ that allows
11+
viewing logs of services running on the node.
12+
13+
## What problem does it solve?
14+
Cluster administrators face issues when debugging malfunctioning services
15+
running on the node. They usually have to SSH or RDP into the node to view the
16+
logs of the service to debug the issue. The _Node log query_ feature helps with
17+
this scenario by allowing the cluster administrator to view the logs using
18+
_kubectl_. This is especially useful with Windows nodes where you run into the
19+
issue of the node going to the ready state but containers not coming up due to
20+
CNI misconfigurations and other issues that are not easily identifiable by
21+
looking at the Pod status.
22+
23+
## How does it work?
24+
25+
The kubelet already has a _/var/log/_ viewer that is accessible via the node
26+
proxy endpoint. The feature supplements this endpoint with a shim that shells
27+
out to `journalctl`, on Linux nodes, and the `Get-WinEvent` cmdlet on Windows
28+
nodes. It then uses the existing filters provided by the commands to allow
29+
filtering the logs. The kubelet also uses heuristics to retrieve the logs.
30+
If the user is not aware if a given system services logs to a file or to the
31+
native system logger, the heuristics first checks the native operating system
32+
logger and if that is not available it attempts to retrieve the first logs
33+
from `/var/log/<servicename>` or `/var/log/<servicename>.log` or
34+
`/var/log/<servicename>/<servicename>.log`.
35+
36+
On Linux we assume that service logs are available via journald, and that
37+
`journalctl` is installed. On Windows we assume that service logs are available
38+
in the application log provider. Also note that fetching node logs is only
39+
available if you are authorized to do so (in RBAC, that's **get** and
40+
**create** access to `nodes/proxy`). The privileges that you need to fetch node
41+
logs also allow elevation-of-privilege attacks, so be careful about how you
42+
manage them.
43+
44+
## How do I use it?
45+
46+
To use the feature, ensure that the `NodeLogQuery`
47+
[feature gate](/docs/reference/command-line-tools-reference/feature-gates/) is
48+
enabled for that node, and that the kubelet configuration options
49+
`enableSystemLogHandler` and `enableSystemLogQuery` are both set to true. You can
50+
then query the logs from all your nodes or just a subset. Here is an example to
51+
retrieve the kubelet service logs from a node:
52+
```shell
53+
# Fetch kubelet logs from a node named node-1.example
54+
kubectl get --raw "/api/v1/nodes/node-1.example/proxy/logs/?query=kubelet"
55+
```
56+
57+
You can further filter the query to narrow down the results:
58+
```shell
59+
# Fetch kubelet logs from a node named node-1.example that have the word "error"
60+
kubectl get --raw "/api/v1/nodes/node-1.example/proxy/logs/?query=kubelet&pattern=error"
61+
```
62+
63+
You can also fetch files from `/var/log/` on a Linux node:
64+
```shell
65+
kubectl get --raw "/api/v1/nodes/<insert-node-name-here>/proxy/logs/?query=/<insert-log-file-name-here>"
66+
```
67+
68+
You can read the
69+
[documentation](/docs/concepts/cluster-administration/system-logs/#log-query)
70+
for all the available options.
71+
72+
## How do I help?
73+
74+
Please use the feature and provide feedback by opening GitHub issues or
75+
reaching out to us on the
76+
[#sig-windows](https://kubernetes.slack.com/archives/C0SJ4AFB7) channel on the
77+
Kubernetes Slack or the SIG Windows
78+
[mailing list](https://groups.google.com/g/kubernetes-sig-windows).

0 commit comments

Comments
 (0)