|
| 1 | +--- |
| 2 | +layout: blog |
| 3 | +title: "Kubernetes 1.27: Query Node Logs Using The Kubelet API" |
| 4 | +date: 2023-04-21 |
| 5 | +slug: node-log-query-alpha |
| 6 | +--- |
| 7 | + |
| 8 | +**Author:** Aravindh Puthiyaparambil (Red Hat) |
| 9 | + |
| 10 | +Kubernetes 1.27 introduced a new feature called _Node log query_ that allows |
| 11 | +viewing logs of services running on the node. |
| 12 | + |
| 13 | +## What problem does it solve? |
| 14 | +Cluster administrators face issues when debugging malfunctioning services |
| 15 | +running on the node. They usually have to SSH or RDP into the node to view the |
| 16 | +logs of the service to debug the issue. The _Node log query_ feature helps with |
| 17 | +this scenario by allowing the cluster administrator to view the logs using |
| 18 | +_kubectl_. This is especially useful with Windows nodes where you run into the |
| 19 | +issue of the node going to the ready state but containers not coming up due to |
| 20 | +CNI misconfigurations and other issues that are not easily identifiable by |
| 21 | +looking at the Pod status. |
| 22 | + |
| 23 | +## How does it work? |
| 24 | + |
| 25 | +The kubelet already has a _/var/log/_ viewer that is accessible via the node |
| 26 | +proxy endpoint. The feature supplements this endpoint with a shim that shells |
| 27 | +out to `journalctl`, on Linux nodes, and the `Get-WinEvent` cmdlet on Windows |
| 28 | +nodes. It then uses the existing filters provided by the commands to allow |
| 29 | +filtering the logs. The kubelet also uses heuristics to retrieve the logs. |
| 30 | +If the user is not aware if a given system services logs to a file or to the |
| 31 | +native system logger, the heuristics first checks the native operating system |
| 32 | +logger and if that is not available it attempts to retrieve the first logs |
| 33 | +from `/var/log/<servicename>` or `/var/log/<servicename>.log` or |
| 34 | +`/var/log/<servicename>/<servicename>.log`. |
| 35 | + |
| 36 | +On Linux we assume that service logs are available via journald, and that |
| 37 | +`journalctl` is installed. On Windows we assume that service logs are available |
| 38 | +in the application log provider. Also note that fetching node logs is only |
| 39 | +available if you are authorized to do so (in RBAC, that's **get** and |
| 40 | +**create** access to `nodes/proxy`). The privileges that you need to fetch node |
| 41 | +logs also allow elevation-of-privilege attacks, so be careful about how you |
| 42 | +manage them. |
| 43 | + |
| 44 | +## How do I use it? |
| 45 | + |
| 46 | +To use the feature, ensure that the `NodeLogQuery` |
| 47 | +[feature gate](/docs/reference/command-line-tools-reference/feature-gates/) is |
| 48 | +enabled for that node, and that the kubelet configuration options |
| 49 | +`enableSystemLogHandler` and `enableSystemLogQuery` are both set to true. You can |
| 50 | +then query the logs from all your nodes or just a subset. Here is an example to |
| 51 | +retrieve the kubelet service logs from a node: |
| 52 | +```shell |
| 53 | +# Fetch kubelet logs from a node named node-1.example |
| 54 | +kubectl get --raw "/api/v1/nodes/node-1.example/proxy/logs/?query=kubelet" |
| 55 | +``` |
| 56 | + |
| 57 | +You can further filter the query to narrow down the results: |
| 58 | +```shell |
| 59 | +# Fetch kubelet logs from a node named node-1.example that have the word "error" |
| 60 | +kubectl get --raw "/api/v1/nodes/node-1.example/proxy/logs/?query=kubelet&pattern=error" |
| 61 | +``` |
| 62 | + |
| 63 | +You can also fetch files from `/var/log/` on a Linux node: |
| 64 | +```shell |
| 65 | +kubectl get --raw "/api/v1/nodes/<insert-node-name-here>/proxy/logs/?query=/<insert-log-file-name-here>" |
| 66 | +``` |
| 67 | + |
| 68 | +You can read the |
| 69 | +[documentation](/docs/concepts/cluster-administration/system-logs/#log-query) |
| 70 | +for all the available options. |
| 71 | + |
| 72 | +## How do I help? |
| 73 | + |
| 74 | +Please use the feature and provide feedback by opening GitHub issues or |
| 75 | +reaching out to us on the |
| 76 | +[#sig-windows](https://kubernetes.slack.com/archives/C0SJ4AFB7) channel on the |
| 77 | +Kubernetes Slack or the SIG Windows |
| 78 | +[mailing list](https://groups.google.com/g/kubernetes-sig-windows). |
0 commit comments