kubectl get <object> <space> throws exception

[bobhenkel]

When I run commands like kubectl get pods and press space from within kube-shell 'm getting the below error. On a vanilla bash shell I can execute kubectl get just fine.

kube-shell> kubectl get ingress Exception in thread Thread-74:
Traceback (most recent call last):
  File "/usr/local/Cellar/python/2.7.13/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 801, in __bootstrap_inner
    self.run()
  File "/usr/local/Cellar/python/2.7.13/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 754, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/local/lib/python2.7/site-packages/prompt_toolkit/interface.py", line 860, in run
    completions = list(buffer.completer.get_completions(document, complete_event))
  File "/usr/local/lib/python2.7/site-packages/kubeshell/completer.py", line 246, in get_completions
    resources = self.get_resources(arg, namespace)
  File "/usr/local/lib/python2.7/site-packages/kubeshell/completer.py", line 321, in get_resources
    ret = extensionsV1Beta1.list_ingress_for_all_namespaces(watch=False)
  File "/usr/local/lib/python2.7/site-packages/kubernetes/client/apis/extensions_v1beta1_api.py", line 3152, in list_ingress_for_all_namesps
    (data) = self.list_ingress_for_all_namespaces_with_http_info(**kwargs)
  File "/usr/local/lib/python2.7/site-packages/kubernetes/client/apis/extensions_v1beta1_api.py", line 3245, in list_ingress_for_all_namespo
    collection_formats=collection_formats)
  File "/usr/local/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 329, in call_api
    _return_http_data_only, collection_formats, _preload_content, _request_timeout)
  File "/usr/local/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 153, in __call_api
    _request_timeout=_request_timeout)
  File "/usr/local/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 361, in request
    headers=headers)
  File "/usr/local/lib/python2.7/site-packages/kubernetes/client/rest.py", line 240, in GET
    query_params=query_params)
  File "/usr/local/lib/python2.7/site-packages/kubernetes/client/rest.py", line 231, in request
    raise ApiException(http_resp=r)
ApiException: (401)
Reason: Unauthorized
HTTP response headers: HTTPHeaderDict({'Date': 'Wed, 05 Jul 2017 14:05:34 GMT', 'Content-Length': '13', 'Content-Type': 'text/plain; charse)
HTTP response body: Unauthorized

[murali-reddy]

Kubectl uses go client to talk to api server. Kube-shell uses client-python to talk to api server for resource auto completion. there seems to some problem python api client accessing api server.

Could you please try below snippet from your python interactive shell and see if this goes through?

from kubernetes import client, config
config.load_kube_config()
v1 = client.CoreV1Api()
v1.list_pod_for_all_namespaces(watch=False)

[bobhenkel]

Still throws an error.

> python                                                                                                                            13:10:54
Python 2.7.13 (default, Dec 18 2016, 07:03:39)
[GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.42.1)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from kubernetes import client, config
>>> config.load_kube_config()
>>> v1 = client.CoreV1Api()
>>> v1.list_pod_for_all_namespaces(watch=False)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 13650, in list_pod_for_all_namespaces
    (data) = self.list_pod_for_all_namespaces_with_http_info(**kwargs)
  File "/usr/local/lib/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 13743, in list_pod_for_all_namespaces_with_http_info
    collection_formats=collection_formats)
  File "/usr/local/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 329, in call_api
    _return_http_data_only, collection_formats, _preload_content, _request_timeout)
  File "/usr/local/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 153, in __call_api
    _request_timeout=_request_timeout)
  File "/usr/local/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 361, in request
    headers=headers)
  File "/usr/local/lib/python2.7/site-packages/kubernetes/client/rest.py", line 240, in GET
    query_params=query_params)
  File "/usr/local/lib/python2.7/site-packages/kubernetes/client/rest.py", line 231, in request
    raise ApiException(http_resp=r)
kubernetes.client.rest.ApiException: (401)
Reason: Unauthorized
HTTP response headers: HTTPHeaderDict({'Date': 'Wed, 05 Jul 2017 18:18:49 GMT', 'Content-Length': '13', 'Content-Type': 'text/plain; charset=utf-8', 'X-Content-Type-Options': 'nosniff'})
HTTP response body: Unauthorized


>>>

[mbohlool]

Can you share details of your cluster? (Is it on gke or gce or minikube).

[bobhenkel]

It's a CoreOS Tectonic cluster on AWS. RBAC is enabled.

[bobhenkel]

Also I'm just using static users at the moment, if that's of any value to the error.

https://coreos.com/tectonic/docs/latest/admin/user-management.html

[clmssz]

Same problem here with GKE

[vogxn]

The authentication problems are described in a bug within client-python here. The workaround provided is to use service accounts.

I'll dig through the client-python code a little today.

Before I make a change to the code, @clmssz are you using GKE with service accounts?

In the short term I propose to handle kube-apiserver endpoint errors like this without disabling the shell's utility.

[clmssz]

@vogxn kubectl uses your gcloud compute credentials I think

[vogxn]

I've redirected the errors to an error log. Kubeshell will not dump stacktraces like this now. The pull request is here

You can test this by installing from the branch

$ pip install -e git://github.com/vogxn/kube-shell.git@26-handle-client-python-errors#egg=kube-shell

[vogxn]

This is now merged and will be part of the next release along with other bugfixes.

[vogxn]

A new release is pushed to pypi with version 0.0.21. An upgrade should fix this issue

[clmssz]

Hi @vogxn , no more stacktrace thanks (but no completion, guess it'a what you wanted in the short term right ?)

[mbohlool]

Kubernetes python client 3.0.0b1 should have a fix for this. Can you guys give it a try and let me know if it fixed your problem?

[pilsy]

Any progress on this? the auto completion is what made we want to use it