How to proxy Kubernetes services via the kube-api server by example

Applications or microservices are typically accessed through ClusterIP, NodePort or Loadbalancer resources when running in Kubernetes, but you can also access them via the kube-api proxy when correctly authenticated through the control-plane.

Ever wondered what these URLs are in this picture? Well read on.

Step by step, we're going to explain and create working examples on how to achieve this.

$ kubectl cluster-info
Kubernetes master is running at https://10.11.2.247:6443
KubeDNS is running at https://10.11.2.247:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy

1. Create a SA (service account)

$ kubectl create sa demo

2. Create a clusterrolebinding

$ kubectl create clusterrolebinding sa-demo --clusterrole=cluster-admin --serviceaccount=default:demo

3. Get the SA token

$ kubectl describe sa demo
Name: demo
Namespace: default
Labels: <none>
Annotations: <none>
Image pull secrets: <none>
Mountable secrets: demo-token-4nznm
Tokens: demo-token-4nznm
Events: <none>

4. Dump the token

$ kubectl describe secret demo-token-4nznm
Name: demo-token-4nznm
Namespace: default
Labels: <none>
Annotations: kubernetes.io/service-account.name: demo
 kubernetes.io/service-account.uid: 75b79864–3ead-4e9a-8213-ac0f4384b9c2Type: kubernetes.io/service-account-tokenData
====
ca.crt: 1025 bytes
namespace: 7 bytes
token: eyJhbGciOiJSUzI1NiIsImtpZCI6…x4wfIyBqOzhV8aDk0H-6r0eOqsRqujg2nkg

5. Export the token

$ export TOKEN=eyJhbGciOiJSUzI1NiIsImtpZCI6…x4wfIyBqOzhV8aDk0H-6r0eOqsRqujg2nkg

6. Build a quick app

$ kubectl create ns wwwnginx

$ kubectl apply -n wwwnginx -f https://raw.githubusercontent.com/xxradar/kuberneteslearning/master/nginx-deployment.yaml

$ kubectl apply -n wwwnginx -f https://raw.githubusercontent.com/xxradar/kuberneteslearning/master/nginx-expose-clusterip.yaml

$ kubectl get svc -n wwwnginx
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
my-nginx-clusterip ClusterIP 10.103.148.83 <none> 80/TCP 0s

7. Build the proxy URL and connect via service account

$ export MASTER=kubernetes.docker.internal

$ curl -kv -H "Authorization: Bearer $TOKEN" https://$MASTER:6443/api/v1/namespaces/wwwnginx/services/http:my-nginx-clusterip:80/proxy/
* Trying 10.11.2.247…
* TCP_NODELAY set
* Connected to 10.11.2.247 (10.11.2.247) port 6443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/certs/ca-certificates.crt
 CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS Unknown, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Unknown (8):
* TLSv1.3 (IN), TLS Unknown, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Request CERT (13):
* TLSv1.3 (IN), TLS Unknown, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS Unknown, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS Unknown, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Client hello (1):
* TLSv1.3 (OUT), TLS Unknown, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Certificate (11):
* TLSv1.3 (OUT), TLS Unknown, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
* subject: CN=kube-apiserver
* start date: May 19 14:12:32 2020 GMT
* expire date: May 19 14:12:33 2021 GMT
* issuer: CN=kubernetes
* SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* TLSv1.3 (OUT), TLS Unknown, Unknown (23):
* TLSv1.3 (OUT), TLS Unknown, Unknown (23):
* TLSv1.3 (OUT), TLS Unknown, Unknown (23):
* Using Stream ID: 1 (easy handle 0x561b29eb0580)
* TLSv1.3 (OUT), TLS Unknown, Unknown (23):
> GET /api/v1/namespaces/wwwnginx/services/http:my-nginx-clusterip:80/proxy/ HTTP/2
> Host: 10.11.2.247:6443
> User-Agent: curl/7.58.0
> Accept: */*
> Authorization: Bearer eyJhbGciOiJSUz … m9F4tJX1TUVHZoXlmBC5KRIuL0RK80e0U3jATk8RGrDrrcfkjtUUKQVKy5wae4Iptzi1cmawJLLMCrx4wfIyBqOzhV8aDk0H-6r0eOqsRqujg2nkg
>
* TLSv1.3 (IN), TLS Unknown, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS Unknown, Unknown (23):
* Connection state changed (MAX_CONCURRENT_STREAMS updated)!
* TLSv1.3 (OUT), TLS Unknown, Unknown (23):
* TLSv1.3 (IN), TLS Unknown, Unknown (23):
* TLSv1.3 (IN), TLS Unknown, Unknown (23):
* TLSv1.3 (IN), TLS Unknown, Unknown (23):
< HTTP/2 200
< accept-ranges: bytes
< content-type: text/html
< date: Mon, 06 Jul 2020 13:35:57 GMT
< etag: “54999765–264”
< last-modified: Tue, 23 Dec 2014 16:25:09 GMT
< server: nginx/1.7.9
< content-length: 612
<
* TLSv1.3 (IN), TLS Unknown, Unknown (23):
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
 body {
 width: 35em;
 margin: 0 auto;
 font-family: Tahoma, Verdana, Arial, sans-serif;
 }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p><p>For online documentation and support please refer to
<a href=”http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href=”http://nginx.com/">nginx.com</a>.</p><p><em>Thank you for using nginx.</em></p>
</body>
</html>
* Connection #0 to host 10.11.2.247 left intact

8. Or alternatively via certificates from the master node

Depending on what cluster you're working, the location of the certificates might be different. This example was tested on a kubeadm cluster.

$ sudo curl --cacert /etc/kubernetes/pki/ca.crt \
--key /etc/kubernetes/pki/apiserver-kubelet-client.key \
--cert /etc/kubernetes/pki/apiserver-kubelet-client.crt \
https://$MASTER:6443/api/v1/namespaces/wwwnginx/services/http:my-nginx-clusterip:80/proxy/

<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
 body {
 width: 35em;
 margin: 0 auto;
 font-family: Tahoma, Verdana, Arial, sans-serif;
 }
</style>
…

9. Or via the extracted certificates from the kubeconfig file

$ cat $HOME/.kube/config
apiVersion: v1
clusters:
- cluster:
 certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSU… NBVEUtLS0tLQo=
 server: https://10.11.2.247:6443
 name: kubernetes
contexts:
- context:
 cluster: kubernetes
 user: kubernetes-admin
 name: kubernetes-admin@kubernetes
current-context: kubernetes-admin@kubernetes
kind: Config
preferences: {}
users:
- name: kubernetes-admin
 user:
 client-certificate-data: LS0tLS1CRUdJTiBDRVJUS…kQgQ0VSVElGSUNBVEUtLS0tLQo=
 client-key-data: LS0tLS1CRUdJTiBSU0EgUFJJVkFURSBLRVktLS0tLQpNSUl…dXpFWS0tLS0tCg==

$ grep 'client-certificate-data: ' .kube/config | sed 's/^.*: //' | base64 -d >cert.pem

$ grep 'client-key-data: ' .kube/config | sed 's/^.*: //' | base64 -d >key.pem

$ grep 'certificate-authority-data: ' .kube/config | sed 's/^.*: //' | base64 -d >ca.pem

sudo curl --cacert ./ca.pem \
 --key ./key.pem \
 --cert ./cert.pem \
https://$MASTER:6443/api/v1/namespaces/wwwnginx/services/http:my-nginx-clusterip:80/proxy/

<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
 body {
 width: 35em;
 margin: 0 auto;
 font-family: Tahoma, Verdana, Arial, sans-serif;
...
 }

10. Via kubectl proxy

$ kubectl proxy &

$ curl http://localhost:8001/api/v1/namespaces/wwwnginx/services/http:my-nginx-clusterip:80/proxy/
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
 ...

11. Conclusion

I hope this small tutorial helped in beter understanding how to authenticate agianst the kube-api server using curl and provided some tips and tricks in helping mastering Kubernetes !!!

More at https://xxradar.medium.com and https://github.com/xxradar.

Feel free to reach out on Philippe Bogaerts or https://twitter.com/xxradar

kubiosec.io