Alex Ellis
Alex shows you some of the frustrations of using kubectl for port-forwarding and how to fix the developer experience.
I wrote up a feature-length article exploring all the ways that you can access a service from within your Kubernetes cluster.
The article covers pros and cons, and how-tos with the following approaches
kubectl port-forward
A Primer: Accessing services in Kubernetes
Out of all the options, the only one that we can guarantee will work on every cloud, Operating System and local desktop Kubernetes engine (Docker Desktop, KinD, K3d, Minikube), is port-forwarding.
That’s the magic of port-forwarding, and both kubectl and inlets can enable it.
For Kubernetes users, who’ve installed OpenFaaS, you’ll get Prometheus and the OpenFaaS gateway deployed as part of the helm chart.
arkade install openfaas
kubectl get svc -n openfaas
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
basic-auth-plugin ClusterIP 10.43.175.214 <none> 8080/TCP 84d
dashboard ClusterIP 10.43.88.246 <none> 8080/TCP 84d
gateway ClusterIP 10.43.114.157 <none> 8080/TCP 84d
prometheus ClusterIP 10.43.132.67 <none> 9090/TCP 84d
When you run kubectl port-forward
a persistent connection is opened between your workstation and the cluster through the Kubernetes API server, usually exposed on port 443 or 6443 over TLS. The traffic is then proxied between your local host and the cluster.
kubectl port-forward -n openfaas \
svc/gateway 8080:8080
curl -i http://127.0.0.1:8080/healthz
HTTP/1.1 200 OK
Content-Length: 2
Content-Type: text/plain; charset=utf-8
Date: Fri, 24 Jun 2022 08:31:27 GMT
OK
If you try to access the service via your local network, it will not work since port-forwarding only binds to the 127.0.0.1 address.
So if that’s something you need, because like me, you run multiple computers on your home network, add the following:
--address 0.0.0.0
Forwarding from 0.0.0.0:8080 -> 8080
1) If you restart a container, you have to restart the command
This may sound trivial, however it’s incredibly inconvenient if you are going through the “change/deploy/test” loop that we do so often when making enhancements to OpenFaaS or inlets itself.
2) There’s no load balancing at all
If you have three replicas of a HTTP service, and port-forward it with kubectl, then the command will only pick one of the replicas and stick to it. You can’t load balance in any way.
3) You have to run the command for every service you want to port-forward
This may be OK if you only have one service to poke at, but when you need OpenFaaS, Prometheus and perhaps one more thing that you’re building, it creates a significant amount of typing and distraction.
4) It’s unreliable and doesn’t reconnect
A developer in the UK government emailed me complaining of how he was port-forwarding NATS only to see it continually time-out and crash. He asked if inlets could help, and started using it in his daily workflow.
It solved his “headache” (his words) and remains connected. If for some reason the tunnel disconnects, only inlets will continue trying to reconnect until it is successful. Read the case-study here
5) You have to mentally map different ports if they clash
If you have three services which are all bound to port 8000, you need to change all your commands and keep them in mind, so that you allocate differing ports for them like 8081, 8082 and 8083. This is clunky, so can we do better?
But apart from the above points, it’s incredibly convenient and I use it a lot myself.
We can fix all of the issues above by using inlets.
“But I don’t want to expose anything on the Internet you say!” I hear you say
I get that. Inlets is a tool for connecting any two networks together, it does not require you to expose anything on the Internet whatsoever.
We need three steps to get this working:
You’ll be able to run this on MacOS, Windows and Linux, including Arm, Raspberry Pi and Apple M1.
So let’s install OpenFaaS and Grafana using arkade, so we have something to play with:
# Create a new cluster, or use one you already have
kind create cluster
arkade install openfaas
arkade install grafana --namespace openfaas
Even though we’re on a local network, with no public access, we’ll create a secure token an enable TLS:
openssl rand -base64 32 > token.txt
Now prepare the server command on your computer:
export LAN_IP="192.168.1.14"
export TOKEN="$(cat token.txt)"
inlets-pro http server \
--auto-tls \
--auto-tls-san $LAN_IP \
--token $TOKEN \
--port 8000
The port 8000 is what we’ll use to access all of our services, then we will multiplex them by using different local addresses in our /etc/hosts file.
Edit /etc/hosts:
127.0.0.1 grafana.svc.local
127.0.0.1 openfaas.svc.local
127.0.0.1 prometheus.svc.local
Save the file, and now we are ready to connect the client from within the cluster.
Save inlets.yaml:
export LICENSE="$(cat $HOME/.inlets/LICENSE)"
export TOKEN="$(cat ./token.txt)"
export WORKSTATION="192.168.1.14"
cat > inlets-client.yaml <<EOF
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: inlets-client
spec:
replicas: 1
selector:
matchLabels:
app: inlets-client
template:
metadata:
labels:
app: inlets-client
spec:
containers:
- name: inlets-client
image: ghcr.io/inlets/inlets-pro:0.9.5
imagePullPolicy: IfNotPresent
command: ["inlets-pro"]
args:
- "http"
- "client"
- "--url=wss://$WORKSTATION:8123"
- "--upstream=prometheus.svc.local=http://prometheus.openfaas:9090"
- "--upstream=gateway.svc.local=http://gateway.openfaas:8080"
- "--upstream=grafana.svc.local=http://grafana.grafana:80"
- "--token=$TOKEN"
- "--license=$LICENSE"
---
EOF
For any of the entries in /etc/hosts
, we just add them to the args
section:
So for Prometheus we have:
- "--upstream=prometheus.svc.local=http://prometheus.openfaas:9090"
That says that for any requests we get that match the hostname of prometheus.svc.local
, forward them to the cluster address of: http://prometheus.openfaas:9090
Now simply apply the deployment file to the cluster:
kubectl apply -f ./inlets-client.yaml
# Check its logs:
kubectl logs deploy/inlets-client -f
2022/06/24 08:51:35 Licensed to: Alex <contact@openfaas.com>, expires: 128 day(s)
2022/06/24 08:51:35 Upstream: prometheus.svc.local => http://prometheus.openfaas:9090
2022/06/24 08:51:35 Upstream: gateway.svc.local => http://gateway.openfaas:8080
2022/06/24 08:51:35 Upstream: grafana.svc.local => http://grafana.grafana:80
2022/06/24 08:51:35 unable to download CA from remote inlets server for auto-tls: Get "https://192.168.1.14:8123/.well-known/ca.crt": dial tcp 192.168.1.14:8123: connect: connection refused
time="2022/06/24 08:51:40" level=info msg="Connecting to proxy" url="wss://192.168.1.14:8123/connect"
time="2022/06/24 08:51:40" level=info msg="Connection established" client_id=1bdab2d4bef546c4b3c310e321a4b321
You can now access any of your forwarded services:
http://prometheus.svc.local:8000
http://grafana.svc.local:8000
http://gateway.svc.local:8000
Got another service you want to add? Just edit inlets.yaml and apply it again with kubectl.
Over on your server, you’ll see the requests being printed out as they come in:
inlets-pro http server --auto-tls --auto-tls-san $LAN_IP --token $TOKEN --port 8000
2022/06/24 09:51:35 Wrote: /tmp/certs/ca.crt
2022/06/24 09:51:35 Wrote: /tmp/certs/ca.key
2022/06/24 09:51:35 Wrote: /tmp/certs/server.crt
2022/06/24 09:51:35 Wrote: /tmp/certs/server.key
2022/06/24 09:51:35 TLS: 192.168.1.14, expires in: 2491.999998 days
2022/06/24 09:51:35 Control Plane Listening with TLS on 0.0.0.0:8123
2022/06/24 09:51:35 Data Plane Listening on 0.0.0.0:8000
2022/06/24 09:51:40 200 /connect
INFO[2022/06/24 09:51:40] Handling backend connection request [1bdab2d4bef546c4b3c310e321a4b321]
curl -i http://gateway.svc.local:8000/healthz
inlets-pro http server --auto-tls --auto-tls-san $LAN_IP --token $TOKEN --port 8000
2022/06/24 09:51:35 Wrote: /tmp/certs/ca.crt
2022/06/24 09:51:35 Wrote: /tmp/certs/ca.key
2022/06/24 09:51:35 Wrote: /tmp/certs/server.crt
2022/06/24 09:51:35 Wrote: /tmp/certs/server.key
2022/06/24 09:51:35 TLS: 192.168.1.14, expires in: 2491.999998 days
2022/06/24 09:51:35 Control Plane Listening with TLS on 0.0.0.0:8123
2022/06/24 09:51:35 Data Plane Listening on 0.0.0.0:8000
2022/06/24 09:51:40 200 /connect
INFO[2022/06/24 09:51:40] Handling backend connection request [1bdab2d4bef546c4b3c310e321a4b321]
2022/06/24 09:53:01 Proxy: gateway.svc.local:8000 GET => /healthz
You can try out inlets with a monthly subscription, and we have discounted plans for developers or for personal use, too.
Here’s what Han had to say about it, after switching over from kubectl port-forward
for his daily work on OpenFaaS.
My preferred method to access kubernetes services while developing locally:
— Han Verstraete (@welteki) June 17, 2022
Run a @inletsdev tunnel server on my local machine and set up a client in the cluster.
Thanks for the great article @alexellisuk https://t.co/ryWFEYUDSC
The port-forwarding I showed you today only needs one command and a simple Kubernetes YAML file to fix all the five issues I mentioned with kubectl port-forward
.
So do I still use kubectl port-forward
, despite its issues and poor developer experience? Of course. It’s the lowest common denominator, which works on every cloud and OS. But it’s really just meant for testing, and not for heavy, daily use.
So if you write and test code on a daily basis, inlets fixes the developer experience, and we hope you’ll try it out in your workflow too.
The example I shows you uses a HTTP tunnel, but if you’re working with TCP services, then you may want to check out the article I mentioned earlier in the post:
Inlets can also replace VPNs for private uplinks, it doesn’t need root or any Kernel modules, so it’s easy to automate just how you like.
And of course, it can be used to expose public endpoints, but we didn’t talk about that use-case today. If you’re on Kubernetes, our operator can provide public IPs and LoadBalancers:
Want to talk to us about inlets or networking in containers?
Subscribe for updates and new content from OpenFaaS Ltd.
By providing your email, you agree to receive marketing emails.