When the nginx-ingress controller has been implemented (see here), the next step is to implement cert-manager. Using cert-manager is a great way of providing your apps (websites) with a legitimate Let's Encrypt SSL certificate, automatically when configuring an ingress rule.
The installation procedure is as follows:
- Install cert-manager.
SSH into the kubernetes master node. This will create a new namespace calle cert-manager.
# Download and create cert-manager installation yaml curl -sL \ https://github.com/jetstack/cert-manager/releases/download/v1.3.1/cert-manager.yaml |\ sed -r 's/(image:.*):(v.*)$/\1-arm:\2/g' > cert-manager-arm.yaml # Apply the cert-manager installation yaml kubectl apply -f cert-manager-arm.yaml
- Public DNS & Portforwarding
- Add public DNS record for required certificate (for example nginx-1.mydomain.com).
- Portforward port 80, 443 from your public internet ip to the K8S external address via your (home) router Check my post on MetalLB here to find your external IP. For more information on how-to portforward ports from your router, this guide might provide you some information.
- Configure letsencrypt-staging.yaml
Next type the following command:
nano issuer-letsencrypt-staging.ymlPaste the text below and save the file (Ctrl-O, Ctrl-X). Change the red value into your own equivalent.
apiVersion: cert-manager.io/v1 kind: ClusterIssuer metadata: name: letsencrypt-staging spec: acme: # The ACME server URL server: https://acme-staging-v02.api.letsencrypt.org/directory # Email address used for ACME registration email: <your_email>@example.com # Name of a secret used to store the ACME account private key privateKeySecretRef: name: letsencrypt-staging # Enable the HTTP-01 challenge provider solvers: - http01: ingress: class: nginxApply the file in kubernetes with the command below
kubectl create -f issuer-letsencrypt-staging.yml -n cert-manager
- Check the staging ClusterIssuer installation
kubectl get clusterissuers
You should see that the letsencrypt-staging ClusterIssuer has been created.
We now have created a staging ClusterIssuer. It's best practice to first test cert-manager with a staging issuer, so you can check if everything is in place (firewall, network etc..) before configuring the production ClusterIssuer.
- Test certificate staging ClusterIssuer
To the the ClusterIssuer, we create a new certificate request via yaml file, so do the following
Adjust and past the text. As in previous posts red are the example values that needs to be adjusted to your own values.
apiVersion: cert-manager.io/v1 kind: Certificate metadata: name: nginx-1-mydomain-com namespace: default spec: secretName: nginx-1-mydomain-com-tls issuerRef: name: letsencrypt-staging kind: ClusterIssuer commonName: nginx-1.mydomain.com dnsNames: - nginx-1.mydomain.com
Before you apply the yaml file, doublecheck if the portforwarding and DNS records are in place. The DNS record doesn't have to be and A record a CNAME record will work as well. Port 80 is required to be forwarded from the router to the external ip of the ingress contoller. Let's encrypt will check if the host is available via port 80, before generating a certificate.
If all prerequistes are in place you can apply the yaml file and a certificate should be generated.
Apply the yaml file via the command:
kubectl apply -f nginx-1-test-certificate.yaml
You can check if the certificate has been generated via the command:
kubectl get certificates
You should see something similar like this (red values are example values)
erikdebont@k8s-master:~$ kubectl get certificates NAME READY SECRET AGE nginx-1.mydomain-com-tls True nginx-1.mydomain.com-tls 3d2h erikdebont@k8s-master:~$
If the READYstatus is FALSE you need to do some troubleshooting. If it's TRUE, the certificate has been generated and the staging proces went ok. It normally should not take longer than 10 minutes to generate a certificate, if it takes longer, than something went wrong. If the status is TRUE you can skip the trouble shooting guide and go the Configure letsencrypt-prod.yaml section
- Troubleshooting certificate request issues
You can start the troubleshooting with the following command:
kubectl describe certificates nginx-1-mydomain-com-tls
This will return information about the certificates proces. There also should be reference to a certificate request. Something like certificaterequest nginx-1-mydomain-com-12324563772
So the next step it to describe the request:
kubectl describe certificaterequest nginx-1-mydomain-com-12324563772
In this description you can find the order entry. This should be something like "Waiting on certificate issuance from order default/nginx-1-mydomain-com-12324563772-2342473670.
So let's describe the order
kubectl describe orders default/nginx-1-mydomain-com-12324563772-2342473670
In the order you look for the event that says Created Challenge resource "nginx-1-mydomain-com-12324563772-2342473670-1834560396" for domain "nginx-1.mydomain.com"
So let's describe the challenge:
kubectl describe challenge nginx-1-mydomain-com-12324563772-2342473670-1834560396
The last event returned from here is Presented challenge using http-01 challenge mechanism. That looks ok, so we scan up the describe output and see a message Waiting for http-01 challenge propagation: failed to perform self check GET request … no such host. Finally! We have found the problem! In this case, no such host means that the DNS lookup failed, so then we would go back and manually check our DNS settings and that our domain's DNS resolves correctly for us and make any changes needed.
(Thanks to https://opensource.com/article/20/3/ssl-letsencrypt-k3s for the excellent troubleshooting guide)
- Configure letsencrypt-prod.yaml
Before you start production configuration first cleanup the test request via the following command:
kubectl delete -f nginx-1-test-certificate.yaml
If creating a certificate via the letsencrypt-staging ClusterIssuer went ok, the backend is properly configured to install the producton ClusterIssuer. To implement the production ClusterIssuer create the following yaml-file. This is basically the same as the staging yaml file, with a couple of minor changes.
So type the following command:
Paste the text below and save the file (Ctrl-O, Ctrl-X). Change the red value into your own equivalent.
apiVersion: cert-manager.io/v1alpha2 kind: ClusterIssuer metadata: name: letsencrypt-prod spec: acme: # The ACME server URL server: https://acme-v02.api.letsencrypt.org/directory # Email address used for ACME registration email: <your_email>@example.com # Name of a secret used to store the ACME account private key privateKeySecretRef: name: letsencrypt-prod # Enable the HTTP-01 challenge provider solvers: - http01: ingress: class: nginx
Apply the file in kubernetes with the command below
kubectl create -f issuer-letsencrypt-prod.yml -n cert-manager
- Check the production ClusterIssuer installation
kubectl get clusterissuers
You now should two ClusterIssuers (staging and production)
Now certificates can be rolled with via Ingress in kubernetes by adding the follwing lines in the ingress.yaml file. Ingress will make sure there is a response when Let's Encrypt check the connection to host on port 80, before generating a valid ssl certificate.
apiVersion: networking.k8s.io/v1beta1 kind: Ingress metadata: name: nginx-1 annotations: kubernetes.io/ingress.class: "nginx" cert-manager.io/cluster-issuer: "letsencrypt-prod" # < This line tells ingress which issues to use spec: rules: - host: nginx-1.mydomain.com http: paths: - path: / backend: serviceName: nginx-1-service servicePort: 80 tls: # < placing a host in the TLS config will indicate a cert should be created - hosts: - nginx-1.mydomain.com secretName: nginx-1-mydomain-com-cloud-tls # < cert-manager will store the created certificate in this secret.
You can do a full test with the yaml file below. This will create test website nginx-1 with a certificate generated by Let's Encrypt through Ingress. Therefore do the following:
Copy and past the text below in nano, and save the file
apiVersion: apps/v1 kind: Deployment metadata: name: nginx-1 labels: app: nginx-1 spec: replicas: 1 selector: matchLabels: app: nginx-1 template: metadata: labels: app: nginx-1 spec: containers: - image: nginx name: nginx-1 --- kind: Service apiVersion: v1 metadata: name: nginx-1-service spec: selector: app: nginx-1 ports: - protocol: TCP port: 80 --- apiVersion: networking.k8s.io/v1beta1 kind: Ingress metadata: name: nginx-1 annotations: kubernetes.io/ingress.class: "nginx" cert-manager.io/cluster-issuer: "letsencrypt-prod" spec: rules: - host: nginx-1.mydomain.com http: paths: - path: / backend: serviceName: nginx-1-service servicePort: 80 tls: - hosts: - nginx-1.mydomain.com secretName: nginx-1-mydomain-com-tls
kubectl apply -f deploy-nginx-1-k8s-ssl.yml
To check if its working you should go to your browser and type https://nginx-1.mydomain.com. (mydomain.com should be replaced with your own domain) you should see default nginx html page, like the example below. You can check the certificate that has been generated.
the http solver is not working for me. any ideas on a dns solver?ReplyDelete
I noticed one step was missing in my blog. The actual apply of the cert-manager-arm.yaml file. I have corrected it, so you may try it again.Delete
My dns registrar doesn´t support dns solver, so I haven´t looked into it.
I am stuck at this stage and I am not sure what is the problem. I googled around without luckReplyDelete
$ kubectl get clusterissuers
NAME READY AGE
letsencrypt-staging False 32m
$ kubectl describe clusterissuers| grep Message
Message: Failed to register ACME account: Get "https://acme-staging-v02.api.letsencrypt.org/directory": dial tcp: lookup acme-staging-v02.api.letsencrypt.org on 10.96.0.10:53: server misbehaving
I get the impression it might be DNS related. Can you verify your workers can resolve acme-staging-v02.api.letsencrypt.org without any issues ?Delete
Thank you, that was it. funny I tested the dns from the master but not the workers.Delete