x509: certificate has expired or is not yet valid

[esbef]

Hi,

all of a sudden, the services stopped working after running stable for 6 months. The console gives me this:

kubectl get pods
Unable to connect to the server: x509: certificate has expired or is not yet valid

or

helm upgrade video team-video/ --values /etc/team-container/values-video.yaml
Error: UPGRADE FAILED: query: failed to query with labels: Get https://127.0.0.1:6443/api/v1/namespaces/default/secrets?labelSelector=name%3Dvideo%2Cowner%3Dhelm%2Cstatus%3Ddeployed: x509: certificate has expired or is not yet valid

I checked all certificates in /var/lib/rancher/k3s/server/tls/ with openssl. None are expired yet.

[esbef]

Setting the date back to 05/2020 resurrected the pods somehow but all with status "Terminating". After cleaning up the mess with "kubectl delete" and a fresh install via helm I receive an "ImagePullBackOff" error from Traefik pod with:

Failed to pull image "traefik:v2.2": rpc error: code = Unknown desc = failed to pull and unpack image "docker.io/library/traefik:v2.2": failed to resolve reference "docker.io/library/traefik:v2.2": failed to do request: Head https://registry-1.docker.io/v2/library/traefik/manifests/v2.2: x509: certificate has expired or is not yet valid

[jamun53]

Hi, I had a similar error after a restart a moth ago. Tried different things but a not sure which one helped. The last fix I applied before it worked again was this: https://linuxnotes.org/unable-to-connect-to-the-server-x509-certificate-has-expired-or-is-not-yet-valid/
If not it could lead you to other solutions. The error seems to occur because the certificates in the ct-teams installer are valid for only 1 year. This would be April 2020. In my case it worked without tampering with date and time. I had to install kubeadm to run the command.

[esbef]

Hi @jamun53 , thanks for the input! I fully agree that the certificates are the problem here. Following the instructions in your link I receive:

kubeadm certs renew all

MISSING! certificate embedded in the kubeconfig file for the admin to use and for kubeadm itself
MISSING! certificate for serving the Kubernetes API
MISSING! certificate the apiserver uses to access etcd
MISSING! certificate for the API server to connect to kubelet
MISSING! certificate embedded in the kubeconfig file for the controller manager to use
MISSING! certificate for liveness probes to healthcheck etcd
MISSING! certificate for etcd nodes to communicate with each other
MISSING! certificate for serving etcd
MISSING! certificate for the front proxy client
MISSING! certificate embedded in the kubeconfig file for the scheduler manager to use

or

kubeadm upgrade node

couldn't create a Kubernetes client from file "/etc/kubernetes/kubelet.conf": failed to load admin kubeconfig: open /etc/kubernetes/kubelet.conf: no such file or directory

Pointing to the k3s config file in /etc/rancher/k3s does not help as well.

Do you have any other suggestions?

[mclane]

I had the same problem. I could fix it by running sudo ./install.sh again. This obviously overwrites the k3s installation with the newest version and renews the certificates. The installed applications and their data (jitsi and nextcloud in my case) remained unchanged.

[kdeiss]

I had the same problem. I could fix it by running sudo ./install.sh again. This obviously overwrites the k3s installation with the newest version and renews the certificates. The installed applications and their data (jitsi and nextcloud in my case) remained unchanged.

I can confirm that this fixed my problem. May be you make a backup of your yaml files before you run the installer again.
Thanks!

[mshoff]

I had the same problem. I could fix it by running sudo ./install.sh again. This obviously overwrites the k3s installation with the newest version and renews the certificates. The installed applications and their data (jitsi and nextcloud in my case) remained unchanged.

After 12 months I get the same error message. Running sudo ./install.sh again did not work for me. I had to uninstall k3s (see rancher docs) and then install it.

However, now there is a new problem: "Error: INSTALLATION FAILED: failed to install CRD crds/traefik-crds.yaml: unable to recognize "": no matches for kind "CustomResourceDefinition" in version "apiextensions.k8s.io/v1beta1" (same was reported in #73 and #74)

[Wollipolli]

Same here.
I assume that we are reaching the end-of-lifetime for this specific docker setup. Also visible from in the commit history.
This is sad, because the main idea of a setup like this would be to keep on using it.

[mclane]

I have dropped the kubernetes / docker stuff altogether. Since I am just running a server for family use, I installed Jitsi and nextcloud directly.

[jamun53]

ran into the same problem again, found a new workaround:
kubectl --insecure-skip-tls-verify delete secret k3s-serving -n kube-system
rm -rf /var/lib/rancher/k3s/server/tls/dynamic-cert.json
systemctl restart k3s
k3s-io/k3s#5163