This is a pretty typical nightmare scenario. An engineer suddenly quit and I was part of a new team that was maintaining a brittle system with little documentation. Someone pings me and says, hey this website isn't working anymore - it says the cert's invalid! halp!
Debugging SSL issues are tough, especially when I hadn't dealt with them before and no one on the team was too familiar with our setup. I understood that using SSL certificates gives you the green Secure lock in Chrome, but that's about it. All the existing tooling assumes some level of knowledge about what certificates are and how they work. I guess that's fair - but why can't it be noob friendly.
Anyway, here's what I've wound up learning to trace down certificate errors. If nothing else, this is the one website that has everything you need - https://www.sslshopper.com/article-most-common-openssl-commands.html. Judging by the page ranking, I'm sure that website has helped a fair number of people get started with SSL issues.
Finding information about your current certificate
Going to Chrome helps you see certificate information if you're dealing with a web app, but where Chrome fails, you need to view things via the CLI. In my case, I was in the middle of setting up a proof-of-concept setup of the Logstash in the Elasticsearch/Logstash/Kibana centralized logging setup. If you want encrypted communication between your servers and logstash, you need to configure it to use a certificate on a specific port.
To get your website's certificate: openssl s_client -connect my.website.com:443 or in logstash's case openssl s_client -connect logstash.example.com:5044
At the bottom of the output, you'll see whether the certificate checks out and if not, for what reason. Common reasons to see a certificate error
- hostname mismatch. The hostname on the certificate is for foo.bar.com, but you are bar.bar.com.
- expired
- self-signed certificate for a public-facing website
Getting new certificates
This used to be bad news. Before Let's Encrypt came along, this happened to me within my first month or two in a new company and this meant that we had a multi-day turnaround & a $1-2k sticker price for a signed certificate from a public certificate authority. I had to take this path for one of our certs, but fortunately, it was a pre-prod environment so customers didn't incur any downtime.
We had a folder containing a bunch of private keys and public keys, but we couldn't tell which private keys were used to sign which certificates. Using the commands below, you can use the md5's of all pieces of the certificate to verify that they all belong to the same certificate.
openssl x509 -noout -modulus -in certificate.crt | openssl md5
openssl rsa -noout -modulus -in i-think-its-this.key | openssl md5If you're on AWS and using some of their services anyway, use the Amazon Certificate Manager. It gives you the same workflow as Let's Encrypt, except you can use the certificates on AWS services. You get free certificates that auto-renew.
Find out who's presenting the certificate
Different places you could find a certificate being presented on AWS:
- A Cloudfront distribution may be presenting a *.cloudfront.net certificate, or if using custom domains, a certificate that you've uploaded to AWS
- Elastic Load Balancers can present a certificate via HTTPS ports and either
- terminate SSL at the load balancer and send plain text to the backend instance:
openssl s_client -connect myservice-xxxx.ec2.amazonaws.com - terminate SSL and then validate a different certificate on the backend:
openssl s_client -connect 10.0.0.10:443 - pass the connection via TCP straight to the backend instance and let the instance present the certificate:
openssl s_client -connect 10.0.0.10:5000
- terminate SSL at the load balancer and send plain text to the backend instance:
You can upload your own certificates to IAM. There's no UI for this feature of IAM certificates. You can only upload and browse them via the CLI.
