GlobalSign/AlphaSSL OCSP/CRL/TimeStamping Error

Last updated March 19, 2020 at 13:00 CET - All systems ok - Incident report available.

The 11th of March at 10:00 am, we were alerted about an issue with GlobalSign's online services being unavailable, which cause temporary failures on CRL / OCSP services, that validate that certificates have not been withdrawn.

GlobalSign did not report the error via email or twitter alerts to their customers or partners, but simply wrote a note on their server status page here https://www.globalsign.com/en/status

GlobalSign is aware of the error and is actively working to solve the problem, which they themselves describe as sporadic, ie. that the error comes and goes. We found out they had the mistake for an hour yesterday, but now have had the mistake for several hours today.

In general, browsers that have previously obtained a GlobalSign certificate will not be affected as they cache the result. But a new connection could potentially fail, which would cause browser errors on websites that use GlobalSign certificates.

We know from customers that it gives errors to some types of clients, especially clients who always check CRL / OCSP as remote applications, RDP and less often in web browsers.

In addition, their CodeSigning TimeStamping services and certificate issuance are also affected.

We have no estimate of when the error has been fixed, but we are constantly updating this page.

We have a single solution for critical systems which is to replace the certificate with a certificate from another issuer, eg DigiCert, Sectigo, Thawte, GeoTrust, RapidSSL.

Since we offer certificates from multiple CAs, we can immediately create a replacement certificate from another root CA without the problem. If you have a critical system, with a GlobalSign or AlphaSSL certificate, we would like to help with a replacement certificate and make sure the cost is as low as possible.

We are available on +45 77 345 678 / +46 (0) 10 101 0334 we keep the phone open until the main services are again available, for assistance with replacement of certificates, also thru the evening / night.

Update 11th March 2020 at 13:35 CET
GlobalSign has posted on twitter just before 1pm and updated their status, stating that it is a network problem in their primary data center, causing the errors on their online services. They are working on the problem and believe they will be back soon.

We have been constantly testing their systems and can see that most tests we run now no longer fail. Which indicates that the problem is being solved.

We also find that most of the customers we have been in contact with do not have errors in their operations.

Update 11th March 2020 at 17:00 CET
Globalsign has not updated their status and we still see intermittent problems.

We don't see problems in most popular browsers on websites, but theoretically there could be issues. We do see issues with API clients, server to server communication, remote desktop, etc.

Below is a graph showing the uptime of a GlobalSign CRL for the last 24 hours.

 

Update 11th March 2020 at 20:15 CET
GlobalSigns status page reports the error has been resolved.
And when they have completed their incident examination, they will publish a report.

The last hours we have not seen any errors in the uptime monitoring.

Update 12th March 2020 at 12:30 CET
GlobalSigns status page reports the error has been resolved, but due to backlog there may still be errors happening. However we see several errors in the testing still. The main CRL lists are improving with half as many errors as the same time yesterday.

Update 12th March 2020 at 16:55 CET
As we are still seeing intermittent errors, we are keeping our support phones open today from 8 am to 9 pm CET, for any customer with issues needing support. We are able to replace all certificates from another CA, with short notice.

Update 13th March 2020 at 14:00 CET
We are now seeing orders going thru again, certificates are being issued from both GlobalSign and AlphaSSL. The primary OCSP services for GlobalSign EV and OV certificates are responding as expected since midnight and are still going strong.

GlobalSign reports having solved the issues and they are now working on clearing any backlog, support requests and tweaking caching to remove old 503 cached responses on OCSP services.

We are still seeing errors on the AlphaSSL OCSP located here http://ocsp2.globalsign.com/gsalphasha2g2

We will continue to monitor the situation and update if there are any changes. 
In case anyone has trouble due to these errors, please contact us after hours on e-mail and we will respond or call back quickly.

Updated March 16, 2020 at 11:45 CET
All OCSP servers are now responding correctly.
Certificates are being issued.
Revokes are being processed.
We believe everyting is back to normal.

We will await the incident report and share it here when it becomes available.

Last updated March 18, 2020 at 13:00 CET
GlobalSign has issued an incident report explaining the cause of the outages.
It appears the main reason is a malware attack that on purpose asks for random non-existent certificate ids, to ensure the request can not be cached.

The report is available here:
https://www.globalsign-media.com/en/incident-report/2020-03-18-singapore-dc