The following is what SchoolMessenger reported as the reason for the downtime that lasted most of the day on April 15th. (It was not so they could finish filing their income tax. They are in Canada, not the US)
We have an update to share regarding recent issues impacting a number of Presence customers. After ongoing analysis and the clearing of various potential root causes by our technical teams and supporting vendors, we have located the source of the Presence site issues.
What Happened
As is standard in the web hosting industry, West partners with various Content Delivery Network (CDN) providers to speed page load times and more efficiently distribute website content to end users. To verify the authenticity of certain types of content, West shares security certificates with its CDN partners. These security certificates are issued by third party certificate authorities.
Beginning late in the evening on Sunday, April 14, a security certificate shared between West and Verizon was unexpectedly revoked by the third party authority. Initially the impact of the revocation was limited to a small number of end points so the scope of the issue was not immediately flagged as high severity. Throughout the morning of Monday, April 15, the certificate issue rapidly expanded to impact a larger percentage of end points. Immediately upon discovery of the scope of the issue, West engineers worked with Verizon and the third party certificate authority to rectify the issue. Once certificates were replaced, it took a few hours for the updates to fully propagate across all of the CDN endpoints and out to the various lookup services across the internet that reference such certificates.
The result is that for users in many regions throughout the US and Canada, web pages did not properly load certain graphic and aesthetic content for several hours throughout Monday and, in some very isolated cases of certain sites with users on specific operating system / browser configurations, into early Tuesday, April 16.
What We Are Doing for the Future
West is taking steps to increase the diversity of its CDN partner portfolio and improve systems to allow for swifter failover between such partners. In the unlikely event that a certificate is unexpectedly revoked in the future, or should other unforeseen events occur with any one such partner, having this multi-partner failover system fully in place will significantly improve response times.
At this time, all Presence sites are stable and have been so for a sustained period. Thank you again for your patience and understanding through these recent events.
Regards,
Customer Engagement team
West (SchoolMessenger solutions)
Watch for a blog post early next week about what accessibility issues in the Monsido reports you should be worried about.