Update: The disruption has been resolved and services should resume as normal.
Summary: Parnassia was generating blocking sessions that caused proxy errors for them when proposing candidates. It appears to also have overflowed to the other customers, resulting in login issues for all customers. Killing the sessions appears to have resolved the issue.
Timeline:
09:18 (time in cest) call came in from Parnassia about a proxy error when proposing candidates.
09:45 Other customers started reporting login issues.
09:57 SiteUptime reported Fastforward Application Play! being Down.
10:10 SiteUptime reported Play being back up.
10:14 P5 was called (on CXS target) for the login issues, but closed out immediately again because the issue appeared resolved again. Tier 2 research into the Parnassia error continued as it was not resolved yet.
10:52 P5 raised on (old) FastForward target for the Parnassia issue with the proposal step. Once on the zoom bridge, reproduction became difficult because of general performance issues.
11:03 Login issues appeared to be back. Sev raised to P4.
11:13 SiteUptime officially confirmed that Play was down again.
~11:15-11:45 Nik, Dan and Matthias joined and started troubleshooting the issue, looking into ways to find and kill blocking sessions with Koen being on PTO.
11:31 SiteUptime showed Play! as up again, but it went back down on 11:53. In the mean time, support still received tickets from customers.
What is the matter?
We are currently experiencing a service disruption that influences the performance of the application and login. The disruption is intermittent, so retrying in a clean browser might improve performance.
What can I expect?
The effect of the disruption is that a login attempt keeps loading and after several minutes will show a proxy error. When already logged in, it could take a longer time for pages to load.
Solution?
We are currently investigating this with the highest urgency. Updates will follow through this article.
Questions?
Do you have any questions about this disruption? Please send an e-mail to support@connexys.com or call Bullhorn FastForward Support via tel. +31 10 498 09 88.
Comments