The Platform team apologise for the inconvenience this outage has caused. Unfortunately, due a deployment oversight that occurred whilst undertaking some testing of a new part of the platform AWS stack, the database and integration layer that supports the vitals telemetry dashboard was stood down and removed - the database on this occasion, was also irreversibly lost, making data reconstruction impossible. It should not have been possible for this to happen in a production environment.
We have now identified that additional automated safety checks that should have prevented this loss were not configured correctly, and after correcting this have now ensured that moving forward, these checks will block such loss of data. Additional process controls will also be put in place to guide the team through the deployment process.
