Gardin - ChF metrics not being processed correctly. – Incident details

The Platform team apologise for the inconvenience this issue caused. Our investigation found that the use of Glue bookmarks to incrementally retrieve **new** data that was queued for processing was silently preventing our pipeline from returning the full range of **historical** data neccessary to compute certain ChF metrics correctly, despite explicitly setting the time window that we required in the query. This issue proved difficult to replicate in a development envrionment, as it is neccessary to disable Glue bookmarks in this envrionment for testing. We have now replaced the use of Glue bookmarks for tracking pipeline progress with a “roll our own” alternative tailored to our needs. This increases visibility and provides ultimate control over the behaviour of incremental query loads. We have also increased the telemetry and logging we collect, to improve visibility further. All affected historcial data has now been reprocessed, and we will continue to monitor closely over the next 24-72 hours.