Even before Covid and the accelerated period of digital transformation, parts of the private sector had undergone a data revolution. Data analytics now sit firmly at the heart of many organisations influencing what communications we receive, what product recommendations we are offered, the news we are served and even the creation of TV shows based on what an algorithm believes will be successful.
But what does this mean for councils and local public services. Local authorities has a wealth of information – everything from social care for vulnerable children, refuse collection, procurement, council tax and planning applications. This information can be used to make services more targeted and effective, to allocate resources where they have the biggest impact and to provide insight into social issues that impact our lives. And increasingly, as a result, more and more councils are starting to work with data specialists to maximise the value of their data.
However, data science, machine learning and AI are all very cutting edge and exciting technologies, but the stark reality is that they are only ever as powerful as the data that drives them. And this is where the importance of data hygiene comes in. The old adage rubbish in, rubbish out is true. If the data is out of date, or wrong then any analytics or algorithms build using that data will ultimately be flawed. At best this means that communications might be sent to the wrong people at worst it could mean unconsciously discriminating against vulnerable social groups. Consequently it is unsurprising that data scientists spend 65 per cent of their time preparing the data for analysis.
However, data hygiene is more than just a fundamental process for analytics – it is also a fundamental process full stop – by law. Under GDPR councils are required to keep the data they hold up to date. We recently carried out a survey amongst councils and discovered that only 12 per cent of UK councils are fully compliant with GDPR when it comes to Article 5 Principles Relating to Processing of Personal Data.
We sent out a Freedom of Information request to all UK councils asking how they kept their personal data up to date and accurate, particularly pertaining to home movers and people that have passed away.
Twelve per cent of councils admitted that they do not keep their data up to date and do not have any processes in place to clean or delete data.
Twenty per cent of councils provided a vague affirmative that they had a process to keep records clean but did not specify what these were or how regularly the data was updated.
One council located in the South West of England responded “we do clean our data if we are aware of individuals who have either passed away or have moved house.”
Thirteen percent of councils refused the request on the basis that it would be too costly and time consuming to respond.
One council in Wales said “I can confirm we do hold the data you are requesting. However, this information is exempt from disclosure under Section 21 of the Freedom of Information Act (FoIA), as it is already reasonably accessible to you”, citing their online retention schedule. The link however, was inactive.
Fifteen per cent of councils requested clarification as to what was meant by ‘data accuracy’ or ‘consumer data’ despite the question using the explicit wording given in GDPR Article 5.
Over a quarter of councils (28 per cent) admitted that they only updated their data when informed by a constituent that they had moved house or that a family member had passed away.
A council in the East of England responded: “We rely on information updates from residents in order to keep our data up to date. Whenever we receive information we update databases to that effect. This an ongoing process rather than a regular scheduled update.”
What was very clear from the study is that most councils have data transformation ambitions and are aware of the fundamentals. But that is reality the data silos and legacy systems form very rigid barriers when it comes to ensuring that data is well maintained. Moving forwards, as councils demand more from their data it will be important to put new processes into place to ensure that the data is kept as clean as possible. Increasingly there are cost efficient and effective ways for doing this. For instance, it was recently announced that the government had signed a new three-year Memorandum of Understanding (MOU) with Microsoft to enable public sector organisations and councils to benefit from Dynamics 365 – Microsoft’s CRM system. Whilst integrating a new CRM platform can be daunting, there is at least a silver lining: data hygiene is taken care of! Since it is possible to utilise an in-platform data hygiene solution which enables Dynamics 356 customers to keep the data stored within the platform clean and up-to-date adhering to both GDPR and data protection best practice. This means that it is no longer necessary to transfer the data out of the Azure ecosystem to be cleaned.
As we move into the Covid recovery period there is increasing pressure on local councils to capitalise on their data, but before doing so getting the basics in order will be the priority.
Contact us to understand the many innovative ways we can address this issue, for councils and any other organisation – ben.warren@thesoftwarebureau.com
