BEIS replaces Excel-based grant system with data management portal

Ryan Moore, head of data and analytics at Aiimi, said: “The blended BEIS and Aiimi project team has set new standards in how we approach data platform design, applying user research techniques to understand and define the solution’s capabilities. Our joint effort has created a solid grounding for an evolving data platform and a powerful tool in measuring the performance of initiatives designed to drive economic change and recovery across the UK.”

Jack Lawton, data science principal at Aiimi, has been the project lead on the engagement with BEIS. “We see content and data as two sides of the same coin,” he said. “If you start treating content like data and vice-versa, you get the most value out of each. That’s our approach.”

Aiimi’s main product, Insight Engine, is designed to treat content and data the same way, he said, but it does not feature in the BEIS project.

The firm has grown from around 30 people when Lawton joined seven years ago, to over 100 now, he said. It moved from London to Milton Keynes to accommodate the growth but operates a remote-fist policy for recruitment.

The BEIS project team comprised 10-15 Aiimi consultants at different times, and a similar number of civil servants.

The project is based on the Microsoft Azure platform. The data lake it has set up is supplemented by a data capture application that is in conformance with BEIS’s IT security policy.

On the data processing and analysis side, the Databricks technology it uses comes into its own, he said, because the data is not always supplied in the right format. “We’re dealing with data submitted by local authorities, in the form of Excel spreadsheets. Unfortunately, Excel allows a lot of freedom to make changes, which can bring problems when you want to bring in a whole lot of them, and somehow unify them,” said Lawton.

“We see content and data as two sides of the same coin. If you start treating content like data and vice-versa, you get the most value out of each. That’s our approach”

Jack Lawton, Aiimi

Databricks allows the running of complex analytics at scale, using Apache Spark, which is the open source technology that the founders of Databricks originally invented. The project team has also made use of Databricks’ Delta Lake data storage system, the virtue of which is to show the history of what has been done on a dataset, as well as processing it, said Lawton.

In terms of the organisational design of the data management system project team, he said it exemplified the department’s general approach, which is to run it as a blended team. “Who knows BEIS data better than BEIS? They are the subject matter experts. And by developing it with them, we are upskilling the team so they can support the DMS when we’ve finished,” said Lawton.

The role of the DMS is to enable BEIS to better report to government on the grants applications data gathered by local authorities from individuals and companies. The cleansed and unified data store can also be analysed using machine learning algorithms or dashboards.

“One of the key elements of the DMS platform is the ‘lab’ and ‘exhibits’ layers in the data lake,” said Lawton. “The purpose of these is to enable analysts to explore and play with the data and show it without too many strings. And then, if that analysis is of value to the business, it is relatively straightforward to adapt it under central management by the DMS team.

“The ability to take data in from external sources into the data capture portal is the differentiating benefit of this system for BEIS. You can take sensitive data from local authorities and securely pipeline it in. Everything is virus scanned.”

The data engineering for the project has been completed, as has a limited roll-out of the Covid grant returns, but the full system is not yet in the hands of local authorities. The contract runs until December 2022.