Storage is key to AI projects that succeed

Let’s first look at AI and the demands it places on compute and storage.

Broadly speaking, AI processing falls into two categories.

These are training, when recognition is generated from a model dataset, with varying degrees of human supervision; and inference, in which the trained model is put to work on real-world datasets. 

The components of a successful AI project start well before training, however.

Here, we’re talking about data collection and preparation, and with datasets that can vary hugely in nature. They can include backups, unstructured data, structured data and data curated into a data warehouse. Data might be held for long periods and prepared for AI training in a lengthy and considered process, or could be required rapidly for needs that were unexpected.

In other words, data for AI can take many forms and produce unpredictable requirements in terms of access.

In other words, AI is very hungry in terms of resources.

The voraciousness of graphics processing units (GPUs) is well-known, but it’s worth recapping. So, for example, when Meta trained its open source Llama 3.1 large language model (LLM), it is reported that it took around 40 million GPU hours on 16,000 GPUs. We’ll come back to what that means for storage below.

A large chunk of this is because AI uses vectorised data. Put simply, when training a model, the attributes of the dataset being trained on are translated to vectorised – high dimensional – data.

That means data – say the numerous characteristics of an image dataset – is converted to an ordered set of datapoints on multiple axes so they can be compared, their proximity to each other calculated, and their similarity or otherwise determined.

The result is that vector databases often see significant growth in dataset size compared to source data, with as much as 10 times possible. That all has to be stored somewhere.

Then there is frequent checkpointing to allow for recovery from failures, to be able to roll back to previous versions of a model should results need tuning, and to be able to demonstrate transparency in training for compliance purposes. Checkpoint size can vary according to model size and the number of checkpoints required, but it is likely to add significant data volume to storage capacity requirements.

Then there is retrieval augmented generation (RAG), which augments the model with internal data from the organisation, relevant to a specific industry vertical or academic specialisation, for example. Here again, RAG data depends on vectorising the dataset to allow it to be integrated into the overall architecture. 

To maximise chances of AI success, organisations need to ensure they have the capacity to store the data needed for AI training and the outputs that result from it, but also that storage is optimised so that energy can be conserved for data processing rather than retaining it in storage arrays