Generative AI at Scale: The Path from Hype to Enterprise Reality

By Phil Wilkinson, Head of Product and Engineering at IOTICS

In October a colleague and I travelled to San Francisco for Anyscale’s Ray Summit.  After the first day of the conference we made our way to a restaurant in Fisherman’s Wharf - about a 2 mile drive from the conference venue. We elected to take a Waymo as neither of us had yet experienced a journey in San Francisco’s autonomous taxis.

What a brilliant experience!

Inside our Waymo from Union Square to Fisherman's Wharf

We were both like school kids: giddy with excitement as the car pulled up at the sidewalk and even more so as the driverless Jaguar I-PACE began navigating us expertly and swiftly through the streets.  Here’s the thing: I understand how LiDAR works; I understand computer vision and machine learning; I am familiar with basic robotics and mechanical control systems; and yet, as I sat in that car I felt pure joy.

I want to feel the same joy about AI within the enterprise, but I don’t – yet.

Yes, I’m incredibly impressed by the capabilities of many of the current foundational models and agentic frameworks, but it doesn’t give me joy.  It doesn’t give me joy because it isn’t able to navigate the chaotic landscape in which it exists, unlike the way that our Waymo beautifully navigated the chaotic streets of San Francisco.

I know the potential is immense and AI could, should and must redefine how businesses operate. But between the concept and the reality lie some particularly thorny challenges, and addressing them is critical to moving AI from the lab to the boardroom.

I want to point out that AI, particularly in the form of machine learning, has been transforming multiple industries over the past two decades, quietly and incrementally solving some incredibly complex challenges such as genome sequencing, materials selection for manufacturing, and fraud detection. But the expectation has changed, and investors are now expecting broader, generalised and more rapid adoption of AI in every business.

The Trouble with Semantics

One of the first hurdles generative AI faces in enterprise environments is understanding the context of the landscape in which it exists. AI models excel at generating fluent text, but their grip on meaning can be shaky—particularly when it comes to the subtle, complex noun and verb phrases that are the lifeblood of domain-specific applications. Many will have chuckled at the now famous example of AI being asked to create an image of “Jesus flipping over tables”.

Viral meme from r/TheDarkTower on reddit

In business or industry scenarios, context is critical. Decisions cost dollars. Mistakes can be literally fatal.

Enterprises need their AI systems to understand their terminology, grasp contextual nuances, and provide outputs that reflect real-world meaning. Semantic layers and knowledge graphs are powerful tools in this regard. By embedding domain-specific relationships and ontologies into the AI ecosystem, these technologies provide the grounding that generative AI desperately needs to avoid misunderstandings.

Aside from avoiding error, the ability to disambiguate and relate knowledge across the enterprise (many of which can be thought of as clusters of mini-companies siloed by function and geography rather than single organisations in their own right) enables organisations to properly harness their data intellectual property and amplify its potential through AI.

Training Drift

Training drift is the tendency for models to gradually lose relevance as the world changes around them - a bit like teaching a kid about floppy disks in 1995 and expecting them to be useful in 2024. Enterprises are dynamic, constantly evolving systems, and AI models trained on yesterday’s data are ill-equipped to meet today’s demands.

The seemingly obvious answer—retraining—is easier said than done. Retraining large models is mind-boggingly expensive, time-consuming, and requires skilled technologists who are in short supply. Worse, by the time your retrained model is deployed, the world might have already moved on.

For example, fraud prevention techniques require a combination of historic, pattern-based analyses augmented with detailed facts about the immediate actions a specific actor has taken in order to make a decision as to whether a crime may be taking place or not.

Instead, enterprises should focus on leveraging live, proprietary data streams that relieve the need for constant retraining. This is where approaches like Retrieval Augmented Generation (RAG) are increasingly being adopted. By allowing AI systems to access live, relevant data during inference, RAG ensures outputs remain current and accurate without the Sisyphean task of perpetual retraining cycles.

Data Management

Enterprises are drowning in data, but most of it is poorly catalogued, siloed, or otherwise inaccessible. In fact, studies estimate that only 65% of data in the typical enterprise is tagged or catalogued in any way, meaning roughly 35% of data isn’t discoverable.  Furthermore, it’s estimated that up to 30% of enterprise data is never read or accessed by humans or machines for any purpose.  Considering that data storage and management consumes 20% of the average IT budget, that’s a pretty damning statistic.

Data needs to be discoverable and accessible, and that’s no small feat in complex organisations. Semantic layers and knowledge graphs come to the rescue here as well, turning chaotic datasets into structured, queryable resources. They allow AI systems—and the humans who depend on them—to find the right data at the right time, whether it’s historical sales figures, IoT sensor readings, or compliance records.

At IOTICS, we’ve seen first hand how better data management can transform enterprises. By enabling real-time data sharing with built-in provenance and control, organisations not only make their data usable but also unlock its potential to drive innovation.

National resilience AI application powered by IOTICS' federated knowledge graph

Trust and Governance

Enterprises need to trust the data their AI is using, trust the processes responsible for generating insights, and, most importantly, trust the outputs themselves. Without this trust, AI adoption stalls, and the technology’s promise goes unrealized.

Trust starts with provenance—knowing exactly where your data comes from and whether it’s reliable. RAG once again proves its worth here. By tying AI outputs to live, trusted data sources, RAG not only enhances inference accuracy but also provides a clear audit trail for every result. This is particularly critical in regulated industries like finance or healthcare, where decision-grade data is a necessity.

Often, organisations will have genuine - sometimes mandatory - reasons to impose security boundaries based on data classification, business function, or geography.  AI systems cannot be the means to bypass those security boundaries and therefore the ability to decentralise becomes an advantage.  With a decentralised approach, AI systems can be deployed within security boundaries whereby they only have access to permitted data for inference, while leveraging common models and pipelines across the enterprise.  At IOTICS, we have become specialists in this particular space, helping multiple organisations and multi-party ecosystems to establish trusted data networks, at scale, through the use of decentralisation.

Bridging the Skills Gap

Enterprises face a stark shortage of technologists capable of implementing and managing AI systems. Expecting every organisation to hire teams of data engineers, data scientists and AI engineers is unrealistic. Furthermore, it’s estimated that between 60-80% of enterprise data is unstructured (PDFs, Excel, Video etc) and the majority of people who interact with this data every day are non-technical knowledge workers.

Generative AI needs to be as accessible to the CFO as it is to the data scientist. Natural language interfaces, role-based access, and intuitive tools are key to making AI usable for all stakeholders. When business users can interact with AI systems as easily as they would with a colleague, adoption accelerates, and the technology’s benefits become universal.

The Way Forward

For enterprises to harness the potential of AI at scale, they must confront and overcome the challenges of semantic complexity, training drift, lack of context, data accessibility, trust and governance, and skills shortages.

The good news is that the tools to address these challenges are already here. Semantic layers, knowledge graphs, RAG, and platforms like IOTICS are enabling enterprises to bridge the gaps between data, AI, and actionable insights at scale. The future of AI isn’t just about building smarter models—it’s about building systems that are trustworthy, adaptable, and accessible to everyone.

In other words, if we want AI to truly earn its keep, it’s time to stop treating it like a shiny toy and start treating it like the transformative tool it was always meant to be.