Drowning in data while the floodwaters rise

co-founder and co-inventor, Mark Wharton explains how we could get more out of technology if data was connected

Educational 19 May 2022 by Mark Wharton

All forward steps in human technology have always met with some resistance. The Luddites were against the mechanisation of the textile industry. People thought that steam trains would go so fast that the breath would be sucked out of the lungs of passengers and I’m sure the cave-dweller who invented the wheel had rocks thrown at them. Despite their detractors, most advances have made a positive impact on society and people’s health, lives and work. On many measures: life expectancy, infant mortality, human rights etc, it could be argued that this is the best time to be alive, ever. Except…

“Any sufficiently advanced technology is indistinguishable from magic.”

Wrote Arthur C Clarke in 1962 – the year of my birth. I’m sure that people in the sixties would think that things we now take for granted: video calls, instant access to information on any topic, step-by-step directions from anywhere to anywhere, and so on… all from a super-computer that fits in your pocket was magic. Except…

Except what? Given the hope of technology and its undoubted magical qualities, why am I not hopeful about the impact of technology on solving the world’s problems? Think of how, rather than being rulers of technology, we’re ruled by it. Think of how bitcoin mining is contributing to climate change. Think of how server farms being installed under the arctic sea for cooling are contributing to rising sea temperatures.

I’ve just had a birthday with a zero on the end. Much as I tell myself numbers are arbitrary and if we had 16 fingers, I’d be thirty-twelve (programmer joke) you do tend to look back on things. With the benefit of experience, you can discern patterns in the evolution of the computing industry and see how the same mistakes have been made time and again irrespective of the contemporaneous fashion for Mainframe, Mini, PC, Distributed system, Cloud, Fog or Ubiquitous computing. There’s always a reason: cost-cutting, limited resources, lack of time are the oft-quoted ones, but there are deeper patterns of behaviour that, unchallenged,  will stop us from addressing the problems we face collectively as a species.

The historical patterns of behaviour that will hold us back in future are mainly due to the way that enterprises have bought software packages from other enterprises. The purchasing enterprise wants to keep their own data secure for corporate confidentiality purposes, the selling enterprise wants to keep the data in their system isolated for product “stickiness” purposes. As there’s no one-stop-shop solution for enterprises that covers everything from CRM to ERP to PLM, purchasing enterprises end up with a lot of different solutions that don’t talk to each other, because “stickiness”. Where does this pattern lead? Isolated silos of un-related (and un-relatable) data in isolated enterprises.

Why does any of this matter? Surely an enterprise that makes say, cars, does just that. They make Fiats, Renaults, Mercedes, whatever and that’s the end of it. Except… An enterprise that runs rail services can just do that. Except… Except that it’s no longer enough.

The future economy is about cooperation. It’s not enough to be a Railway or a Car manufacturer, as the future needs Mobility as a Service solutions with blended journeys. Your old business model won’t wash in a servitised business model – especially one that relies on a global supply chain.

The big problems of the future (sustainability, ageing population, pandemics…) have two things in common

  1. They will be solved using data
  2. They won’t be solved by only one organisation

Go figure. Data sharing is the future and the pandemic has proven that the old rules don’t apply. Historical precedent doesn’t work when the ground rules have changed. How does transport stand up given the ubiquity of video conferencing? Do you need a smart city when people move to the country as they can work from home? Do historical patterns of use of shopping malls matter when a pandemic hits and everyone shops online?

What the economy of the future needs is:

  1. Data agility
  2. Data interoperability
  3. Data reuse

A lot of people think that the “old” ways will work in the new paradigm. It worked before, therefore it’ll work again. Try telling that to the cavalry officers facing barbed-wire, mud and machine guns in WW1.

This “it worked before” thinking is not just confined to old lags like me. You don’t have to be old to be an old lag. People’s whole careers have been built on the “nobody got fired for buying IBM” philosophy. (Substitute anything trendy for IBM in that last quote.) The big-data and REST-API brigade have had a good run, but REST APIs are bad for streaming and therefore data and business agility. Big data relies too much on the activities of data consumers and slows down interoperability owing to lack of metadata. Complex governance rules destroy any hopes of data reuse – especially cross-enterprise.

Give up hope, then. The problems are species-threatening. The solutions are old-fashioned. The industry is conservative and risk-averse. Enterprises won’t cooperate and share data.  Except…

Except that the answer has been there all along. I only recently saw Tim Berners-Lee’s original “Vague but Interesting” diagram from 1989. The answer is decentralised webs of data with the semantic web to link it all together. The data-centric architecture as preached by Dave McComb since around 2000. We all use a data-centric application hundreds of times a day. A web browser. Browsers don’t know how to *do* anything. They’re told what to do by the data they receive. Embedding intelligence in the data means you can have Facebook in one tab and Compare the market in another. Giants of our industry saw the answer many years ago. That gives me hope.

Can we see any evidence of these new, cooperative and data-centric approaches? Independently of each other? Definitely yes. The Amazon Web Services (AWS) Partner ecosystem espousing doing what you are great at and collaborating with others. The rise and engagement of the Digital Twin Hub and cooperation between players. Work by AMRC in Sheffield cooperating on manufacturing. Independently, the data-centric approach has its champions too: Semantic ArtsCinchyGo-FAIR and their “Internet of FAIR Data & Services” are all advocating an approach that puts data cooperation at its centre.

Are there any instances of the two approaches being used in conjunction? The Clean Maritime project comprising 50-odd projects, 100s of organisations creating a community of players. IOTICS has built a digital twin ecosystem that is data centric (semantically described Digital Twins) and cooperative (each player has their own space in a network of spaces). We’re seeing Portsmouth Port and Utilities companies, universities and SMEs cooperating on a problem that is too intractable for any one of them to solve. That gives me hope.

What if all data was as distributed, searchable, accessible and shareable as web-pages? What new business models, economies and solutions could we come up with then? Our partners, customers, community and friends are starting to see what was under our noses all along, and to see the impact of doing different things rather than doing the same old things differently. That gives me hope.


Join Our Community

We enable the world’s data to interact safely and securely with other data, of all types, in all places, dynamically.