One thing was apparent in the data management space over the past year; the job of chief data officers became increasingly complex as the volume of data their organisations ingested swelled and the uses to which it was put expanded. All that despite the flowering of technologies with the potential to make life easier for them.
Data Management Insight’s blogs and publications during 2024 reflected this and here we present the key themes that dominated our coverage of this fast-evolving corner of the FinTech world.
Unstructured Data
Tackling the challenges of gathering and managing unstructured data seemed to be at the forefront of data stewards more than any other issue in the past year.
More and more institutions are relying on information the resides in reports, publications, messages, transcripts and website pages but that has been hitherto difficult to mine systematically.
The surge has been driven in part by technology; artificial intelligence (AI) – generative AI in particular – is being deployed to pull valuable nuggets knowledge from huge unstructured pools at speeds and levels of accuracy undreamed of even a few years ago. And the expanding uses cases for data has also stimulated the need for more information.
While it may now be easier to obtain, unstructured data still poses management challenges. Wrangling it into a form that can be easily ingested into organisations’ tech stacks remains difficult and quality issues abound.
“Unstructured data has always existed but not at the scale it does now,” Ashly Joseph, data management lead at JP Morgan, told DMI in September. “To look at unstructured data we need better capabilities like AI, natural language processing, the ability to look at images and extract that information – lots of technology capability.”
Vendors including BlueFlame, TurinTech and Informatica released tools this year that are built around large language models to help organisations harness the potential of GenAI in supporting their data strategies. Bloomberg also extended its AI capabilities with release of an earnings call summary tool at the beginning of the year.
“The rapidly expanding capabilities of AI have proven very effective at scaling this approach, especially against large amounts of unstructured data,” Brian Greenberg, business engagement lead for enterprise data management at BNY told DMI before our Data Management Summit New York in the autumn.
The challenges have not been confined to technology.
Governance and aspiration have also posed obstacles to the roll out of many AI programmes. A poll in one of our webinars found that the biggest headache for organisations is managing the sheer volume of unstructured data that GenAI can collect. Consequently, while most of the online event’s viewers said that they were using such data to at least a “large extent”, only 20 per cent said they were able to take advantage of it to a “great extent”.
Nevertheless, the opportunity for deep analysis provided by the latest frontier in data capture and management should not be underestimated. Unearthing insights can lead to broadly felt benefits: innovative products and services can be developed from them; better business models could be built and better decisions made. Further, the time to market for those decisions can be accelerated, and risks can be identified and mitigated faster to help prevent financial losses and reputational damage.
Entity Identifiers and Standards
As organisations absorbed a continuing tsunami of data into their systems, their data management processes came under greater stress. That was made more challenging by the broader array of use cases to which that data is being applied as asset managers and banks move into more diverse asset classes and regulators pile more reporting obligations on them.
Outside the orderly world of public markets, organisations have encountered an unruly landscape of competing entity identification systems that has forced them to adjust their data management processes accordingly.
Having a lucid identification system for issuers ensures data consistency, accuracy and interoperability so that institutions can use it in their investment, risk, regulatory compliance and other processes.
“The amount of data that financial services firms are engaging with in their financial instrument processes is growing exponentially. Therefore, the need for data standards and identifiers is growing alongside this,” Laura Stanley, director of entity data and symbology at LSEG told DMI in the summer.
LSEG is among several issuers of commercial identifiers, along with Dun & Bradstreet, both of whom collaborated to bring transparency to private market entities in November.
Most prominent this year in building its own structure has been the Global Legal Entity Identifier Foundation (GLEIF), a not-for-profit standard setter. GLEIF had a busy 2024 testing a digital vLEI entity validation platform and changing its leadership, with the appointment of Alexandre Kech as its chief executive.
Over the past 12 months GLEIF has forged collaborations and partnerships with FinTechs, including Finbridge Gobal, and opened new validation agents including in China and India.
“Identifiers and standards play a critical role in data management,” Kech told DMI. “They facilitate clear identification and categorisation of data, enabling efficient data integration, sharing, and analysis.”
Financial institutions, corporates and other legal entities would struggle in many of their data processes without them, he said in July.
Our webinar on the issue in July identified many benefits that a solid system of legal identifiers and standards would bring, including better decision making, reduced costs through the elimination of manual data reconciliation and lower friction during data exchanges.
“I see financial institutions using data standards and identifiers – beyond compliance – to a great extent,” Robert Muller, director and senior group manager, technology product owner, at BNY told DMI. “There are a number of best practices firms can employ, for instance strategy, design and education, to ensure standards and identifiers deliver value through associated business cases.”
AI Loses Some of its Lustre
The initial rush to build AI into organisations’ data strategies appeared to wane a little over the past 12 months. As the costs of implementation became apparent and mounting evidence suggested that the returns on those investments might be moderate at best, some early adopters took stock of their AI transformations.
Observers began questioning the value of a still evolving technology and others warned that the euphoria surrounding the blind adoption of the technology had echoes of the dot-com hysteria that eventually led to a catastrophic crash at the turn of the millennium.
At the heart of the challenge is the data that fuels the technology. Without good-quality and complete data sets, AI models are no better than existing technologies. In fact, with the propensity of some models to offer an answer no matter the quality of data fed into it, the potential for erroneous outputs increased over traditional alternatives.
The upshot: Getting the AI data piece is not easy.
“You really can’t just say to AI,’ just go do this stuff’,” Ellen Gentile, enterprise data quality team leader at Edward Jones told DMI in September. “What is not always apparent is the pre- or prep-work you have to do with the data and the tool itself. Telling it to write a rule sounds great, but there’s a back end to it, and what happens is you must create the data sets, or you must profile the data. It’s not straightforward and easy.”
A slew of surveys over the year presented contradictory accounts of AI’s uptake and value to institutional data managers, too. From organisations’ legacy technology stacks holding back adoption to accelerated implementation of GenAI, studies of the use of AI illustrated an industry that is still testing the waters on the technology.
AI, nevertheless, has huge potential for data managers. At our ESG Data and Tech Summit in May, S&P Global Market Intelligence managing director of commercial strategy Neil Robertson said the company had built a range of AI-led products because the technology had become indispensable to institutions’ sustainability risk and portfolio managers.
AI can also help address the data challenge that dogs its very adoption, panellists at a webinar in September agreed. A poll of market participants, data consumers and vendors that tuned into the webinar said they were deploying AI to cleanse and validate data. That finding was bolstered by another poll, which found data quality time to value was the main challenge for respondents.
Gil Cross, platform and AI product manager at Xceptor said AI’s appeal lies in its ability to ensure good data quality so that it can be deployed to make enterprise-wide work easier and reduce human touch points.
ESG and Private Markets Data
Despite a concerted political effort to shift focus away from sustainability in capital markets, regulators and consumers have ensured that the need for institutions to absorb and manage ESG data remains as crucial as ever.
It’s a realm that sits at the confluence of all the challenges listed above that bedevil individual elements of the data pipeline. ESG data is largely unstructured, its quality is often less than optimal and full of gaps, it suffers from a lack of standards and experiments with the application of AI to solve these challenges remain to bear fruit.
One challenging aspect of ESG data management appears to be on the wane, however: the reliance on sustainability ratings. Providers of such aggregated data scores were the obvious first call for organisations as they began their ESG data journey almost a decade ago. But the often confusing and seemingly arbitrary assessments that the ratings providers offered called into question their methodologies, which usually lay locked from scrutiny in “black boxes”.
The ESG pushback focussed on the inconsistencies of within ratings products as the root cause for greenwashing. That pricked up the ears of regulators around the world, some of whom decided in 2024 year to bring issuers under their purview.
Other data challenges associated with ESG integration have been solved too. For instance, the past 12 months saw the variety of ESG data feeds blossom as the move away from one-size-fits all ratings paved the way for the disaggregation and decompositions of data feeds. This has stimulated the emergence of smaller vendors selling niche data sets and larger vendors creating targeted products from their broad pools of information.
This was most evident in the rise of biodiversity as a key focus in 2024. Bloomberg, ISS ESG and S&P Dow Jones were among companies that released nature-related data services and indices this year as another year of worsening severe weather events brought the risks of global warming into sharp relief.
Even so, the ESG data management space remains subject to long-established headwinds. Key among them is mapping climate and other non-financial information to entity data, a challenge that’s been compounded by the swelling number of data feeds that organisations now use.
“We are still in the early stages, and there are still limitations in how companies report their data,” Ángel Agudo, board director and SVP of product at Clarity AI, told DMI before an ESG data sourcing webinar in June. “There remains a need to put all that unstructured data together to make it comparable and to complement what’s missing. That means there will be a need to emulate that data through estimates and leverage other sources of information, which could include reports of other organisations, NGO information, news, asset-level data – and more. Ultimately, investors need to make sure the data sourced is fit for purpose.”
Another relatively new data imperative of institutions has arisen with their investment diversification strategies. About a third of all institutional capital is invested within private and alternative markets, creating demand for yet more data. It’s a demand that has been notoriously difficult to satisfy owing to the opacity of these markets and the immaturity of their data management practices.
That’s been doubly worse for investors who have wanted ESG data from private companies; the markets comprise smaller companies that aren’t attributed the same legal identification codes as listed public businesses and face fewer sustainability performance reporting obligations.
Again, Bloomberg joined the charge of companies that came to investors’ rescue, offering private-market data services and gauges, along with LSEG and a variety of startups including BlueFlame.
Subscribe to our newsletter
