Many financial institutions are still struggling to safely move their data into the cloud, hammered by a paucity of suitable technology and the prevalence of fragmented pipelines.
Decades of piecemeal system development, critical business logic, order flows, risk models and market data feeds have created architectures that has left data isolated an disconnected. As well, organisations risk replicating the disjointed nature of that data in the cloud unless interventions remediate the issue.
Fortunately, solutions are at hand that won’t require organisations to undertake huge, costly and potentially erroneous “lift-and-shift” projects to get their data into cloud environments, a recent A-Team Group Data Management Insight webinar established.
The increasing complexity of data and firms’ data needs have placed similarly complex pressures on their data management processes. Eliminating siloes is no longer a cost-cutting exercise but a requirement for scaling operations and driving innovation, especially artificial intelligence applications and agents, panellists at the webinar said.
The overriding message from was that succeeding in this transition requires a clear understanding of technical debt, architectural choices, and automated governance frameworks.
Preserving Knowledge
The webinar, entitled Executing the Migration to Cloud to Enable Scalability and Innovation, saw four leading market practitioners offer their views on the topic and discuss some of the best practices behind moving data.
Andrea DeSosa, Global Head of Capital Markets, GTM at Databricks was joined by Suemee Shin, Head of Enterprise Data Office Standards, Controls and Governance at US Bank; Tim Anderson, Director of Tick History at LSEG Data and Analytics; and Adrian Murray, Head of Product, Pricing and Reference Services – Data & Feeds at LSEG Data and Analytics.
Data Management Insight Editor Mark McCord moderated the discussion.
The webinar established that a central challenge lies in preserving enterprise context and institutional knowledge. Over time, operational intelligence often comes to reside in personnel rather than formal documentation, making it difficult to trace data origin, transformations and downstream dependencies.
Without mapping these relationships prior to migration, firms risk moving unnecessary or redundant data into cloud environments.
Shift or Refactor
Financial technology leaders must evaluate whether to execute a simple lift-and-shift migration or refactor applications for cloud-native operation. The former approach remains appropriate for commodity applications or legacy software nearing retirement, where testing like-for-like data is straightforward and transformation yields limited commercial return. However, true operational return on investment stems from application refactoring, the webinar heard.
Refactoring enables institutions to replace traditional time-series or relational setups with elastic architectures that support dynamic compute scaling on demand. It also prepares data structures for Model Context Protocol (MCP) integration and broad natural language prompting used in modern analytical workflows.
Vendor Feeds
Managing data egress and licensing costs represents a critical operational concern when porting vendor feeds to cloud platforms. Traditional patterns that rely on continuous data replication across regional cloud instances accumulate significant network egress fees and storage overheads.
To mitigate these expenses, modern architectures favour zero-copy data sharing, landing vendor content in unified environments and bringing analytical workloads directly to the data, the webinar heard.
Through native open-sharing frameworks, financial institutions can query vendor datasets alongside internal proprietary data without executing cumbersome ingestion pipelines. This model allows market data providers to manage primary storage while end-user firms maintain granular control over internal access permissions.
Operating across multi-cloud, multi-instance environments introduces severe governance and compliance complexities. Manual governance mechanisms, such as static spreadsheets or periodic committee reviews, fail to scale alongside expanding cloud workloads, panellists said. To maintain regulatory compliance and operational safety, institutions are transitioning toward automated governance approaches, including “controls as code”.
Under such a model, governance, lineage, data quality and regulatory policies are consolidated within central configuration files and deployed uniformly across infrastructure environments.
Semantic Layers
Feeding real-time analytics and artificial intelligence applications requires robust data pipelines grounded in strict data quality and standardised metadata. Because machine learning models amplify input errors, embedding semantic layers and standard identifiers – such as Legal Entity Identifiers – is essential for ensuring output accuracy and lineage tracking, the panellists said.
For high-frequency market data operating at nanosecond tick rates, streaming pipelines require dedicated infrastructure planning. Porting massive real-time data volumes directly into frontier AI models introduces unacceptable latency. Instead, institutions leverage hardware acceleration and open-source models running locally on specialised chips, combined with pre-indexed historical data layers, to execute backtesting and live execution efficiently.
Subscribe to our newsletter
