For investment firms seeking an informational edge, the web represents an extraordinarily rich source of potential data. Product inventories and pricing, corporate activity, technology adoption, infrastructure development and countless other indicators can all provide insights that may not be available through conventional market and financial data sources.
Extracting that information reliably, however, has traditionally required significant technical resources. That has tended to favour larger investment firms with the engineering expertise and budgets to build and maintain their own web-scraping infrastructure, while smaller firms have often had to outsource the work or buy pre-packaged datasets.
Kadoa, a Zurich-based web data platform primarily serving hedge funds, is using AI to change those economics by making it easier for investment professionals to build and maintain their own web data collection processes.
“If you were a smaller firm you probably wouldn’t have the resources to do it, or you’d outsource it and have somebody build a scraper for you. Or you’d buy already-scraped data,” says Tavis Lochhead, Co-Founder and CRO of Kadoa, in conversation with Market & Alt Data Insight. “Because of how hard web scraping was for a long time, the most economical or accessible way of sourcing web data would be just to buy it. But that data is quite commoditised, if anybody could just buy it off the shelf.”
From buying data to creating it
That distinction is important because the value of a dataset can lie not just in the information it contains, but in the fact that an investment firm has identified, collected and analysed information that its competitors may not be using in the same way.
Kadoa’s approach is therefore focused on providing the tools for firms to create their own datasets rather than supplying a catalogue of pre-packaged data products. Users describe in natural language the information they want to collect, with AI then used to research where that information resides, generate the web-scraping code, run and evaluate it, and iterate until it produces the required output.
AI also helps address one of the persistent operational problems associated with web scraping: websites change. A conventional scraper designed around a particular site structure can stop working when that structure is modified, potentially resulting in missing or delayed data. Kadoa uses self-healing capabilities to adapt the collection process as sources change.
The result is effectively a self-service approach to web data collection, lowering the technical barrier between an investment analyst’s research idea and the data required to investigate it.
Following the investment hypothesis
The potential applications are broad because the starting point is the user’s investment question rather than a predefined dataset.
A firm analysing a retailer, for example, might collect inventory and pricing information from its website and monitor changes over time. Other strategies might examine indicators around AI adoption, data-centre development, energy or semiconductor pricing. What matters is the ability to identify web-based information relevant to a particular investment thesis and turn it into structured, repeatable data.
For discretionary investors in particular, that potentially puts capabilities previously requiring programming and data-engineering expertise directly into the hands of analysts.
“Our main user base are discretionary firms,” says Lochhead. “They’re people that live in Excel and definitely enjoy chat interfaces. We built our product initially for those types of users, and then we’re starting to now move towards the other side, the systematic, which requires purely technical experience, and comes in the form of MCP or API.”
That expansion towards systematic users highlights a second aspect of the changing web-data landscape. Natural-language interfaces can make data collection accessible to non-technical users, but the same underlying capabilities can also become part of an automated investment data infrastructure through machine-to-machine interfaces.
Lowering barriers while raising the ceiling
The impact of AI on web data is therefore not solely about democratisation.
For smaller investment firms, reducing the engineering burden can make proprietary web-data collection viable where previously the practical option might have been to buy an existing dataset. But for larger firms that already have sophisticated web-data operations, AI can potentially increase the scale and complexity of what they can do.
Lochhead points to the ability to collect from more sources, handle blocking for any website cleanly and reliably at scale, operate at lower latency, chain different scraping processes together and handle more sophisticated unstructured-data processing. In that sense, AI is simultaneously lowering the barrier to entry and raising the ceiling for established users of web data.
That could have wider implications for the alternative data market. As the technology required to create bespoke datasets becomes more accessible, differentiation may increasingly depend less on access to off-the-shelf data and more on the questions investment firms choose to ask, the sources they identify and how effectively they incorporate the resulting information into their investment process.
For Kadoa, the opportunity is to provide the infrastructure connecting those questions with the vast quantity of information available on the open web. For investment firms, the bigger shift may be from choosing which alternative datasets to buy towards deciding which ones they want to create.
Subscribe to our newsletter
