Result description
The Open Web Index (OWI) is the first European Web Index that allows European citizens and organisations to access Web information directly and securely, without having to rely on the benevolence of private companies. This means Europeans do not have to rely on commercial or otherwise biased gatekeepers, whose algorithms determine what they can find on the internet. Additionally, personal data are not being exploited without the consent of users. The OWI respects European legislation, such as the Digital Markets and Digital Services Act as well as the AI Act. Aside from the ethical & legal compliance, the OWI also differs in nature from established web indexes regarding its federated infrastructure. Five European data centres currently host the OWI, which makes it more resilient and reduces the before mentioned dependencies on commercial gatekeepers.
Who can benefit from the index and how? The index stores well structured open web data, making it available for search, analytics and AI applications including LLMs.Technically, anybody can download index files and build their own application or front-end.
A big benefit lies in the access to well-structured web data for AI applications.
Addressing target audiences and expressing needs
- To raise awareness and possibly influence policy
For further OWI development:
- Growth: The current federated infrastructure is hosted by 5 European data centres. We are looking for further data centres, from the academic/public as well as commercial domain.
- Funding: To scale the OWI, long term funding, incl. public funding, must be secured. The OWI should not be in the sole hands of commercial market players but aims for public ownership & curation.
- Legalities: Currently, the OWI can be used under a research license; a commercial license is outstanding, due to pending legal clarifications, regarding GDPR, DSA and copyrights.
- EU and Member State Policy-makers
- Research and Technology Organisations
- Academia/ Universities
R&D, Technology and Innovation aspects
The OWI has crawled more than a Petabyte of web data to date, and is indexing approximately 100 million new URLs per day. Currently the focus lies on full text bodies, metadata like titles and descriptions, embeddings and text excerpts. Images and videos are not yet being indexed.
The OWI currently offers a TLR 5. There is massive scale-up potential for the European economy. In 2024 macro-economic market potential study establishe, that the OWI could achieve a return on investment within four years of operation, with a projected net benefit of around €4.5 billion over a decade. These benefits are believed to derive from economic gains and societal improvements such as strengthening European digital sovereignty and global technological competitiveness across a wide range of industries and use cases. Find the full study for download here: https://openwebsearch.eu/the-project/research-results/market-potential-assessment-of-an-european-open-web-index/
The Open Web Index was modeled according to open-source standards. The Index is open by nature and research results from the project are publicly available on Github and Zenodo.
Replicability is therefore given, however, the true value lies in the federated web search and analytics infrastructure, which requires huge computing power and a solid network of long-term partnerships. Additionally, the fast-paced developments, especially in the AI domain, require continuous development resources to make the OWI adaptable to future technologies.
The federated, infrastructure-oriented and pan European approach to scale and operate the Open Web Index (OWI) as a public good ensures long term sustainability, once it is fully adopted and implemented in Europe.
- Europe
