Result description
Verification and validation (V&V) of automated systems is becoming time-consuming due to the increasing complexity of these systems. Moreover, V&V of these systems can be hindered if the methods and processes are not properly described and selected. It is essential that researchers use suitable V&V methods and enact adequate V&V processes to confirm that the systems work as intended. VALU3S web-based repository for V&V automated systems is a user-friendly, open access, multi-layered framework consisting of a searchable catalogue of V&V methods and tools. These methods and tools have been applied across specific domains and tested under multiple application scenarios with respect to safety, cybersecurity, and privacy (SCP) requirements. The framework provides a standardised V&V process for automated systems by identifying eight dimensions that relate and answer questions about what, where, how and why the system assessment is performed.
V&V is becoming more costly due to the increasing complexity of automated systems. The ever-growing need of improving traditional V&V workflows to reduce costs and increase productivity has been driving the implementation of new methods and tools. However, the heterogeneous and domain-specific nature of V&V processes makes the evaluation of state-of-the-art methods and workflows challenging, which is highlighted by the fact that every industrial sector must comply with its distinct norms, practices and terminologies. To overcome these obstacles, the repository defines a domain-agnostic way of characterising and classifying V&V processes, with the aim of addressing a set of characterisation and classification aspects around the V&V methods, tools, and concepts. The structure of the V&V framework builds on two concepts: dimensions and artefacts. Dimensions are the set of properties that allows classifying the relevant features of the artefacts. Artefacts are the methods and tools used in the V&V process and applied to specific systems. Additionally, the framework enables the storage of other artefacts such as evaluation scenarios, and requirements.
Addressing target audiences and expressing needs
- Other blended financing
- Use of research Infrastructure
- Collaboration
For the V&V of any automated system, methods and processes have to be properly described, analysed, and selected. The VALU3S web-based repository provides a broad, comprehensive, and easy to use framework to address all these characterisation needs. Therefore, the consortium expects that the repository can gain high visibility and appeal to multiple stakeholders in the V&V domain so that it can become a benchmark where all practitioners and researchers may easily learn about and identify suitable V&V processes.
- Public or private funding institutions
- Research and Technology Organisations
- Academia/ Universities
R&D, Technology and Innovation aspects
The repository is populated with 55 V&V methods, 29 tools and 12 use cases with 52 evaluation scenarios, 193 test cases related to 155 requirements and 23 standards. The content is expected to be updated with new dimensions and layers, V&V methods, workflows and tools, and data on the evaluation criteria measured. The repository will be publicly generated resource.
Currently, each VALU3S partner has its own account to access the repository and add, modify or update its content. Being the consortium aware that updated information makes users more likely to visit the repository and extend its usefulness in the long term, it is expected that the repository content will be kept up to date by adding new dimensions and layers of the repository, new or improved V&V methods, workflows and tools, and data on the evaluation criteria. To this end, the VALU3S consortium expects the repository to be publicly generated resource. Similar to other information repositories such as Wikipedia, it is intended that anyone can participate in the population of this web repository. For this purpose, it is expected to establish reviewers, a committee, and an editor for each review. This designated technical board will be responsible for conducting a peer review process of external contributions, which will have an associated legal liability if accepted.
The repository is for public access, it can be set up and instantiated as a private tool. It is based on the Plone content management system, currently using Plone version 5 on Python 2.7 with a ZOPE database. Plone content types (dexterity types) have been created to store the data based on the multidimensional framework. To develop these elements, a Plone extension called VALU3S Extension has been created. This extension integrates the different types of content created (tools, methos, workflows…) with views and searches, and can be reused and installed in different Plone installations so that it can be used even in any company privately and without the project data. Another alternative would be to make a snapshot of the server and install as a replica of the current server in another location. Lastly, another option would be to create a docker container replica of the current installation. Currently, this solution is used for the local development of the Plone Extension of VALU3S.
Preserving the repository will help others identify project results, use the content to guide their V&V processes, clearly understand what has already been addressed and is still open, and even promote their own V&V methods and tools. As such, the repository will be updated regularly, either through the release of a new version or on a rolling basis. These releases may add new content or features, fix bugs, or address security issues. In addition, thanks to the consortium’s scalability strategy, it is expected that external organisations will also be able to populate the repository with their own V&V methods and tools following a peer-review process. This strategy will provide up-to-date content and is expected to encourage the use of the repository by third parties who will have a showcase to promote their research while maintaining their full intellectual property rights.
- Europe
- Asia
- South America
- North America
Result submitted to Horizon Results Platform by AIT AUSTRIAN INSTITUTE OF TECHNOLOGY GMBH