Evaluation of results of digital testing

The application of specified performance metrics to experimental data can be a complex process, often requiring the development of custom software to manage data and perform the necessary processing actions. A customer may decide to leave the application of the metrics to agrifoodTEF to focus instead on system development and testing and on the definition of performance metrics. This service allows this customer to directly receive the results of the performance evaluation.The report containing the results is prepared by the engineers who prepare and execute the metrics-driven processing. These engineers are also experts in digital systems (e.g., AI models) and robotics for agrifood applications, which enables them to complement the results in the report with an in-depth analysis, e.g., to identify performance issues. This analysis, which is part of the report, highlights issues in system performance and identifies the elements in the processing toolchain of the system under test that are most likely involved in the issues. The customer can therefore leverage the report not only to understand if the performance of the system measures up to expectations but also to decide where to focus additional development effort to improve system performance.

The duration of this service is on average 3-4 weeks but depends on the quantity of data to be processed and the complexity of the operations needed to set up and execute data processing.The preparation phase of the service involves one or more interviews where the customer shares with agrifoodTEF information about the data to be analysed and the performance metrics to be applied. Additional information (e.g., internal system features) may be useful to support the analysis accompanying the data in the report (see below); any such information can be shared under NDA if confidential.The preliminary exchange of information is followed by the transmission to agrifoodTEF of the data to be processed. At the end of the service, the customer receives a report containing the results of the application of the performance metrics to the data and an analysis of such results with indications about possible ways to improve system performance.

This service description is intentionally generic. Every instance of this service is, in fact, customised to adapt it to the needs and requirements of the specific customer. The following is an example of a service instance.Example service: The performance of an intra-row robot navigation system is tested on different datasets in virtual simulation environments (e.g., via service S00183). A set of metrics previously designed (e.g., via service S00178) are used to track specific aspects of the performance of the customer’s solution. The chosen metrics consider, in particular, the misplacement of the (simulated) robot from the centre row line, the number of waypoints correctly visited, and the number of collisions with plants. Metrics are computed for different runs and trajectories across the reference data, plotted/tabulated in a detailed report, which also includes qualitative videos demonstrating the navigation trajectory followed by the system in simulation. The report also highlights the most significant deviations of the system from expected performance and provides links between them and particular situations/events, as documented in the (simulated) sensor data that have been collected during the execution of the tests, along with the datasets used for the performance metrics. A set of metrics previously designed (e.g., via service S00178) are used to track specific aspects of the performance of the customer’s solution. The chosen metrics consider, in particular, the misplacement of the (simulated) robot from the centre row line, the number of waypoints correctly visited, and the number of collisions with plants. Metrics are computed for different runs and trajectories across the reference data, plotted/tabulated in a detailed report, which also includes qualitative videos demonstrating the navigation trajectory followed by the system in simulation. The report also highlights the most significant deviations of the system from expected performance and provides links between them and particular situations/events, as documented in the (simulated) sensor data that have been collected during the execution of the tests, along with the datasets used for the performance metrics.