Image
Overview
This service concerns the processing of data collected during digital testing of a system, i.e., experimentation done in a computational environment. Digital testing enables easy collection of experimental data; however, such data then need to be suitably processed to extract information about system performance.
The service applies a predefined set of performance metrics (defined either by the customer or via service S00178) to the collected data. Data collection during test execution is expected to have been done by the customer (on request, this activity can be supported by agrifoodTEF via service S00183). The service includes all the activities needed to apply the performance metrics, including the development of any software component needed for data processing or data preparation.
The output of the service is a report providing the quantitative results of the application of the performance metrics to the experimental data. The report also includes an evaluation of these results aimed at highlighting possible issues and pointing out lines for improvement.
More about the service
The application of specified performance metrics to experimental data can be a complex process, often requiring the development of custom software to manage data and perform the necessary processing actions.
A customer may decide to leave the application of the metrics to agrifoodTEF to focus instead on system development and testing and on the definition of performance metrics. This service allows this customer to directly receive the results of the performance evaluation.The report containing the results is prepared by the engineers who prepare and execute the metrics-driven processing.
These engineers are also experts in digital systems (e.g., AI models) and robotics for agrifood applications, which enables them to complement the results in the report with an in-depth analysis, e.g., to identify performance issues.
This analysis, which is part of the report, highlights issues in system performance and identifies the elements in the processing toolchain of the system under test that are most likely involved in the issues.
The customer can therefore leverage the report not only to understand if the performance of the system measures up to expectations but also to decide where to focus additional development effort to improve system performance.
A customer may decide to leave the application of the metrics to agrifoodTEF to focus instead on system development and testing and on the definition of performance metrics. This service allows this customer to directly receive the results of the performance evaluation.The report containing the results is prepared by the engineers who prepare and execute the metrics-driven processing.
These engineers are also experts in digital systems (e.g., AI models) and robotics for agrifood applications, which enables them to complement the results in the report with an in-depth analysis, e.g., to identify performance issues.
This analysis, which is part of the report, highlights issues in system performance and identifies the elements in the processing toolchain of the system under test that are most likely involved in the issues.
The customer can therefore leverage the report not only to understand if the performance of the system measures up to expectations but also to decide where to focus additional development effort to improve system performance.
The duration of this service is on average 3-4 weeks but depends on the quantity of data to be processed and the complexity of the operations needed to set up and execute data processing.The preparation phase of the service involves one or more interviews where the customer shares with agrifoodTEF information about the data to be analysed and the performance metrics to be applied.
Additional information (e.g., internal system features) may be useful to support the analysis accompanying the data in the report (see below); any such information can be shared under NDA if confidential.The preliminary exchange of information is followed by the transmission to agrifoodTEF of the data to be processed. At the end of the service, the customer receives a report containing the results of the application of the performance metrics to the data and an analysis of such results with indications about possible ways to improve system performance.
Additional information (e.g., internal system features) may be useful to support the analysis accompanying the data in the report (see below); any such information can be shared under NDA if confidential.The preliminary exchange of information is followed by the transmission to agrifoodTEF of the data to be processed. At the end of the service, the customer receives a report containing the results of the application of the performance metrics to the data and an analysis of such results with indications about possible ways to improve system performance.
This service description is intentionally generic. Every instance of this service is, in fact, customised to adapt it to the needs and requirements of the specific customer. The following is an example of a service instance.Example service: The performance of an intra-row robot navigation system is tested on different datasets in virtual simulation environments (e.g., via service S00183).
A set of metrics previously designed (e.g., via service S00178) are used to track specific aspects of the performance of the customer’s solution. The chosen metrics consider, in particular, the misplacement of the (simulated) robot from the centre row line, the number of waypoints correctly visited, and the number of collisions with plants. Metrics are computed for different runs and trajectories across the reference data, plotted/tabulated in a detailed report, which also includes qualitative videos demonstrating the navigation trajectory followed by the system in simulation.
The report also highlights the most significant deviations of the system from expected performance and provides links between them and particular situations/events, as documented in the (simulated) sensor data that have been collected during the execution of the tests, along with the datasets used for the performance metrics.
A set of metrics previously designed (e.g., via service S00178) are used to track specific aspects of the performance of the customer’s solution. The chosen metrics consider, in particular, the misplacement of the (simulated) robot from the centre row line, the number of waypoints correctly visited, and the number of collisions with plants. Metrics are computed for different runs and trajectories across the reference data, plotted/tabulated in a detailed report, which also includes qualitative videos demonstrating the navigation trajectory followed by the system in simulation.
The report also highlights the most significant deviations of the system from expected performance and provides links between them and particular situations/events, as documented in the (simulated) sensor data that have been collected during the execution of the tests, along with the datasets used for the performance metrics.
A set of metrics previously designed (e.g., via service S00178) are used to track specific aspects of the performance of the customer’s solution. The chosen metrics consider, in particular, the misplacement of the (simulated) robot from the centre row line, the number of waypoints correctly visited, and the number of collisions with plants. Metrics are computed for different runs and trajectories across the reference data, plotted/tabulated in a detailed report, which also includes qualitative videos demonstrating the navigation trajectory followed by the system in simulation.
The report also highlights the most significant deviations of the system from expected performance and provides links between them and particular situations/events, as documented in the (simulated) sensor data that have been collected during the execution of the tests, along with the datasets used for the performance metrics.
A set of metrics previously designed (e.g., via service S00178) are used to track specific aspects of the performance of the customer’s solution. The chosen metrics consider, in particular, the misplacement of the (simulated) robot from the centre row line, the number of waypoints correctly visited, and the number of collisions with plants. Metrics are computed for different runs and trajectories across the reference data, plotted/tabulated in a detailed report, which also includes qualitative videos demonstrating the navigation trajectory followed by the system in simulation.
The report also highlights the most significant deviations of the system from expected performance and provides links between them and particular situations/events, as documented in the (simulated) sensor data that have been collected during the execution of the tests, along with the datasets used for the performance metrics.
Service provider
Politecnico di Milano (POLIMI)
Università degli Studi di Milano (UMIL)
Location
Italy
Remote
Type of Sector
Arable farming
Food processing
Greenhouse
Horticulture
Livestock farming
Tree Crops
Viticulture
Type of service
Performance evaluation
Accepted type of products
Data
Design / Documentation
