Catalogue of Services

Any physical testing activity comprises three main components:

• the environment (where the tests are conducted),
• the protocol (which defines the tests to be executed and their methodology),
• and the evaluation metrics (used to assess the test results).

This service focuses on the protocol, while the environment and metrics can be designed as needed through services S00106 and S00108.

In the context of testing customer solutions within physical facilities, this service aims to create a suitable testing protocol based on the use cases specified by the customer. The components of the testing protocol defined in this phase may include:

• Defining the different phases of the protocol and their duration
• Outlining the operations to be executed in each phase
• Quantifying the effort required for each phase, including the number and qualifications of personnel involved
• Distributing testing operations over time, considering seasonal and daily variations, as well as weather conditions
• Establishing acceptable and desired variation ranges for each configurable element in the testing environment
• Determining the number of test repetitions required and their distribution across the variation ranges

To ensure a comprehensive approach to protocol design, this service involves a collaborative team of both engineers and agronomists.

Any test activity involves three main components, i.e., environment (where the tests take place), protocol (defining what activities are executed and how), and evaluation metrics (used to assess the results of the tests). This service concerns the second element, i.e., the design of the testing procedure for digital systems such as (for instance) AI models or computer vision software. The digital environment and the evaluation metrics can be designed – if required – via services S00176 and S00178.

In the context of testing customers’ solutions within digital environments, this service is targeted at designing a suitable protocol for digital testing based on the use cases specified by the customer. The components of the testing protocol can include:

• Selecting the datasets to be used for testing

• Selecting reference AI models to be used for testing (if needed)

• Choosing data formats and metadata standards 

• Defining data pre-processing and preparation steps 

• Defining values and ranges of test parameters

• Defining the different phases of the protocol

• Outlining the operations to be executed in each phase

• Thoroughly describing the protocol specifics to ensure reproducibility

Datasets to be used for testing can be provided by agrifoodTEF and/or by the customer; if nothing suitable is available, other agrifoodTEF services can be leveraged to collect and/or generate tailored data.

The technical team executing this service comprises expert engineers but can also involve agronomists when this is necessary to ensure the relevance of the tests for the use case, e.g., to determine the distribution of test repetitions across the variation ranges.

By having your AI system tested by the LNE, you ensure it meets the highest standards of safety and performance, boosting your product’s reliability and trustworthiness in the market. Partnering with the LNE for rigorous testing of your AI technologies and a detailed evaluation report used for demonstrating performance enhances credibility and opens doors to new market opportunities both locally and globally. This service proposes to assess a range of agricultural devices (with respect to the physical constraints of the testing bench) integrating AI, particularly those utilising vision processing, within our advanced hybrid testing environment. Our novel hybrid testing facility consists of placing devices in the heart of a simulation projected onto a 300° screen while a motion capture system and instrumented conveyor belt measure its movements, if any. These data are continuously sent to the simulator in real time so that the device's digital twin follows the same movements and the projected environment is updated accordingly. The simulator also incorporates a physics engine and advanced sensor models, enabling a virtual sensor output to be substituted for the sensors in real time in the cases where devices require specific data and/or if the assessment is orientated toward a special kind of sensor degradation. Typical agricultural products evaluated include mobile robots for autonomous weeding or harvesting that use visual navigation (based on 2D cameras) and intelligent cameras (with AI functionalities) for crop health monitoring or livestock tracking. Other sensors commonly used in agriculture, such as 3D cameras for yield estimation, GPS for autonomous vehicle control, Lidar for terrain mapping, and sonar for obstacle detection, are also supported through data injection from the simulator.

By having your AI system tested by the LNE, you ensure it meets the highest standards of safety and performance, boosting your product’s reliability and trustworthiness in the market. Partnering with the LNE for rigorous testing of your AI technologies gives you a competitive edge, as their certification enhances credibility and opens doors to new market opportunities both locally and globally. LNE's AI performance evaluation service uses comprehensive testing datasets to assess the accuracy, robustness, and efficiency of your AI systems by comparing the outputs of the system with a dataset of reference values. By testing real-world scenarios, LNE ensures that your AI models meet industry standards and regulatory requirements, helping you improve performance, reliability, and market readiness. LNE utilises a diverse range of carefully curated datasets that simulate various operating conditions and environments in which the AI may be deployed. These datasets allow for in-depth testing of the system’s ability to process information, make decisions, and produce accurate outputs. The service covers a broad spectrum of AI applications, from machine learning models and deep learning algorithms to computer vision systems, natural language processing (NLP), and autonomous robotics. The evaluation process examines key performance metrics such as accuracy, precision, recall, response time and scalability. It also identifies any potential biases in the system, ensuring that the AI behaves fairly and ethically across different user groups or environmental variables.

Through this service we offer customers a complete feasibility study of their digital systems and/or data, evaluating their suitability for the test use cases identified by the customer. The study involves both technical and agronomic aspects. The goal of this service is to provide the customer with a comparison between the features of the system or data submitted for review and the requirements of the use cases chosen for (possible) subsequent testing.

Examples of systems and data requiring digital testing include (but are not limited to) software modules for data processing (e.g., machine vision modules), AI models, software architectures (e.g., control system of a robot), simulations, datasets collected in the field, synthetic datasets, designs of software systems, and so on.

This assessment is crucial to provide customers with key information on elements of their system that need to be further developed to become “fit for testing”. It can also identify changes to apply to either the system or agrifoodTEF’s testing facilities to meet the requirements for experimentation.

Through this assessment, we offer customers a complete feasibility study of the use cases and systems that customers would like to test, comprising both technical and agronomic aspects. Examples of systems requiring physical testing include (but are not limited to) precision sprayers, weeding machines, robotised fruit picking systems, and other implements; examples of subsystems include (but are not limited to) perception systems (e.g., based on vision), autonomous driving modules, localisation systems, and mapping and navigation subsystems.

This assessment is crucial to provide customers with key information on elements of the system that need to be further developed to become “fit for testing”. It can also identify changes to apply to either the system or AgrifoodTEF’s testing facilities to meet the test requirements.

This service concerns the processing of data collected during digital testing of a system, i.e., experimentation done in a computational environment. Digital testing enables easy collection of experimental data; however, such data then need to be suitably processed to extract information about system performance.

The service applies a predefined set of performance metrics (defined either by the customer or via service S00178) to the collected data. Data collection during test execution is expected to have been done by the customer (on request, this activity can be supported by agrifoodTEF via service S00183). The service includes all the activities needed to apply the performance metrics, including the development of any software component needed for data processing or data preparation.

The output of the service is a report providing the quantitative results of the application of the performance metrics to the experimental data. The report also includes an evaluation of these results aimed at highlighting possible issues and pointing out lines for improvement.

This service performs the crucial step that follows the execution of physical experimentation, i.e., evaluation of the performance of the system under test. This activity requires the application of suitable performance metrics to specific test data collected during the tests and then interpreting the results in view of the features of the system and the experimental setting. Depending on the specific use case, performance metrics can involve pure computer processing, human expertise by agronomists, or a combination of both.

Application of performance metrics may include the development of custom software to extract the necessary information from experimental data and/or to apply suitable processing to the information. If the performance metrics require the data to be subjected to some form of pre-processing, Service S00115 may be leveraged to prepare the data.

Beside the object of this service (i.e., evaluation of results), agrifoodTEF can, on request, support the customer along any other aspect and phase of the physical experimentation pipeline. For instance, for the design of the environment and protocol for the test, the customer can leverage services S00106 and S00107. Preparation of the test environment can be done either by the customer or by agrifoodTEF via service S00110; in both cases, assistance in interconnecting the system under test with agrifoodTEF’s test infrastructure is available via service S00111. Finally, data collection during the test can, if needed, be provided by agrifoodTEF via service S00113.  

By having your AI or robotic system tested by LNE, you ensure it meets the highest standards of safety and performance, boosting your product’s reliability and trustworthiness in the market. Partnering with LNE for rigorous testing of your robotic and AI technologies gives you a competitive edge and trusted third-party assessment that enhances your product credibility and opens doors to new market opportunities both locally and globally. This service provides testing of devices with AI-integrated control systems, focusing on energy efficiency and consumption during the product life cycle. By simulating a wide range of scenarios, we provide detailed insights into the energy performance of these AI systems with regard to their expected operational environment. These tests can be conducted under different conditions, either in a completely simulated way, through a hybrid test bench where the actual physical device is assessed within a simulated environment, or using physical infrastructures based on the device's operational environment to test it in a 'real' setting. Tests are conducted under controlled laboratory conditions, ensuring optimal reproducibility and repeatability, providing reliable insights into the system's efficiency and safety-critical functionalities. Typical assessments may cover the energy efficiency of SLAM, various types of end effectors, and other relevant components such as sensors, motors or AI software. This rigorous testing process helps identify potential areas for improvement, enabling companies to enhance the efficiency and sustainability of their products, ultimately leading to cost savings and reduced environmental impact.

This service assists customers in executing digital tests using agrifoodTEF’s digital infrastructure. Tests can either be defined by the customer or designed through agrifoodTEF services (e.g., S00176, S00177, S00178). If specific activities are required to properly interface the system under test with the digital infrastructure, these can be handled by either the customer or agrifoodTEF (via Service S00180).

Support provided by this service includes general assistance during the test execution phase, such as monitoring to ensure compliance with the testing protocol or offering immediate help in identifying and resolving issues. Additionally, the service provides technical support during the test execution of the system under test, including ensuring that all necessary processes are correctly configured and running, as well as helping the customer address any unforeseen challenges that may arise when complex digital systems interact.

Finally, through this service, agrifoodTEF manages the technical infrastructure for test execution, ensuring smooth data exchange between the system under test and the agrifoodTEF infrastructure, and verifying that all testing requirements—potentially identified during the desk assessment phase (Service S00179)—are met during the actual test runs.

Any physical test activity involves three main components: environment (where the tests take place), protocol (defining what tests are executed and how), and evaluation metrics (used to assess the results of the tests).

The execution of physical testing, i.e., the object of this service, corresponds to supporting the customer in executing with the system(s) under test the procedures specified by the protocol in the specified environment and identifying and solving any issues that may negatively affect the tests. In most cases, support by agrifoodTEF involves assistance by both agronomists (for agricultural aspects) and engineers (for aspects related to the technological infrastructure supporting the tests).

Beside the object of this service, i.e., test execution, agrifoodTEF can, if requested, support the customer along any other aspect and phase of the physical experimentation pipeline. For instance, for the design of the environment and protocol to be used for the test, the customer can either proceed autonomously or exploit services S00106 and S00107. Similarly, preparation of the test environment can be done either by the customer or by agrifoodTEF via service S00110; in both cases, assistance in interconnecting the system under test with agrifoodTEF’s test infrastructure is available via service S00111. Finally, data collection during the test or data processing and performance evaluation after the conclusion of the test, if needed, can be provided by agrifoodTEF via services S00113 and S00114, respectively.