Catalogue of Services

With the development of AI models in most sectors of society, the importance of the data used to train and test these models has been an important concern for the AI supplier. Understanding the quality of the datasets used, especially in sectors such as the agricultural and agrifood domains where data can be rare, is thus a prerequisite to ensure the correct development of AI models and build trust regarding these approaches for society. This service aims to answer this challenge by assessing the quality of datasets provided by the client. The features of the datasets that will be analyzed are based on agreed definition by the AI community: completeness, balance, diversity, accuracy... The dataset's quality assessment will begin by establishing the list of the system's influencing factors necessary for the constitution of a database. These factors will be identified both by using LNE expertise and outside sources, such as agrifood-relevant experts. Using these factors, LNE will analyse the distribution (or weight) of the various influencing factors in the databases and compare it to that under normal conditions of use in order to assess the representativity and balance of the dataset.

Today’s AI suppliers and AI users need to stand out from the competition, guarantee the performance of their algorithms or systems, and prepare to comply with the future European "AI Act" regulation, particularly with the obligation of transparency. With its dual expertise in certification and evaluation of AI algorithms, LNE has created a voluntary certification that meets this need. The aim of this certification standard is to define common requirements for the design, development, evaluation, and maintenance processes of all types of AI functionalities based on machine learning. It therefore covers all business sectors in which AI is used in order to guarantee the application of best practices that promote trustworthiness in AI. Website: https://www.lne.fr/en/service/certification/certification-processes-ai Webinar: https://www.lne.fr/fr/webinars/matinee-information-ia-act-certification-ia-l Technical documentation (FR): https://www.lne.fr/sites/default/files/bloc-telecharger/FTC-ISO-42001-LNE.pdf

Through this service, we offer dedicated technical systems (hardware and software) and expertise for the collection of relevant data during testing.

The scope of data collected can range from data generated by the AgrifoodTEF technical infrastructure (e.g., through the installation of IoT sensors) to the collection of data generated by the system under test to the generation of ground truth for evaluating the system. If necessary, service S00111 may be leveraged to take care of the interconnection between AgrifoodTEF data collection systems and the customer’s systems.

In addition to the raw collected data, we will also provide customers with complete documentation describing the logged features and conditions of the testing environment at the time of testing, as well as any parameter values, variation ranges and specifics required for reproducibility purposes.

Where necessary, data collection and documentation activities can require the involvement of field experts (e.g., engineers, agronomists).

Note: annotation and labelling of collected data, if needed, can be provided via service S00115.

Through this service, we offerers the ability to validate and augment the data collected during the different testing phases. Examples of operations that this service can provide include quality checks (e.g., presence of all datastreams, presence of drop-outs, technical quality such as focus or exposure of camera images, agronomic significance of data); preparation for performance evaluation (e.g., identification of the images containing selected visual markers or specific plants, association between images from the system under test and the ground truth); data processing (e.g., transcoding); and augmentation (e.g., generation and incorporation of metadata). Data labelling (e.g., to enable training of AI models) is not included as it is the object of a separate service (S00291).

Any physical test activity involves three main components: environment (where the tests take place), protocol (defining what tests are executed and how) and evaluation metrics (used to assess the results of the tests). This service concerns the last element; its goal is to design the best metrics to evaluate the performance of a customer solution, taking into consideration the use cases specified by the customer and the environment and protocol chosen for the tests (which, if needed, can be designed via services S00106 and S00107).

Our team will identify and define with customers the most adequate set of quantitative (i.e., based on instrumental measurements) and/or qualitative (i.e., relying on expert human judgement) metrics to assess the system functionalities of interest. This phase will involve, in particular, agronomists and experts in agricultural machinery.

Based on the defined evaluation metrics, a set of requirements for the collection of required data and ground truth annotations will also be defined accordingly. For instance, the service may lay out the specifications for dedicated data collection campaigns (possibly executed via service S00113). This phase will involve engineers and experts in AI and robotics.

On request, the output of the service will include analyses on additional environmental factors than those directly tracked through the designed metrics (e.g., seasonal effects, impact of test distribution over time on results).

Any physical test activity involves three main components: environment (where the tests take place), protocol (defining what tests are executed and how) and evaluation metrics (used to assess the results of the tests). This service concerns the first element, i.e., the design of a physical testing environment for use cases such as (for instance) weeding, plant phenotyping, and precision spraying solutions. The protocol and the evaluation metrics can -if required- be designed via services S00107 and S00108.

Depending on your requirements and reference system/solution to be tested, our team will design an ad hoc setup equipped with all the required features for testing. Environmental features include, for example:

• the crop and weed species to be prepared and their growth stage,
• the plant layout and intra-row configuration,
• seasonal weather and climate-related conditions (e.g., lighting conditions, wind, rain), 
• the type of soil, moisture level, and terrain conditions (e.g., uneven terrain, presence of any slopes, and so forth),
• the technical infrastructure supporting the tests (e.g., electrical layout, network infrastructure, environmental sensors, data acquisition systems…).

In order to consider all aspects of the environment, this service involves a team comprising both engineers and agronomists.

By having your AI system tested by the LNE, you ensure it meets the highest standards of safety and performance, boosting your product’s reliability and trustworthiness in the market. Partnering with the LNE for rigorous testing of your AI technologies and a detailed evaluation report used for demonstrating performance enhances credibility and opens doors to new market opportunities both locally and globally. This service proposes to assess a range of agricultural devices (with respect to the physical constraints of the testing bench) integrating AI, particularly those utilising vision processing, within our advanced hybrid testing environment. Our novel hybrid testing facility consists of placing devices in the heart of a simulation projected onto a 300° screen while a motion capture system and instrumented conveyor belt measure its movements, if any. These data are continuously sent to the simulator in real time so that the device's digital twin follows the same movements and the projected environment is updated accordingly. The simulator also incorporates a physics engine and advanced sensor models, enabling a virtual sensor output to be substituted for the sensors in real time in the cases where devices require specific data and/or if the assessment is orientated toward a special kind of sensor degradation. Typical agricultural products evaluated include mobile robots for autonomous weeding or harvesting that use visual navigation (based on 2D cameras) and intelligent cameras (with AI functionalities) for crop health monitoring or livestock tracking. Other sensors commonly used in agriculture, such as 3D cameras for yield estimation, GPS for autonomous vehicle control, Lidar for terrain mapping, and sonar for obstacle detection, are also supported through data injection from the simulator.

By having your AI system tested by the LNE, you ensure it meets the highest standards of safety and performance, boosting your product’s reliability and trustworthiness in the market. Partnering with the LNE for rigorous testing of your AI technologies gives you a competitive edge, as their certification enhances credibility and opens doors to new market opportunities both locally and globally. LNE's AI performance evaluation service uses comprehensive testing datasets to assess the accuracy, robustness, and efficiency of your AI systems by comparing the outputs of the system with a dataset of reference values. By testing real-world scenarios, LNE ensures that your AI models meet industry standards and regulatory requirements, helping you improve performance, reliability, and market readiness. LNE utilises a diverse range of carefully curated datasets that simulate various operating conditions and environments in which the AI may be deployed. These datasets allow for in-depth testing of the system’s ability to process information, make decisions, and produce accurate outputs. The service covers a broad spectrum of AI applications, from machine learning models and deep learning algorithms to computer vision systems, natural language processing (NLP), and autonomous robotics. The evaluation process examines key performance metrics such as accuracy, precision, recall, response time and scalability. It also identifies any potential biases in the system, ensuring that the AI behaves fairly and ethically across different user groups or environmental variables.

We offer a specialised service to companies, particularly small and medium-sized enterprises (SMEs), that develop AI- and robotics-based products and services for the agri-food sector. Our service supports these companies in integrating ethical considerations throughout the design and development process of their AI- or robotics-based products and services. By utilising Value Sensitive Design (VSD) methods, we ensure that human values such as sustainability, fairness, transparency, and privacy are incorporated into technological solutions from the outset. Through stakeholder analysis, participatory design, value scenarios, and ethical reflection, we guide companies in creating responsible, innovative technologies that align with societal needs and ethical standards.

This service performs the crucial step that follows the execution of physical experimentation, i.e., evaluation of the performance of the system under test. This activity requires the application of suitable performance metrics to specific test data collected during the tests and then interpreting the results in view of the features of the system and the experimental setting. Depending on the specific use case, performance metrics can involve pure computer processing, human expertise by agronomists, or a combination of both.

Application of performance metrics may include the development of custom software to extract the necessary information from experimental data and/or to apply suitable processing to the information. If the performance metrics require the data to be subjected to some form of pre-processing, Service S00115 may be leveraged to prepare the data.

Beside the object of this service (i.e., evaluation of results), agrifoodTEF can, on request, support the customer along any other aspect and phase of the physical experimentation pipeline. For instance, for the design of the environment and protocol for the test, the customer can leverage services S00106 and S00107. Preparation of the test environment can be done either by the customer or by agrifoodTEF via service S00110; in both cases, assistance in interconnecting the system under test with agrifoodTEF’s test infrastructure is available via service S00111. Finally, data collection during the test can, if needed, be provided by agrifoodTEF via service S00113.