Catalogue of Services

Evaluating or verifying AI solutions is technically challenging and imposes specific requirements on the application, which can vary from setting internal benchmarks to establishing safety conditions. Evaluation and verification are performed using a broad set of tools ranging from straightforward performance evaluation metrics to formal methods for the verification of safety requirements. Formal methods can also be used to evaluate requirement fulfilment and systematically discover scenarios showing that AI systems need further improvements. Alternatively, customers can receive technical assistance and advice on improving AI systems that do not meet requirements.

Conformity assessment

ELSA assessment

Test execution

AI model training

This service concerns training AI models on behalf of the customer for a specific task and optimisation objective, e.g., improving accuracy on crop classification from image data. The target model is the solution provided by the customer that needs to be enhanced with respect to a set of pre-determined features to reach the desired performance level. However, if required, the training can also be applied to additional state-of-the-art models available in the market for benchmarking purposes. If not defined by the customer, some features of the training process can be identified via service S00179 (desk assessment activities for digital systems and/or data): for instance, model features to improve, reference model baselines to include in the performance comparison, as well as benchmark datasets. The data used for training the model can be either provided by the customer or annotated ad hoc as a preparatory activity to model training (via service S00290 - Data Labelling); another possibility is that data are retrieved among reference benchmark datasets that are openly available. We will also agree with customers on the level of hardware acceleration required, based on the considered AI models: e.g., GPU acceleration via connection to a remote server vs. on-device training.

Assessment and evaluation of applied natural language processing

This service explores the potential of implementing applied Natural Language Processing (NLP) systems, rigorously evaluating them to ensure high quality and performance while identifying and addressing any limitations. It includes developing improvements to resolve issues uncovered during evaluation and ensures the NLP systems remain up-to-date.

By integrating NLP technology, businesses can enhance customer interactions through automated yet personalized responses. NLP also enables the analysis of large volumes of text data to uncover trends, identify opportunities, and provide valuable insights.

In the context of agricultural AI and robotics, NLP can significantly improve decision-making, operational efficiency, and communication.

Assessment of AI performance and software optimisation on edge computing devices

Fondazione Bruno Kessler

The tests in this service are aimed at products that run complex apps (based on a multitude of microservices, including heavy AI algorithms) in the fields, with limited to no connectivity to the cloud. In these cases, we propose a benchmarking designed to understand what the bottlenecks are in the implemented solution architecture to help TEF customers improve the efficiency and reliability of such products/solutions. The service may also encompass virtual facility elements (enriched datasets, virtual machines hosting containerised modules) blended with physical sensors and infrastructure in our test field or directly installed at the client's premises if need be for the duration of the experimentation.

Test execution

Test setup

Assessment of conformity roadmap for an innovation

Are you ready to launch your innovative product with complete confidence in its compliance and market success? Our service offers a thorough assessment of all necessary steps to ensure your product meets the latest standards, laws, and directives. We will guide you through the regulatory landscape, identifying the specific rules applicable to your product and providing tailored advice on how to adapt your innovation to meet these requirements.

Unsure if current laws cover your groundbreaking idea? We will also investigate whether new regulations are needed to support your innovation's potential.

In addition to compliance, we provide a comprehensive range of supplementary services that AgrifoodTEF can offer to further accelerate your success. From testing and validation to expert guidance on market strategy, we are here to support you every step of the way.

Conformity assessment

Desk assessment

ELSA assessment

Collection of test data during physical testing

Through this service, we provide specialised technical systems (both hardware and software) and expertise for collecting relevant data during testing.

The scope of the data collected can vary, encompassing information generated by the agrifoodTEF technical infrastructure (e.g., through the installation of IoT sensors), data from the system under test, and ground truth data for system evaluation. If needed, this service can facilitate the interconnection between agrifoodTEF's data collection systems and the customer's systems.

In addition to the raw data collected, we will supply customers with comprehensive documentation detailing the logged features, conditions of the testing environment at the time of testing, and any parameter values, variation ranges, and specifics necessary for reproducibility.

When required, data collection and documentation activities may involve field experts such as engineers and agronomists.

Please note that annotation and labeling of collected data, if necessary, can be provided through service S00115.

Compliance assessment for robotic machines

Ensure your company develops robotic machinery that complies with all relevant regulations, standards, and directives with the support of expert guidance. Beyond meeting existing requirements, expert assistance is provided in areas where the rules are unclear or undefined. For situations with ambiguous or missing guidelines, practical solutions are offered based on extensive experience in functional safety.

Autonomous robotic machinery often relies on complex sensor technologies, such as radar, LIDAR, and cameras, to detect nearby individuals. Demonstrating how these advanced sensors can be integrated into safety functions is challenging, as it involves considering not only traditional functional safety aspects but also emerging factors like AI and Safety of the Intended Functionality (SOTIF).

Support includes performing comprehensive risk assessments, identifying relevant safety functions, and establishing high-level safety requirements for each. Depending on the project's focus, further guidance can be provided to ensure the selected safety concept aligns with applicable functional safety standards.

Desk assessment

ELSA assessment

Data labelling

RISE - Research Institutes of Sweden

This service concerns the curation of ground truth annotations for testing and experimentation data. More specifically, data labelling concerns the association of portions of data (e.g., image regions) to descriptions that provide information about the significance of those portions. In particular, labelling is required whenever the data are meant to be used for model training. We also provide labelling services for domain-specific data attributes, such as disease indicators and their confounding factors, by experts in agronomy. The labelled data will be accompanied by a report synthesising quality metrics and statistics like the number of data points annotated, the percentage of the full set covered for the annotations, the number of annotators and the inter-annotator agreement.

Data augmentation

Data pipeline evaluation for AI applications

The purpose of the service is to ensure that data protocols and pipelines are adequate to the AI application developed by the customer. The service can be delivered at different stages of product maturity to ensure that it can provide targeted testing when it is most useful. During earlier product development stages, the service concentrates on data collection, quality and storage protocols, whilst, at later stages, attention can be extended to scalability, access and control mechanisms.

Data analysis

Test execution

Test setup

Design of computational environments for digital testing

Test activities involve three key components: the environment (where the tests are conducted), the protocol (defining what tests are performed and how), and evaluation metrics (used to assess the test results). Through this service, we assist customers in designing a computational environment tailored to running the digital tests required for their software components. Specifically, this service:

Defines the data (and metadata) requirements needed to support the test
Identifies the necessary software environment (e.g., operating system, libraries)
Selects an appropriate simulator and defines simulated environments, if required
Chooses remote access tools, if applicable.

As a result, we deliver a comprehensive design of the testing environment, including specifications such as memory requirements, GPU/TPU acceleration, and necessary libraries. This environment can be deployed on either the agrifoodTEF digital infrastructure or the customer's computing infrastructure (i.e., on premises).

Design of evaluation metrics for digital testing

Any test activity involves three main components, i.e., environment (where the tests take place), protocol (defining what activities are executed and how), and evaluation metrics (used to assess the results of the tests). This service concerns the last element; its goal is to design the best metrics to evaluate the performance of digital systems such as (for instance) AI models or computer vision software. The digital environment and the testing protocol metrics can be designed—if required—via services S00176 and S00177. Our team will identify and define with customers the most adequate set of quantitative metrics to assess the outcome of the digital testing activities. In order to ensure the relevance of the metrics with respect to the real-world use cases, the team will involve engineers and agronomists. The metrics will be adapted not only to the task that the digital system under test (e.g., a piece of software) is designed to perform, but also to the features of the data used for the tests. For instance, a customer that has developed a machine incorporating an AI model will be interested in testing the model on data generated by their own machine: the performance metrics will therefore need to be adapted to the specific features of that data.

Design of evaluation metrics for physical testing

Any physical test activity involves three main components: environment (where the tests take place), protocol (defining what tests are executed and how) and evaluation metrics (used to assess the results of the tests). This service concerns the last element; its goal is to design the best metrics to evaluate the performance of a customer solution taking into consideration the use cases specified by the customer and the environment and protocol chosen for the tests (which, if needed, can be designed via services S00106 and S00107). Our team will identify and define with customers the most adequate set of quantitative (i.e., based on instrumental measurements) and/or qualitative (i.e., relying on expert human judgement) metrics to assess the system functionalities of interest. This phase will involve, in particular, agronomists and experts in agricultural machinery. Based on the defined evaluation metrics, a set of requirements for the collection of required data and ground truth annotations will also be defined accordingly. For instance, the service may lay out the specifications for dedicated data collection campaigns (possibly executed via service S00113). This phase will involve engineers and experts in AI and robotics. On request, the output of the service will include analyses on additional environmental factors than those directly tracked through the designed metrics (e.g., seasonal effects, impact of test distribution over time on results).