Catalogue of Services

This service concerns training AI models on behalf of the customer for a specific task and optimisation objective, e.g., improving accuracy on crop classification from image data.

The target model is the solution provided by the customer that needs to be enhanced with respect to a set of predetermined features to reach the desired performance level. However, if required, the training can also be applied to additional state-of-the-art models available in the market for benchmarking purposes.

If not defined by the customer, some features of the training process can be identified via service S00179 (desk assessment activities for digital systems and/or data): for instance, model features to improve, reference model baselines to include in the performance comparison, as well as benchmark datasets.

The data used for training the model can be either provided by the customer or annotated ad hoc as a preparatory activity to model training (via service S00290 – Data labelling); another possibility is that data are retrieved among reference benchmark datasets that are openly available. We will also agree with customers on the level of hardware acceleration required, based on the considered AI models, e.g., GPU acceleration via connection to a remote server vs on-device training.

One of the key activities during digital testing is the collection of data concerning the progression and the final outcome of the tests. Such data enable the evaluation of system performance by the customer or – if needed – by agrifoodTEF (via Service S00184). This service manages the collection of data relevant to performance evaluation produced during the tests by both the system under test and the computational environment where the tests take place.

Examples of collected data comprise information produced within a virtual environment to simulate sensor data collection in a physical environment; statistics about AI model performance in the test and deployment phase (e.g., occupied memory, number of trainable parameters, training/optimisation loss, etc.); specific labels and annotations to use as ground truth for evaluating the system; and system output when subjected to a range of test conditions.  

The minimum set of data to be collected is defined by the evaluation metrics that the user chose (either on their own or with agrifoodTEF support, via Service S00178) to process them; generally, a larger set of data wrt the minimum is selected by agrifoodTEF together with the customer to provide a richer view of the system’s performance and to enable the application of other metrics in the future, if needed.

As an output of the service, in addition to the raw data, we also provide the customer with documentation describing logged features and conditions of the testing environment at the time of testing, as well as any parameter values, variation ranges and specifics required for reproducibility purposes.

Through this service, we offer dedicated technical systems (hardware and software) and expertise for the collection of relevant data during testing.

The scope of data collected can range from data generated by the AgrifoodTEF technical infrastructure (e.g., through the installation of IoT sensors) to the collection of data generated by the system under test to the generation of ground truth for evaluating the system. If necessary, service S00111 may be leveraged to take care of the interconnection between AgrifoodTEF data collection systems and the customer’s systems.

In addition to the raw collected data, we will also provide customers with complete documentation describing the logged features and conditions of the testing environment at the time of testing, as well as any parameter values, variation ranges and specifics required for reproducibility purposes.

Where necessary, data collection and documentation activities can require the involvement of field experts (e.g., engineers, agronomists).

Note: annotation and labelling of collected data, if needed, can be provided via service S00115.

This service concerns the curation of ground truth annotations for testing and experimentation data. More specifically, data labelling concerns the association of portions of data (e.g., image regions) to descriptions that provide information about the significance of those portions. In particular, labelling is required whenever the data are meant to be used for model training.

We also provide labelling services for domain-specific data attributes, such as disease indicators and their confounding factors, by experts in agronomy. The labelled data will be accompanied by a report synthesising quality metrics and statistics like the number of data points annotated, percentage of the full set covered for the annotations, number of annotators and the inter-annotator agreement.

Through this service, we offerers the ability to validate and augment the data collected during the different testing phases. Examples of operations that this service can provide include quality checks (e.g., presence of all datastreams, presence of drop-outs, technical quality such as focus or exposure of camera images, agronomic significance of data); preparation for performance evaluation (e.g., identification of the images containing selected visual markers or specific plants, association between images from the system under test and the ground truth); data processing (e.g., transcoding); and augmentation (e.g., generation and incorporation of metadata). Data labelling (e.g., to enable training of AI models) is not included as it is the object of a separate service (S00291).

Any test activity involves three main components, i.e.: environment (where the tests take place), protocol (defining what activities are executed and how) and evaluation metrics (used to assess the results of the tests). This service concerns the last element; its goal is to design the best metrics to evaluate the performance of digital systems such as (for instance) AI models or Computer Vision software. The digital environment and the testing protocol metrics can be designed -if required- via services S00176 and S00177. 

Our team will identify and define with customers the most adequate set of quantitative metrics to assess the outcome of the digital testing activities. In order to ensure the relevance of the metrics with respect to the real-world use cases, the team will involve engineers and agronomists.  

The metrics will be adapted not only to the task that the digital system under test (e.g., a piece of software) is designed to perform, but also to the features of the data used for the tests. For instance, a customer that has developed a machine incorporating an AI model will be interested in testing the model on data generated by their own machine: the performance metrics will therefore need to be adapted to the specific features of those data.  

Any physical test activity involves three main components: environment (where the tests take place), protocol (defining what tests are executed and how) and evaluation metrics (used to assess the results of the tests). This service concerns the last element; its goal is to design the best metrics to evaluate the performance of a customer solution, taking into consideration the use cases specified by the customer and the environment and protocol chosen for the tests (which, if needed, can be designed via services S00106 and S00107).

Our team will identify and define with customers the most adequate set of quantitative (i.e., based on instrumental measurements) and/or qualitative (i.e., relying on expert human judgement) metrics to assess the system functionalities of interest. This phase will involve, in particular, agronomists and experts in agricultural machinery.

Based on the defined evaluation metrics, a set of requirements for the collection of required data and ground truth annotations will also be defined accordingly. For instance, the service may lay out the specifications for dedicated data collection campaigns (possibly executed via service S00113). This phase will involve engineers and experts in AI and robotics.

On request, the output of the service will include analyses on additional environmental factors than those directly tracked through the designed metrics (e.g., seasonal effects, impact of test distribution over time on results).

Any test activity involves three main components, i.e., environment (where the tests take place), protocol (defining what activities are executed and how), and evaluation metrics (used to assess the results of the tests). This service concerns the second element, i.e., the design of the testing procedure for digital systems such as (for instance) AI models or computer vision software. The digital environment and the evaluation metrics can be designed – if required – via services S00176 and S00178.

In the context of testing customers’ solutions within digital environments, this service is targeted at designing a suitable protocol for digital testing based on the use cases specified by the customer. The components of the testing protocol can include:

• Selecting the datasets to be used for testing

• Selecting reference AI models to be used for testing (if needed)

• Choosing data formats and metadata standards 

• Defining data pre-processing and preparation steps 

• Defining values and ranges of test parameters

• Defining the different phases of the protocol

• Outlining the operations to be executed in each phase

• Thoroughly describing the protocol specifics to ensure reproducibility

Datasets to be used for testing can be provided by agrifoodTEF and/or by the customer; if nothing suitable is available, other agrifoodTEF services can be leveraged to collect and/or generate tailored data.

The technical team executing this service comprises expert engineers but can also involve agronomists when this is necessary to ensure the relevance of the tests for the use case, e.g., to determine the distribution of test repetitions across the variation ranges.

Through this service we offer customers a complete feasibility study of their digital systems and/or data, evaluating their suitability for the test use cases identified by the customer. The study involves both technical and agronomic aspects. The goal of this service is to provide the customer with a comparison between the features of the system or data submitted for review and the requirements of the use cases chosen for (possible) subsequent testing.

Examples of systems and data requiring digital testing include (but are not limited to) software modules for data processing (e.g., machine vision modules), AI models, software architectures (e.g., control system of a robot), simulations, datasets collected in the field, synthetic datasets, designs of software systems, and so on.

This assessment is crucial to provide customers with key information on elements of their system that need to be further developed to become “fit for testing”. It can also identify changes to apply to either the system or agrifoodTEF’s testing facilities to meet the requirements for experimentation.

This service concerns the processing of data collected during digital testing of a system, i.e., experimentation done in a computational environment. Digital testing enables easy collection of experimental data; however, such data then need to be suitably processed to extract information about system performance.

The service applies a predefined set of performance metrics (defined either by the customer or via service S00178) to the collected data. Data collection during test execution is expected to have been done by the customer (on request, this activity can be supported by agrifoodTEF via service S00183). The service includes all the activities needed to apply the performance metrics, including the development of any software component needed for data processing or data preparation.

The output of the service is a report providing the quantitative results of the application of the performance metrics to the experimental data. The report also includes an evaluation of these results aimed at highlighting possible issues and pointing out lines for improvement.

The AgrifoodTEF consortium comprises a set of testing and experimentation facilities and targeted services aimed at supporting customers in the testing and validation of systems and devices based on advanced technologies. Given the technical complexity of these facilities and services and the associated complexity of many activities making use of them, many customers can benefit from a preliminary training activity. The goal of such training is to make the best use of the facilities and services and fully exploit the allotted time when actual testing/experimentation occurs.

Customer training – involving both technical and procedural aspects – can either be sought by the customer autonomously leveraging the documentation provided by agrifoodTEF or directly supported by agrifoodTEF via this service; a combination of both approaches is also possible.

This service does not limit its applicability to customers that already know what they require from agrifoodTEF. The service also provides an opportunity for customers to discuss their specific needs and requirements with our team and be directed towards the set of services in the catalogue that best suit their needs. Especially where the customer envisages an articulate testing campaign, exploring their idea together with agrifoodTEF via this service can be very beneficial in planning subsequent activities.

This service can, if needed, be integrated with services S00109 (desk assessment activities for physical systems) or S00179 (desk assessment activities for digital systems or data) to anchor its operation, and possible further testing activities, more closely to the specific features of the system(s) developed by the customer.

To successfully conduct a digital testing campaign, preparatory activities are usually required to set up the computational environment used for testing. This service performs activities such as:

• setting up the hardware resources needed to run the tests

• configuring and initialising the virtual environments used for testing (e.g., Docker images, private “data rooms” for safe data sharing)

• installation and configuration of required software packages and dependencies 

• setting up authentication layers, user roles and credentials as needed

• migration and/or exchange of required data and ground truth annotations 

• installation and configuration of a simulator

• importing and configuring a previously defined simulated environment

• executing dry-runs to check that all elements of the test environment operate as required

Environment preparation is done according to an environment design provided by the customer. If needed, such a design can be done by agrifoodTEF for the customer via Service S00176.

Interested customers can get support from agrifoodTEF for the entire pipeline involved in digital testing, from the design of its other elements beside the environment (namely, the testing protocol via service S00177 and the evaluation metrics via service S00178) to test execution (service S00182), data collection (service S00183), and evaluation (S00184). Support in interconnecting the systems under test to the digital testing environment, if needed, is available via service S00181.