Evaluation of AI performance based on mixed testing environments

Evaluate your AI embedded in physical devices in the LNE facility designed to test systems in a semi-simulated environment. 

Overview

By having your AI system tested by the LNE, you ensure it meets the highest standards of safety and performance, boosting your product’s reliability and trustworthiness in the market. Partnering with the LNE for rigorous testing of your AI technologies and a detailed evaluation report used for demonstrating performance enhances credibility and opens doors to new market opportunities both locally and globally. This service proposes to assess a range of agricultural devices (with respect to the physical constraints of the testing bench) integrating AI, particularly those utilising vision processing, within our advanced hybrid testing environment. Our novel hybrid testing facility consists of placing devices in the heart of a simulation projected onto a 300° screen while a motion capture system and instrumented conveyor belt measure its movements, if any. These data are continuously sent to the simulator in real time so that the device's digital twin follows the same movements and the projected environment is updated accordingly. The simulator also incorporates a physics engine and advanced sensor models, enabling a virtual sensor output to be substituted for the sensors in real time in the cases where devices require specific data and/or if the assessment is orientated toward a special kind of sensor degradation. Typical agricultural products evaluated include mobile robots for autonomous weeding or harvesting that use visual navigation (based on 2D cameras) and intelligent cameras (with AI functionalities) for crop health monitoring or livestock tracking. Other sensors commonly used in agriculture, such as 3D cameras for yield estimation, GPS for autonomous vehicle control, Lidar for terrain mapping, and sonar for obstacle detection, are also supported through data injection from the simulator.

More about the service

This service can significantly benefit agricultural robotics and AI companies by providing a reliable and standardised way to assess the effectiveness of their AI systems.

Here’s how: Accurate AI Performance Assessment: Agricultural devices rely heavily on AI for tasks like crop detection, disease identification, or autonomous navigation. LNE's AI performance evaluation using a diverse representative sample of real agricultural data allows companies to precisely measure how well their AI models perform in real-world agricultural scenarios, improving decision-making and overall system effectiveness.

Improved Model Accuracy and Efficiency: By testing with diverse test benches, agricultural robotics companies can identify potential weaknesses or biases in their AI algorithms, ensuring more accurate detection, prediction, and classification. This leads to better outcomes in tasks like yield estimation, precision weeding, and pest management.

Safety and Reliability: Performance evaluations help ensure that AI systems in agricultural devices can handle complex environments with precision and consistency. This is crucial for ensuring the safety and reliability of systems when operating alongside humans, animals, and delicate crops. By leveraging this service, agricultural robotics and AI companies can improve the accuracy, reliability, and market readiness of their AI systems, ultimately offering more efficient and trustworthy solutions to their clients.

The service includes several technical meetings with AI and robotics evaluation experts to define the devices and the test bench settings, evaluation strategy and the testing procedure. The evaluation and testing will be conducted at LNE located in Trappes, France. Once the evaluation is complete, LNE provides a detailed report that includes both quantitative and qualitative analyses of the AI's strengths and areas for improvement. This report is invaluable for developers looking to refine their systems; potential investors or end-users looking for trusted third-party performance evaluation; and for businesses wanting to demonstrate compliance with industry standards and regulations in the future.

The test bench can be adapted to some type of physical devices, and the evaluation dataset, procedure and metrics are customised according to the customer needs of evaluation. The test bench can be adapted to simulate various agricultural terrains and lighting conditions.

Evaluation datasets can be customised with imagery of specific crop diseases or weed types relevant to the customer’s focus. Testing procedures can be tailored to assess performance in specific agricultural tasks like fruit picking or autonomous spraying, and performance metrics can be aligned with industry-specific benchmarks. The test bench can host robots up to 2 tonnes of weight and up to 3 square meters.
Location
France
Remote
Type of Sector
Arable farming
Food processing
Greenhouse
Horticulture
Livestock farming
Tree Crops
Viticulture
Type of service
Performance evaluation
Test design
Test execution
Test setup
Accepted type of products
Physical system
Software or AI model