ADAS field operations testing: Four essential considerations for program success

In the dynamic world of automotive technology, the integration of advanced driver assistance systems (ADAS) and artificial intelligence (AI) are revolutionizing vehicle safety, efficiency and autonomy. Using a network of cameras, radar and sensors, these systems create a real-time picture of the surroundings, enabling vehicles to react swiftly to potential dangers.

Core ADAS features such as automatic emergency braking (AEB), adaptive cruise control (ACC), lane departure warning (LDW) and blind spot detection (BSD) are becoming a requirement in the modern production of vehicles. Given their impact on the safety of humans, it's crucial that every manufacturer prioritizes the reliability and regulatory compliance of these systems.

This is done through a robust field operational testing (FOT) program, with training data specifically created for ADAS model testing and validation. Since this process involves technical and operational investments that are outside of the strategic business focus of most manufacturers, many choose to partner with external solution providers capable of managing end-to-end FOT projects. This includes everything from sourcing vehicles and qualifying drivers, to planning routes and calibrating sensors, as well as specialized technical support for collecting, labeling and validating the data.

An FOT data collection program is a massive undertaking with many integral components. Read on to explore the processes involved in such a program, the importance of sourcing high-quality data and some key points to consider when selecting an optimal FOT data solution to enhance your ADAS.

Beyond the drive: Considerations for top-tier FOT data collection programs

Embarking on a comprehensive FOT data collection program for ADAS and autonomous vehicles includes a number of considerations. Some of these include sourcing different vehicle types, fitting those vehicles with sensors, planning optimal routes with specified conditions and hiring experienced engineers, drivers and technical staff.

Beyond these practical considerations are some key criteria that often go overlooked when commencing this type of project. These include data bias concerns, data privacy considerations, the complexities involved in storing and managing such massive amounts of data and the expertise and tools needed to efficiently label the data.

Let's look at these four critical components in detail.

1. Data bias

One of the biggest challenges in FOT data collection lies in ensuring the data collected accurately reflects real-world driving scenarios. Insufficient or biased data can lead to misinterpretations of cause and effect. As a result, your advanced driver assistance systems might struggle to adapt and perform safely in unforeseen driving situations.

One of the critical ways to combat bias in ADAS is to ensure that FOT datasets encompass diverse driving conditions, including various weather, lighting and road surfaces. For example, one of our client's projects required leveraging our expertise to hire multiple drivers, one acting as a driver and one as a co-driver, to drive in turns during both day and night, in various cars. Using our proprietary software, we optimized route planning to cover a wide range of cities and scenarios of interest in more than 30 countries — including Taiwan, Japan, the United States, South Korea and several countries in the European Union (EU) — with daily data collection schedules. The end result was high-quality data that was representative of a wide breadth of real-world scenarios.

For driver monitoring systems (DMS) and occupant monitoring systems (OMS), FOT data collection also includes monitoring the vehicle's interior, driver and passengers, as well as collecting in-cabin audio data for car assistants. To mitigate dataset bias, it's crucial that the vehicle's occupants represent diverse demographics. For example, to develop a driver monitoring system that accounted for diversity, we crowdsourced 400 global participants from our AI Community who satisfied demographic variances such as skin tone, physicality, ethnicity, age and gender for driver data collection. This eliminated the need to ship the vehicle and recording systems out of the country and provided data to develop gesture recognition and gaze-detection models.

2. Balancing data utility and privacy

FOT data includes potentially sensitive information such as personally identifiable information (PII), as well as information about drivers' destinations, routes and locations near their homes. Such video data, GPS information and vehicle data can pose significant privacy risks. Further, the data can often be subject to regulatory compliance. For example, FOT data often includes video footage of other road users, including pedestrians, cyclists and passengers who haven't consented to being recorded. Under Article 4 of the EU's General Data Protection Regulation (GDPR), images showing faces and license plates are classified as personal data, making it critical to ensure collected data is well-protected and inaccessible to unauthorized parties.

Best practices for ensuring privacy protection and compliance with regulations such as the GDPR include implementing robust anonymization techniques, including:

• Data masking, PII scrubbing and data aggregation to obscure personal identifiers.
• Blurring faces and license plates in video footage to protect non-consenting individuals' identities.
• Using end-to-end encryption for data storage and transmission and implementing strict access control mechanisms, including multi-factor authentication and access log audits.
• Optimizing data collection by selectively gathering only essential data for ADAS functionality and storing it securely.

3. Meeting the computational and storage demands

Computational demands of real-world driving data collection present an operational challenge. Take for example the fact that a single vehicle equipped with six high-definition cameras and radar systems generates over 2 terabytes of data per hour. To effectively manage this massive and unpredictable influx of data, you need to implement storage systems like Azure data disks, external solid-state drives, AWS Snowball devices and cloud storage solutions. These storage systems can also scale efficiently with the increasing data volume and complexity.

Recently, we led a project mapping the city of Los Angeles using a vehicle moving at a speed below 20 miles per hour with minimal route repetition and a 50:50 ratio of night to day. To achieve this, we developed proprietary route planning software to avoid city route repetition and optimize the data collected to reduce computational overheads. Our enterprise-grade relationship with cloud storage providers helped us deliver one-week vehicle-to-cloud data transfer with a daily capture of 11 terabytes.

Further, a network of high-performance servers and storage is necessary to house the preprocessed and enriched data, which includes information on drivers, trips, situations encountered and performance indicators. Driving data collected in these projects usually amasses between 10-30 terabytes per day, running into hundreds of terabytes per week and reaching petabytes within a quarter. To adequately collect, store and transfer this data, the infrastructure requires careful orchestration, including the ability to schedule and manage all resources (e.g., storage, computing power and workforce) in a centralized location to maximize throughput and productivity. Additionally, regular technical maintenance tasks like data backups, software updates and hardware inspections are vital to prevent system failures and ensure smooth operation.

Recently, we collected roughly 125,000 miles of data across 30 countries, with a daily throughput of up to 7 terabytes per vehicle. Due to vast amounts of sensor data, video feeds and diagnostics generated by collection vehicles, the customer required robust data storage infrastructure and efficient data management techniques. To address this, we used Microsoft Azure to design a data storage and delivery solution. Leveraging Azure's capabilities, we implemented efficient data transfer and logging processes, ensuring seamless data management.

4. Transforming raw data into actionable insights

Ultimately, the raw data sourced during FOT data collection is just the beginning. This data needs preprocessing, which involves tasks like filtering, cleaning and enriching it with additional contextual data such as weather, traffic conditions and road classification.

The data labeling task for FOT involves a sophisticated and time-consuming process to extract and annotate relevant information about driving situations from the vast amount of data from various sensors (e.g., cameras, lidar and radar). It also requires a specialized workforce with the right skill set, including knowledge of driving rules and scenarios, attention to detail, good visual perception and the ability to follow complex guidelines. To increase the efficiency of this process, we leverage automated labeling from our industry-leading Ground Truth Studio (GTS) platform. With built-in project and user management tools, AI-enabled annotations and prelabeling and configurable workflows, GTS supports multiple data types such as image, video and 3D sensor fusion. Further, our data labeling work is carried out by thousands of annotators who continue to support ambitious labeling pipelines that drive global ADAS, autonomous vehicle and in-cabin technologies.

Leverage the benefits of working with a long-term ADAS data services partner

High-quality, vetted data is crucial to train and test your computer vision systems for navigating the complexities of the real world. Working with a partner that offers a comprehensive, fully-managed solution encompassing all of your automotive computer vision data collection and labeling needs can help. Such a partner can assist with everything from project management to scaling the workforce you need for collection and annotation to providing that workforce with efficient labeling tools.

Our global reach, supported by a robust partner network, enables us to conduct diverse and comprehensive data collection worldwide. To help reduce real-world testing costs and accelerate your ADAS development, we also offer a comprehensive library of curated and fully managed FOT datasets.

Leverage our team of AI and FOT specialists to design and execute robust automotive data collection programs that will help you build the future of safe and secure autonomous driving.

Contact our team of experts today to discuss your next ADAS enhancement project.