Acadia

An iPhone or iPad as the computer for autonomous driving.

Acadia runs three neural networks and a decision model on an iPhone or iPad from 2019. Together, they turn the camera image into steering, acceleration and braking.

Steering Prediction

A convolutional neural network, trained with behavioral cloning on recorded drives. It takes the camera image and predicts the steering angle.

Lane Segmentation

A network based on U-Net. It marks which pixels of the image belong to the lane.

Object Detection

MobileNet SSD and YOLO. They find and classify the objects in the image.

How It Works

All three networks run on each camera frame. The decision model combines their results. It sets steering, acceleration and braking.

In the App

Steering Angle

The steering network runs on every frame. The wheel on the left turns by the predicted angle.

Screen recording on iPhone.

In the App

Detected Objects

The detector returns a box and a class for each object. The app draws the boxes over the image.

Object detection on a Mac, on a street in San Francisco.

Testing

On the Road

The iPad on the dashboard uses its own camera. All models run on the device.

Object detection on an iPad during a drive.

From Prototype to Device

The models were trained in Python and converted to Core ML.

  1. Prototype

    Built and trained with TensorFlow and Keras. Tested on recorded drives with OpenCV.

  2. Training Data

    About 325 GB of custom and public datasets. Prepared with pandas, NumPy and SciPy.

  3. Core ML

    The models run with Core ML. Core Image and Metal prepare each frame on the GPU.

  4. Other Hardware

    The models also ran on an NVIDIA Jetson Nano and a Google Coral TPU.

An early prototype evaluating a night drive.
~325 GB
of training data
3
neural networks
2019
iPhone and iPad hardware

Tech Specs

Platform
iPhone and iPad from 2019; also runs on Mac
Other inference hardware
NVIDIA Jetson Nano, Google Coral TPU
Steering prediction
CNN, outputs 0–1 (steering angle in radians)
Lane segmentation
U-Net based encoder–decoder
Object detection
MobileNet SSD, YOLO
Decision model
Decides steering, acceleration and braking
Training data
About 325 GB from custom and public datasets
Training
TensorFlow, Keras, pandas, NumPy, SciPy, OpenCV
On the device
Core ML, Metal, Core Image
Acadia icon

About the Project

Acadia is a research project about running autonomous driving models on an iPhone or iPad.