Acadia
An iPhone or iPad as the computer for autonomous driving.
Acadia runs three neural networks and a decision model on an iPhone or iPad from 2019. Together, they turn the camera image into steering, acceleration and braking.
Steering Prediction
A convolutional neural network, trained with behavioral cloning on recorded drives. It takes the camera image and predicts the steering angle.
Lane Segmentation
A network based on U-Net. It marks which pixels of the image belong to the lane.
Object Detection
MobileNet SSD and YOLO. They find and classify the objects in the image.
How It Works
All three networks run on each camera frame. The decision model combines their results. It sets steering, acceleration and braking.
In the App
Steering Angle
The steering network runs on every frame. The wheel on the left turns by the predicted angle.
In the App
Detected Objects
The detector returns a box and a class for each object. The app draws the boxes over the image.
Testing
On the Road
The iPad on the dashboard uses its own camera. All models run on the device.
From Prototype to Device
The models were trained in Python and converted to Core ML.
-
Prototype
Built and trained with TensorFlow and Keras. Tested on recorded drives with OpenCV.
-
Training Data
About 325 GB of custom and public datasets. Prepared with pandas, NumPy and SciPy.
-
Core ML
The models run with Core ML. Core Image and Metal prepare each frame on the GPU.
-
Other Hardware
The models also ran on an NVIDIA Jetson Nano and a Google Coral TPU.
- ~325 GB
- of training data
- 3
- neural networks
- 2019
- iPhone and iPad hardware
Tech Specs
- Platform
- iPhone and iPad from 2019; also runs on Mac
- Other inference hardware
- NVIDIA Jetson Nano, Google Coral TPU
- Steering prediction
- CNN, outputs 0–1 (steering angle in radians)
- Lane segmentation
- U-Net based encoder–decoder
- Object detection
- MobileNet SSD, YOLO
- Decision model
- Decides steering, acceleration and braking
- Training data
- About 325 GB from custom and public datasets
- Training
- TensorFlow, Keras, pandas, NumPy, SciPy, OpenCV
- On the device
- Core ML, Metal, Core Image
About the Project
Acadia is a research project about running autonomous driving models on an iPhone or iPad.