Case study · T-Systems InnovationCenter · 2021ColorizeBringing black-and-white photographs back to life with deep learning.
This project was done as part of my apprenticeship at Deutsche Telekom AG
Built with
ArchitectureColorize is built on a Generative Adversarial Network — two convolutional neural networks that improve by correcting each other during unsupervised training.The generator's encoder-decoder architecture takes grayscale input and predicts the full RGB colour image. Initially, it used a U-Net style structure with self-attention mechanisms to capture spatial relationships in the images.A later iteration introduced ResNet-101 as the generator's backbone, significantly improving class-specific colorization by learning typical colors for different object categories.The discriminator rates how believable each predicted image looks. It was pre-trained on a small training set to mitigate early exhaustion while the generator was still finding its feet.
Architecture overview showing the generator and discriminator layers.
Architecture overview showing the generator and discriminator layers.Training EnvironmentColorize was trained on two NVIDIA Tesla GPUs alongside a performant IBM PowerPC machine with 150 CPUs and half a terabyte of working memory.Because of the PowerPC processor architecture, various patches had to be made across the TensorFlow and OpenCV libraries before compilation would even succeed — a substantial part of getting training off the ground.
150 CPUsIBM PowerPC processors
500GB memoryHalf a terabyte of working memory
2x NVIDIA TeslaGPU acceleration for training
MonitoringA full training run — first pre-training the discriminator, then training the generator and discriminator together, unsupervised — took roughly 48 hours on 5 million images from Google's OpenImages dataset.Various optimizations balanced keeping training data in memory for fast access against overall memory use, filling available memory as much as possible without risking memory pressure. Frequent checkpoints persisted the current weights to prevent data loss mid-run, and TensorBoard was used to watch the run over time, with testing checkpoints along the way to catch model failures early.
Grayscale inputs beside the model's predictions at successive training checkpoints.
Grayscale inputs beside the model's predictions at successive training checkpoints.ResultsThe final model produces natural-looking colourisations with only minor artifacts. Thanks to the training data and the upgraded ResNet-101 backbone, it generalises best on nature, landscapes and portraits, with significantly improved class-specific colorization.Images of everyday objects are more likely to fragment, picking up a red, blue or yellow colour bias, a limitation that traces back to the dataset rather than the architecture.To date many thousand images have been colorized successfully already.
Destitute pea pickers in California. Mother of seven children.
Group of Florida migrants on their way to Cranberry, New Jersey, to pick potatoes. Near Shawboro, North Carolina.
Manhattan Bridge under construction, New York City, 1909.
Destitute pea pickers in California. Mother of seven children.Group of Florida migrants on their way to Cranberry, New Jersey, to pick potatoes. Near Shawboro, North Carolina.Manhattan Bridge under construction, New York City, 1909.