Unsupervised Eyeglasses Removal in the Wild

Authors: Bingwen Hu, Zhedong Zheng, Ping Liu, Wankou Yang, Mingwu Ren

Published in IEEE Transactions on Cybernetics, 2020

Recommended citation: Bingwen Hu, Zhedong Zheng, Ping Liu, Wankou Yang, Mingwu Ren, "Unsupervised Eyeglasses Removal in the Wild." IEEE Transactions on Cybernetics, 2020. DOI: 10.1109/tcyb.2020.2995496
Download PDF: https://zdzheng.xyz/files/Hu_CYB20.pdf

Code is available at: https://github.com/Bingwen-Hu/ERGAN-Pytorch

Abstract: Eyeglasses removal is challenging in removing different kinds of eyeglasses, e.g., rimless glasses, full-rim glasses and sunglasses, and recovering appropriate eyes. Due to the large visual variants, the conventional methods lack scalability. Most existing works focus on the frontal face images in the controlled environment, such as the laboratory, and need to design specific systems for different eyeglass types. To address the limitation, we propose a unified eyeglass removal model called Eyeglasses Removal Generative Adversarial Network (ERGAN), which could handle different types of glasses in the wild. The proposed method does not depend on the dense annotation of eyeglasses location but benefits from the large-scale face images with weak annotations. Specifically, we study the two relevant tasks simultaneously, i.e., removing and wearing eyeglasses. Given two facial images with and without eyeglasses, the proposed model learns to swap the eye area in two faces. The generation mechanism focuses on the eye area and invades the difficulty of generating a new face. In the experiment, we show the proposed method achieves a competitive removal quality in terms of realism and diversity. Furthermore, we evaluate ERGAN on several subsequent tasks, such as face verification and facial expression recognition. The experiment shows that our method could serve as a pre-processing method for these tasks.

@article{Hu_2020,
author = "Hu, Bingwen and Zheng, Zhedong and Liu, Ping and Yang, Wankou and Ren, Mingwu",
doi = "10.1109/tcyb.2020.2995496",
year = "2020",
publisher = "IEEE",
pages = "1--13",
title = "Unsupervised Eyeglasses Removal in the Wild",
code = "https://github.com/Bingwen-Hu/ERGAN-Pytorch",
url = "https://zdzheng.xyz/files/Hu\_CYB20.pdf",
journal = "{IEEE} Transactions on Cybernetics",
abs = "Eyeglasses removal is challenging in removing different kinds of eyeglasses, e.g., rimless glasses, full-rim glasses and sunglasses, and recovering appropriate eyes. Due to the large visual variants, the conventional methods lack scalability. Most existing works focus on the frontal face images in the controlled environment, such as the laboratory, and need to design specific systems for different eyeglass types. To address the limitation, we propose a unified eyeglass removal model called Eyeglasses Removal Generative Adversarial Network (ERGAN), which could handle different types of glasses in the wild. The proposed method does not depend on the dense annotation of eyeglasses location but benefits from the large-scale face images with weak annotations. Specifically, we study the two relevant tasks simultaneously, i.e., removing and wearing eyeglasses. Given two facial images with and without eyeglasses, the proposed model learns to swap the eye area in two faces. The generation mechanism focuses on the eye area and invades the difficulty of generating a new face. In the experiment, we show the proposed method achieves a competitive removal quality in terms of realism and diversity. Furthermore, we evaluate ERGAN on several subsequent tasks, such as face verification and facial expression recognition. The experiment shows that our method could serve as a pre-processing method for these tasks." }