Joint Representation Learning and Keypoint Detection for Cross-view Geo-localization

Authors: Jinliang Lin, Zhedong Zheng, Zhun Zhong, Zhiming Luo, Shaozi Li, Yi Yang, Nicu Sebe

Published in IEEE Transactions on Image Processing (TIP), 2022

Recommended citation: Jinliang Lin, Zhedong Zheng, Zhun Zhong, Zhiming Luo, Shaozi Li, Yi Yang, Nicu Sebe, "Joint Representation Learning and Keypoint Detection for Cross-view Geo-localization." IEEE Transactions on Image Processing, 2022. DOI: 10.1109/TIP.2022.3175601
Download PDF: https://zdzheng.xyz/files/2022/TIP_RKNet.pdf

Code is available at: https://github.com/AggMan96/RK-Net

Abstract: In this paper, we study the cross-view geo- localization problem to match images from different viewpoints. The key motivation underpinning this task is to learn a dis- criminative viewpoint-invariant visual representation. Inspired by the human visual system on mining local patterns, we propose a new framework called RK-Net, to jointly learn the discriminative Representation and detect salient Keypoints with a single Network. Specifically, we introduce a Unit Subtraction Attention Module (USAM) that can automatically discover repre- sentative keypoints from feature maps and draw attention to the salient regions. USAM contains very few learning parameters but yields significant performance improvement, and can be easily plugged into different networks. We demonstrate through extensive experiments that: (1) By incorporating USAM, RK- Net facilitates end-to-end joint learning without the prerequi- site of extra annotations. Representation learning and keypoint detection are two highly-related tasks. Representation learning helps keypoint detection. Keypoint detection, in turn, enriches the model capability against large appearance changes caused by viewpoint variants. (2) USAM is easy to implement and can be integrated with existing methods, further improving the state- of-the-art performance. We achieve competitive geo-localization accuracy on three challenging datasets, i.e., University-1652, CVUSA and CV ACT. Code is available at https://github.com/ AggMan96/RK-Net.

@article{lin2022joint,
author = "Lin, Jinliang and Zheng, Zhedong and Zhong, Zhun and Luo, Zhiming and Li, Shaozi and Yang, Yi and Sebe, Nicu",
title = "Joint Representation Learning and Keypoint Detection for Cross-view Geo-localization",
abstract = "In this paper, we study the cross-view geo- localization problem to match images from different viewpoints. The key motivation underpinning this task is to learn a dis- criminative viewpoint-invariant visual representation. Inspired by the human visual system on mining local patterns, we propose a new framework called RK-Net, to jointly learn the discriminative Representation and detect salient Keypoints with a single Network. Specifically, we introduce a Unit Subtraction Attention Module (USAM) that can automatically discover repre- sentative keypoints from feature maps and draw attention to the salient regions. USAM contains very few learning parameters but yields significant performance improvement, and can be easily plugged into different networks. We demonstrate through extensive experiments that: (1) By incorporating USAM, RK- Net facilitates end-to-end joint learning without the prerequi- site of extra annotations. Representation learning and keypoint detection are two highly-related tasks. Representation learning helps keypoint detection. Keypoint detection, in turn, enriches the model capability against large appearance changes caused by viewpoint variants. (2) USAM is easy to implement and can be integrated with existing methods, further improving the state- of-the-art performance. We achieve competitive geo-localization accuracy on three challenging datasets, i.e., University-1652, CVUSA and CV ACT. Code is available at https://github.com/ AggMan96/RK-Net.",
journal = "IEEE Transactions on Image Processing",
url = "https://zdzheng.xyz/files/2022/TIP\_RKNet.pdf",
code = "https://github.com/AggMan96/RK-Net",
doi = "10.1109/TIP.2022.3175601",
year = "2022" }