ISSN 0253-2778

CN 34-1054/N

open
Free to Publish. Free to Read.
Open AccessOpen Access JUSTC Information Science and Technology 17 June 2024

A feature transfer model with Mixup and contrastive loss in domain generalization

  • Department of Statistics and Finance, School of Management, University of Science and Technology of China, Hefei 230026, China

Cite this:
https://doi.org/10.52396/JUSTC-2023-0010
More Information
  • Author Bio:

    Yuesong Wang is currently a graduate student under the tutelage of Prof. Hong Zhang at the University of Science and Technology of China. His research interests focus on machine learning

    Hong Zhang is a Full Professor with the University of Science and Technology of China (USTC). He received his Bachelor’s degree in Mathematics and Ph.D. degree in Statistics from USTC in 1997 and 2003, respectively. His major research interests include statistical genetics, causal inference, and machine learning

  • Corresponding author: E-mail: zhangh@ustc.edu.cn
  • Received Date: 20 January 2023
  • Accepted Date: 04 May 2023
    • Available Online: 17 June 2024
    Abstract

  • Abstract

    When domains, which represent underlying data distributions, differ between training and test datasets, traditional deep neural networks suffer from a substantial drop in their performance. Domain generalization methods aim to boost generalizability on an unseen target domain by using only training data from source domains. Mainstream domain generalization algorithms usually make modifications on some popular feature extraction networks such as ResNet, or add more complex parameter modules after the feature extraction networks. Popular feature extraction networks are usually well pre-trained on large-scale datasets, so they have strong feature extraction abilities, while modifications can weaken such abilities. Adding more complex parameter modules results in a deeper network and is much more computationally demanding. In this paper, we propose a novel feature transfer model based on popular feature extraction networks in domain generalization, without making any changes or adding any module. The generalizability of this feature transfer model is boosted by incorporating a contrastive loss and a data augmentation strategy (i.e., Mixup), and a new sample selection strategy is proposed to coordinate Mixup and contrastive loss. Experiments on the benchmarks PACS and Domainnet demonstrate the superiority of our proposed method against conventional domain generalization methods.

    Graphical abstract

    The proposed model with feature transfer and contrastive loss.

    Abstract

    When domains, which represent underlying data distributions, differ between training and test datasets, traditional deep neural networks suffer from a substantial drop in their performance. Domain generalization methods aim to boost generalizability on an unseen target domain by using only training data from source domains. Mainstream domain generalization algorithms usually make modifications on some popular feature extraction networks such as ResNet, or add more complex parameter modules after the feature extraction networks. Popular feature extraction networks are usually well pre-trained on large-scale datasets, so they have strong feature extraction abilities, while modifications can weaken such abilities. Adding more complex parameter modules results in a deeper network and is much more computationally demanding. In this paper, we propose a novel feature transfer model based on popular feature extraction networks in domain generalization, without making any changes or adding any module. The generalizability of this feature transfer model is boosted by incorporating a contrastive loss and a data augmentation strategy (i.e., Mixup), and a new sample selection strategy is proposed to coordinate Mixup and contrastive loss. Experiments on the benchmarks PACS and Domainnet demonstrate the superiority of our proposed method against conventional domain generalization methods.

    • Public Summary

    • We propose a feature transfer model for domain generalization and a new sampling strategy based on Mixup in cooperation with contrastive loss.
    • Experiments on two mainstream datasets demonstrate the superiority of our method.

    FullText(HTML)
  • loading
    • References(38)
  • [1]
    Shao R, Lan X, Li J, et al. Multi-adversarial discriminative deep domain generalization for face presentation attack detection. In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Long Beach, CA, USA: IEEE, 2020 : 10015–10023.
    [2]
    Ouyang C, Chen C, Li S, et al. Causality-inspired single-source domain generalization for medical image segmentation. IEEE Transactions on Medical Imaging, 2023, 42: 1095–1106. doi: 10.1109/TMI.2022.3224067
    [3]
    Guo L L, Pfohl S R, Fries J, et al. Evaluation of domain generalization and adaptation on improving model robustness to temporal dataset shift in clinical medicine. Scientific Reports, 2022, 12: 2726. doi: 10.1038/s41598-022-06484-1
    [4]
    Motiian S, Piccirilli M, Adjeroh D A, et al. Unified deep supervised domain adaptation and generalization. In: 2017 IEEE International Conference on Computer Vision (ICCV). Venice, Italy: IEEE, 2017 : 5716–5726.
    [5]
    Li H, Pan S J, Wang S, et al. Domain generalization with adversarial feature learning. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City, UT, USA: IEEE, 2018 : 5400–5409.
    [6]
    Li D, Yang Y, Song Y Z, et al. Learning to generalize: Meta-learning for domain generalization. Proceedings of the AAAI Conference on Artificial Intelligence, 2018, 32 (1). doi: 10.1609/aaai.v32i1.11596
    [7]
    Mancini M, Bulò S R, Caputo B, et al. Robust place categorization with deep domain generalization. IEEE Robotics and Automation Letters, 2018, 3: 2093–2100. doi: 10.1109/LRA.2018.2809700
    [8]
    Mancini M, Bulò S R, Caputo B, et al. Best sources forward: Domain generalization through source-specific nets. In: 2018 25th IEEE International Conference on Image Processing (ICIP). Athens, Greece: IEEE, 2018 : 1353–1357.
    [9]
    Segu M, Tonioni A, Tombari F. Batch normalization embeddings for deep domain generalization. Pattern Recognition, 2023, 135: 109115. doi: 10.1016/j.patcog.2022.109115
    [10]
    Zhang H, Cisse M, Dauphin Y N, et al. Mixup: Beyond empirical risk minimization. arXiv: 1710.09412, 2017.
    [11]
    Ding Z, Fu Y. Deep domain generalization with structured low-rank constraint. IEEE Transactions on Image Processing, 2018, 27: 304–313. doi: 10.1109/TIP.2017.2758199
    [12]
    Nalisnick E, Matsukawa A, Teh Y W, et al. Do deep generative models know what they don’t know? arXiv: 1810.09136, 2019 .
    [13]
    Li Y, Tian X, Gong M, et al. Deep domain generalization via conditional invariant adversarial networks. In: Ferrari V, Hebert M, Sminchisescu C, editors. Computer Vision–ECCV 2018. Cham: Springer, 2018 , 11219: 647–663.
    [14]
    Long M, Cao Y, Wang J, et al. Learning transferable features with deep adaptation networks. In: Proceedings of the 32nd International Conference on Machine Learning. New York: ACM, 2015 , 37: 97–105.
    [15]
    Ganin Y, Lempitsky V. Unsupervised domain adaptation by backpropagation. In: Proceedings of the 32nd International Conference on Machine Learning. New York: ACM, 2015 , 37: 1180–1189.
    [16]
    Bousmalis K, Silberman N, Dohan D, et al. Unsupervised pixel-level domain adaptation with generative adversarial networks. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Honolulu, HI, USA: IEEE, 2017 : 95–104.
    [17]
    Xu R, Chen Z, Zuo W, et al. Deep cocktail network: Multi-source unsupervised domain adaptation with category shift. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City, UT, USA: IEEE, 2018 : 3964–3973.
    [18]
    Zhou K, Yang Y, Hospedales T, et al. Learning to generate novel domains for domain generalization. In: Vedaldi A, Bischof H, Brox T, editors. Computer Vision–ECCV 2020. Cham: Springer, 2020 , 12361: 561–578.
    [19]
    Bai H, Sun R, Hong L, et al. DecAug: Out-of-distribution generalization via decomposed feature representation and semantic augmentation. Proceedings of the AAAI Conference on Artificial Intelligence, 2021, 35: 6705–6713. doi: 10.1609/aaai.v35i8.16829
    [20]
    Chattopadhyay P, Balaji Y, Hoffman J. Learning to balance specificity and invariance for in and out of domain generalization. In: European Conference on Computer Vision. Cham: Springer, 2020 : 301–318.
    [21]
    Peng X, Bai Q, Xia X, et al. Moment matching for multi-source domain adaptation. In: 2019 IEEE/CVF International Conference on Computer Vision (ICCV). Seoul, Korea (South): IEEE, 2020 : 1406–1415.
    [22]
    Li D, Zhang J, Yang Y, et al. Episodic training for domain generalization. In: 2019 IEEE/CVF International Conference on Computer Vision (ICCV). Seoul, Korea (South): IEEE, 2020 : 1446–1455.
    [23]
    S Seo, Y Suh, D Kim, G Kim, J Han, and B Han. Learning to optimize domain specific normalization for domain generalization. In: Vedaldi A, Bischof H, Brox T, et al. editors. Computer Vision─ECCV 2020. Cham: Springer, 2020 : 68–83.
    [24]
    K Zhou, Y Yang, Y Qiao, et al. Domain generalization with mixstyle. arXiv: 2104.02008, 2021 .
    [25]
    Cai Q, Wang Y, Pan Y, et al. Joint contrastive learning with infinite possibilities. In: Proceedings of the 34th International Conference on Neural Information Processing Systems. New York: ACM, 2020 : 12638–12648.
    [26]
    Chen T, Kornblith S, Norouzi M, et al. A simple framework for contrastive learning of visual representations. In: Proceedings of the 37th International Conference on Machine Learning. New York: ACM, 2020 : 1597–1607.
    [27]
    Mitrovic J, McWilliams B, Walker J, et al. Representation learning via invariant causal mechanisms. arXiv: 2010.07922, 2020 .
    [28]
    He K, Fan H, Wu Y, et al. Momentum contrast for unsupervised visual representation learning. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Seattle, WA, USA: IEEE, 2020 : 9726–9735.
    [29]
    Wu Z, Xiong Y, Yu S X, et al. Unsupervised feature learning via non-parametric instance discrimination. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City, UT, USA: IEEE, 2018 : 3733–3742.
    [30]
    Yao T, Zhang Y, Qiu Z, et al. SeCo: Exploring sequence supervision for unsupervised representation learning. Proceedings of the AAAI Conference on Artificial Intelligence, 2021, 35: 10656–10664. doi: 10.1609/aaai.v35i12.17274
    [31]
    Li D, Yang Y, Song Y Z, et al. Deeper, broader and artier domain generalization. In: 2017 IEEE International Conference on Computer Vision (ICCV). Venice, Italy: IEEE, 2017 : 5543–5551.
    [32]
    Carlucci F M, D'Innocente A, Bucci S, et al. Domain generalization by solving jigsaw puzzles. In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Long Beach, CA, USA: IEEE, 2020 : 2224–2233.
    [33]
    Matsuura T, Harada T. Domain generalization using a mixture of multiple latent domains. Proceedings of the AAAI Conference on Artificial Intelligence, 2020, 34: 11749–11756. doi: 10.1609/aaai.v34i07.6846
    [34]
    Mahajan D, Tople S, Sharma A. Domain generalization using causal matching. In: Proceedings of the 38th International Conference on Machine Learning. Volume 139 of Proceedings of Machine Learning Research. PMLR, 2021 , 139: 7313–7324.
    [35]
    Nam H, Lee H, Park J, et al. Reducing domain gap by reducing style bias. arXiv: 1910.11645, 2019.
    [36]
    Zhou K, Yang Y, Hospedales T, et al. Deep domain-adversarial image generation for domain generalisation. Proceedings of the AAAI Conference on Artificial Intelligence, 2020, 34: 13025–13032. doi: 10.1609/aaai.v34i07.7003
    [37]
    Balaji Y, Sankaranarayanan S, Chellappa R. MetaReg: towards domain generalization using meta-regularization. In: Proceedings of the 32nd International Conference on Neural Information Processing Systems. New York: ACM, 2018 : 1006–1016.
    [38]
    Chattopadhyay P, Balaji Y, Hoffman J. Learning to balance specificity and invariance for in and out of domain generalization. In: Vedaldi A, Bischof H, Brox T, editors. Computer Vision–ECCV 2020. Cham: Springer, 2020 : 301–318.
    • Supplements(0)
    • Related Articles
    • Track Citations
  • 加载中

Catalog

    |
    Get Citation
    PDF
    XML

    Figure  1.  Illustration of our proposed method.

    Figure  2.  A causal model underlying our proposed method. $ y $: label variable; $ f_y $: label feature; $ d $: domain variable; $ f_d $: domain feature; $ z $: hidden variable; $ f_z $: hidden feature; $ x $: sample; $ f_x $: sample feature; $ f(x) $: domain invariable feature (extracted from CNN); $ \hat d $: new domain variable; $ f_{\hat d} $: new domain feature; $ f_{x,\hat d} $: new feature generated by $ f(x) $ and $ f_{\hat d} $. Solid arrow: causal relationship; dotted arrow: prediction.

    [1]
    Shao R, Lan X, Li J, et al. Multi-adversarial discriminative deep domain generalization for face presentation attack detection. In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Long Beach, CA, USA: IEEE, 2020 : 10015–10023.
    [2]
    Ouyang C, Chen C, Li S, et al. Causality-inspired single-source domain generalization for medical image segmentation. IEEE Transactions on Medical Imaging, 2023, 42: 1095–1106. doi: 10.1109/TMI.2022.3224067
    [3]
    Guo L L, Pfohl S R, Fries J, et al. Evaluation of domain generalization and adaptation on improving model robustness to temporal dataset shift in clinical medicine. Scientific Reports, 2022, 12: 2726. doi: 10.1038/s41598-022-06484-1
    [4]
    Motiian S, Piccirilli M, Adjeroh D A, et al. Unified deep supervised domain adaptation and generalization. In: 2017 IEEE International Conference on Computer Vision (ICCV). Venice, Italy: IEEE, 2017 : 5716–5726.
    [5]
    Li H, Pan S J, Wang S, et al. Domain generalization with adversarial feature learning. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City, UT, USA: IEEE, 2018 : 5400–5409.
    [6]
    Li D, Yang Y, Song Y Z, et al. Learning to generalize: Meta-learning for domain generalization. Proceedings of the AAAI Conference on Artificial Intelligence, 2018, 32 (1). doi: 10.1609/aaai.v32i1.11596
    [7]
    Mancini M, Bulò S R, Caputo B, et al. Robust place categorization with deep domain generalization. IEEE Robotics and Automation Letters, 2018, 3: 2093–2100. doi: 10.1109/LRA.2018.2809700
    [8]
    Mancini M, Bulò S R, Caputo B, et al. Best sources forward: Domain generalization through source-specific nets. In: 2018 25th IEEE International Conference on Image Processing (ICIP). Athens, Greece: IEEE, 2018 : 1353–1357.
    [9]
    Segu M, Tonioni A, Tombari F. Batch normalization embeddings for deep domain generalization. Pattern Recognition, 2023, 135: 109115. doi: 10.1016/j.patcog.2022.109115
    [10]
    Zhang H, Cisse M, Dauphin Y N, et al. Mixup: Beyond empirical risk minimization. arXiv: 1710.09412, 2017.
    [11]
    Ding Z, Fu Y. Deep domain generalization with structured low-rank constraint. IEEE Transactions on Image Processing, 2018, 27: 304–313. doi: 10.1109/TIP.2017.2758199
    [12]
    Nalisnick E, Matsukawa A, Teh Y W, et al. Do deep generative models know what they don’t know? arXiv: 1810.09136, 2019 .
    [13]
    Li Y, Tian X, Gong M, et al. Deep domain generalization via conditional invariant adversarial networks. In: Ferrari V, Hebert M, Sminchisescu C, editors. Computer Vision–ECCV 2018. Cham: Springer, 2018 , 11219: 647–663.
    [14]
    Long M, Cao Y, Wang J, et al. Learning transferable features with deep adaptation networks. In: Proceedings of the 32nd International Conference on Machine Learning. New York: ACM, 2015 , 37: 97–105.
    [15]
    Ganin Y, Lempitsky V. Unsupervised domain adaptation by backpropagation. In: Proceedings of the 32nd International Conference on Machine Learning. New York: ACM, 2015 , 37: 1180–1189.
    [16]
    Bousmalis K, Silberman N, Dohan D, et al. Unsupervised pixel-level domain adaptation with generative adversarial networks. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Honolulu, HI, USA: IEEE, 2017 : 95–104.
    [17]
    Xu R, Chen Z, Zuo W, et al. Deep cocktail network: Multi-source unsupervised domain adaptation with category shift. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City, UT, USA: IEEE, 2018 : 3964–3973.
    [18]
    Zhou K, Yang Y, Hospedales T, et al. Learning to generate novel domains for domain generalization. In: Vedaldi A, Bischof H, Brox T, editors. Computer Vision–ECCV 2020. Cham: Springer, 2020 , 12361: 561–578.
    [19]
    Bai H, Sun R, Hong L, et al. DecAug: Out-of-distribution generalization via decomposed feature representation and semantic augmentation. Proceedings of the AAAI Conference on Artificial Intelligence, 2021, 35: 6705–6713. doi: 10.1609/aaai.v35i8.16829
    [20]
    Chattopadhyay P, Balaji Y, Hoffman J. Learning to balance specificity and invariance for in and out of domain generalization. In: European Conference on Computer Vision. Cham: Springer, 2020 : 301–318.
    [21]
    Peng X, Bai Q, Xia X, et al. Moment matching for multi-source domain adaptation. In: 2019 IEEE/CVF International Conference on Computer Vision (ICCV). Seoul, Korea (South): IEEE, 2020 : 1406–1415.
    [22]
    Li D, Zhang J, Yang Y, et al. Episodic training for domain generalization. In: 2019 IEEE/CVF International Conference on Computer Vision (ICCV). Seoul, Korea (South): IEEE, 2020 : 1446–1455.
    [23]
    S Seo, Y Suh, D Kim, G Kim, J Han, and B Han. Learning to optimize domain specific normalization for domain generalization. In: Vedaldi A, Bischof H, Brox T, et al. editors. Computer Vision─ECCV 2020. Cham: Springer, 2020 : 68–83.
    [24]
    K Zhou, Y Yang, Y Qiao, et al. Domain generalization with mixstyle. arXiv: 2104.02008, 2021 .
    [25]
    Cai Q, Wang Y, Pan Y, et al. Joint contrastive learning with infinite possibilities. In: Proceedings of the 34th International Conference on Neural Information Processing Systems. New York: ACM, 2020 : 12638–12648.
    [26]
    Chen T, Kornblith S, Norouzi M, et al. A simple framework for contrastive learning of visual representations. In: Proceedings of the 37th International Conference on Machine Learning. New York: ACM, 2020 : 1597–1607.
    [27]
    Mitrovic J, McWilliams B, Walker J, et al. Representation learning via invariant causal mechanisms. arXiv: 2010.07922, 2020 .
    [28]
    He K, Fan H, Wu Y, et al. Momentum contrast for unsupervised visual representation learning. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Seattle, WA, USA: IEEE, 2020 : 9726–9735.
    [29]
    Wu Z, Xiong Y, Yu S X, et al. Unsupervised feature learning via non-parametric instance discrimination. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City, UT, USA: IEEE, 2018 : 3733–3742.
    [30]
    Yao T, Zhang Y, Qiu Z, et al. SeCo: Exploring sequence supervision for unsupervised representation learning. Proceedings of the AAAI Conference on Artificial Intelligence, 2021, 35: 10656–10664. doi: 10.1609/aaai.v35i12.17274
    [31]
    Li D, Yang Y, Song Y Z, et al. Deeper, broader and artier domain generalization. In: 2017 IEEE International Conference on Computer Vision (ICCV). Venice, Italy: IEEE, 2017 : 5543–5551.
    [32]
    Carlucci F M, D'Innocente A, Bucci S, et al. Domain generalization by solving jigsaw puzzles. In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Long Beach, CA, USA: IEEE, 2020 : 2224–2233.
    [33]
    Matsuura T, Harada T. Domain generalization using a mixture of multiple latent domains. Proceedings of the AAAI Conference on Artificial Intelligence, 2020, 34: 11749–11756. doi: 10.1609/aaai.v34i07.6846
    [34]
    Mahajan D, Tople S, Sharma A. Domain generalization using causal matching. In: Proceedings of the 38th International Conference on Machine Learning. Volume 139 of Proceedings of Machine Learning Research. PMLR, 2021 , 139: 7313–7324.
    [35]
    Nam H, Lee H, Park J, et al. Reducing domain gap by reducing style bias. arXiv: 1910.11645, 2019.
    [36]
    Zhou K, Yang Y, Hospedales T, et al. Deep domain-adversarial image generation for domain generalisation. Proceedings of the AAAI Conference on Artificial Intelligence, 2020, 34: 13025–13032. doi: 10.1609/aaai.v34i07.7003
    [37]
    Balaji Y, Sankaranarayanan S, Chellappa R. MetaReg: towards domain generalization using meta-regularization. In: Proceedings of the 32nd International Conference on Neural Information Processing Systems. New York: ACM, 2018 : 1006–1016.
    [38]
    Chattopadhyay P, Balaji Y, Hoffman J. Learning to balance specificity and invariance for in and out of domain generalization. In: Vedaldi A, Bischof H, Brox T, editors. Computer Vision–ECCV 2020. Cham: Springer, 2020 : 301–318.
    Recommended articles
    TrendMD
    Volume 54 Issue 4 page: 0404
    Cover

    Article Metrics

    Article views (175) PDF downloads(643)
    Proportional views

    Copyright © Editorial Office of JUSTC, All Rights Reserved. 皖ICP备05002528号

    Supported by: Beijing Renhe Information Technology Co. Ltd    

    /

    DownLoad:  Full-Size Img  PowerPoint
    Return
    Return