ISSN 0253-2778

CN 34-1054/N

open
Free to Publish. Free to Read.
Open AccessOpen Access JUSTC Original Paper

Reduced rank regression based on hard-thresholding singular value penalization

  • Department of Statistics and Finance, School of Management, University of Science and Technology, Hefei 230026, China

Cite this:
https://doi.org/10.3969/j.issn.0253-2778.2020.02.012
  • Received Date: 12 January 2019
  • Accepted Date: 02 June 2019
  • Rev Recd Date: 02 June 2019
  • Publish Date: 28 February 2020
    Abstract

  • Abstract

    Reduced rank estimation using penalty functions to restrict ranks of variety matrices is often used for solving the multi-collinearity of high-dimensional multivariate regression. Here a hard-thresholding singular value penalization was considered to get more efficient results. Through local linear approximate method, non-convex models were converted to computable ones. This model is computationally efficient, and the resulting solution path is continuous. Experiment results from simulation and public datasets show that this kind of reduced rank regression has better accuracy than some frequently-used ones in most situations.

    Abstract

    Reduced rank estimation using penalty functions to restrict ranks of variety matrices is often used for solving the multi-collinearity of high-dimensional multivariate regression. Here a hard-thresholding singular value penalization was considered to get more efficient results. Through local linear approximate method, non-convex models were converted to computable ones. This model is computationally efficient, and the resulting solution path is continuous. Experiment results from simulation and public datasets show that this kind of reduced rank regression has better accuracy than some frequently-used ones in most situations.
    • FullText(HTML)
  • loading
    • References(22)
  • [1]
    YUAN M, EKICI A, LU Z, et al. Dimension reduction and coefficient estimation in multivariate linear regression[J]. Journal of the Royal Statistical Society, 2010, 57(3): 329-346.
    [2]
    NEGAHBAN S N, WAINWRIGHT M J. Estimation of (near) low-rank matrices with noise and high-dimensional scaling[J]. International Conference on Machine Learning, 2010, 39(2): 823-830.
    [3]
    BUNEA F, SHE Y, WEGKAMP M H. Optimal selection of reduced rank estimators of high-dimensional matrices[J]. Annals of Statistics, 2010, 39(2): 1282-1309.
    [4]
    CHEN K, DONG H, CHAN K S. Reduced rank regression via adaptive nuclear norm penalization[J]. Biometrika, 2013, 100(4): 901-920.
    [5]
    FAN J, LI R. Variable selection vianonconvave penalized likelihood and its oracle properties[J]. Publications of the American Statistical Association, 2001, 96(456): 1348-1360.
    [6]
    ZHENG Z, FAN Y, LV J. High dimensional thresholded regression and shrinkage effect[J].Journal of the Royal Statistical Society B, 2014, 76(3) : 627-649.
    [7]
    HOERL A E, KENNARDR W. Ridge regression: Biased estimation for nonorthogonal problems[J]. Technometrics, 2000, 42(1): 80-86.
    [8]
    ROHDE A, TSYBAKOV A B. Estimation of high-dimensional low-rank matrices[J]. Annals of Statistics, 2011, 39(2): 887-930.
    [9]
    DONOHO D L, ELAD M. Optimally sparse representation in general (nonorthogonal) dictionaries via l minimization[J]. Proceedings of the National Academy of Sciences of the United States of America, 2003, 100(5): 2197-2202.
    [10]
    BICKEL P J, RITOV Y, TSYBAKOV A B. Simultaneous analysis of lasso and Dantzig selector[J]. Annals of Statistics, 2008, 37(4): 1705-1732.
    [11]
    FAN J, LV J.Nonconcave penalized likelihood with NP-dimensionality[J]. IEEE Transactions on Information Theory, 2011, 57(8): 5467-5484.
    [12]
    REINSEL G C, VELU R P. Multivariate Reduced-Rank Regression[M]. New York: Springer, 1998: 369-370.
    [13]
    ZOU H, LI R. One-step sparse estimates innonconcave penalized likelihood models[J]. Annals of Statistics, 2008, 36(4): 1509-1533.
    [14]
    LANGE K, HUNTER D R, YANG I. Optimization transfer using surrogate objective functions[J]. Journal of Computational and Graphical Statistics, 2000, 9(1): 1-20.
    [15]
    ZOU H, HASTIE T. Regularization and variable selection via the elastic net[J]. J Roy Statist Soc Ser B, 2005, 67(2): 301-320.
    [16]
    TIBSHIRANI R. Regression shrinkage and selection via the lasso[J]. Journal of the Royal Statistical Society, 2011, 73(3): 273-282.
    [17]
    HUANG J, HOROWITZ J L, MA S. Asymptotic properties of bridge estimators in sparse high-dimensional regression models[J]. Annals of Statistics, 2008, 36(2): 587-613.
    [18]
    KLOPP O. Rank penalized estimators for high-dimensional matrices[J]. Electronic Journal of Statistics, 2011, 5(2011): 1161-1183.
    [19]
    KOLTCHINSKII V, LOUNICI K, TSYBAKOV A B. Nuclear-norm penalization and optimal rates for noisy low-rank matrix completion[J]. Annals of Statistics, 2011, 39(5): 2302-2329.
    [20]
    WITTEN D M, TIBSHIRANI R, HASTIE T. A penalized matrix decomposition, with applications to sparse principal components and canonical correlation analysis[J]. Biostatistics, 2009, 10(3): 515-534.
    [21]
    CHIN K, DEVRIES S, FRIDLYAND J, et al. Genomic and transcriptional aberrations linked to breast cancer pathophysiologies[J]. Cancer Cell, 2006, 10(6): 529-541.
    [22]
    VON NEUMANN J. Some matrix-inequalities and metrization of matrix-space[J]. Tomsk Univ Rev, 1937, 11(1): 286-300.
    • Supplements(0)
    • Related Articles
    • Track Citations
  • 加载中

Catalog

    |
    Get Citation
    PDF
    XML
    [1]
    YUAN M, EKICI A, LU Z, et al. Dimension reduction and coefficient estimation in multivariate linear regression[J]. Journal of the Royal Statistical Society, 2010, 57(3): 329-346.
    [2]
    NEGAHBAN S N, WAINWRIGHT M J. Estimation of (near) low-rank matrices with noise and high-dimensional scaling[J]. International Conference on Machine Learning, 2010, 39(2): 823-830.
    [3]
    BUNEA F, SHE Y, WEGKAMP M H. Optimal selection of reduced rank estimators of high-dimensional matrices[J]. Annals of Statistics, 2010, 39(2): 1282-1309.
    [4]
    CHEN K, DONG H, CHAN K S. Reduced rank regression via adaptive nuclear norm penalization[J]. Biometrika, 2013, 100(4): 901-920.
    [5]
    FAN J, LI R. Variable selection vianonconvave penalized likelihood and its oracle properties[J]. Publications of the American Statistical Association, 2001, 96(456): 1348-1360.
    [6]
    ZHENG Z, FAN Y, LV J. High dimensional thresholded regression and shrinkage effect[J].Journal of the Royal Statistical Society B, 2014, 76(3) : 627-649.
    [7]
    HOERL A E, KENNARDR W. Ridge regression: Biased estimation for nonorthogonal problems[J]. Technometrics, 2000, 42(1): 80-86.
    [8]
    ROHDE A, TSYBAKOV A B. Estimation of high-dimensional low-rank matrices[J]. Annals of Statistics, 2011, 39(2): 887-930.
    [9]
    DONOHO D L, ELAD M. Optimally sparse representation in general (nonorthogonal) dictionaries via l minimization[J]. Proceedings of the National Academy of Sciences of the United States of America, 2003, 100(5): 2197-2202.
    [10]
    BICKEL P J, RITOV Y, TSYBAKOV A B. Simultaneous analysis of lasso and Dantzig selector[J]. Annals of Statistics, 2008, 37(4): 1705-1732.
    [11]
    FAN J, LV J.Nonconcave penalized likelihood with NP-dimensionality[J]. IEEE Transactions on Information Theory, 2011, 57(8): 5467-5484.
    [12]
    REINSEL G C, VELU R P. Multivariate Reduced-Rank Regression[M]. New York: Springer, 1998: 369-370.
    [13]
    ZOU H, LI R. One-step sparse estimates innonconcave penalized likelihood models[J]. Annals of Statistics, 2008, 36(4): 1509-1533.
    [14]
    LANGE K, HUNTER D R, YANG I. Optimization transfer using surrogate objective functions[J]. Journal of Computational and Graphical Statistics, 2000, 9(1): 1-20.
    [15]
    ZOU H, HASTIE T. Regularization and variable selection via the elastic net[J]. J Roy Statist Soc Ser B, 2005, 67(2): 301-320.
    [16]
    TIBSHIRANI R. Regression shrinkage and selection via the lasso[J]. Journal of the Royal Statistical Society, 2011, 73(3): 273-282.
    [17]
    HUANG J, HOROWITZ J L, MA S. Asymptotic properties of bridge estimators in sparse high-dimensional regression models[J]. Annals of Statistics, 2008, 36(2): 587-613.
    [18]
    KLOPP O. Rank penalized estimators for high-dimensional matrices[J]. Electronic Journal of Statistics, 2011, 5(2011): 1161-1183.
    [19]
    KOLTCHINSKII V, LOUNICI K, TSYBAKOV A B. Nuclear-norm penalization and optimal rates for noisy low-rank matrix completion[J]. Annals of Statistics, 2011, 39(5): 2302-2329.
    [20]
    WITTEN D M, TIBSHIRANI R, HASTIE T. A penalized matrix decomposition, with applications to sparse principal components and canonical correlation analysis[J]. Biostatistics, 2009, 10(3): 515-534.
    [21]
    CHIN K, DEVRIES S, FRIDLYAND J, et al. Genomic and transcriptional aberrations linked to breast cancer pathophysiologies[J]. Cancer Cell, 2006, 10(6): 529-541.
    [22]
    VON NEUMANN J. Some matrix-inequalities and metrization of matrix-space[J]. Tomsk Univ Rev, 1937, 11(1): 286-300.
    Recommended articles
    TrendMD
    Volume 50 Issue 2 page: 163-175
    Cover

    Article Metrics

    Article views (87) PDF downloads(626)
    Proportional views

    Copyright © Editorial Office of JUSTC, All Rights Reserved. 皖ICP备05002528号

    Supported by: Beijing Renhe Information Technology Co. Ltd    

    /

    DownLoad:  Full-Size Img  PowerPoint
    Return
    Return