!g~dZddlZddlmZddlmZddlmZddl Z ddl m Z ddl m Z mZddZddZdd Z dd Z dd Z dd Zd ZGddZ ddZdZddZGddZGddeZd dZy)!zM Created on Fri Aug 17 13:10:52 2012 Author: Josef Perktold License: BSD-3 N)svds) fminbound)Bunch)IterationLimitWarningiteration_limit_docctjj|\}}tj||k}tj|tj ||z|j }||fSN)nplinalgeighanydotmaximumT)xvalueevalsevecsclippedx_news `/var/www/dash_apps/app1/venv/lib/python3.12/site-packages/statsmodels/stats/correlation_tools.py clip_evalsrsX99>>!$LE5ffUU]#G FF52::eU33UWW =E '>c|jd}||jdk7r tdtj|j}|j }tj |}t tt||zD]:}||z }t||\} } | s| }|S| |z }| j }d|||f<<tjtt|S)a Find the nearest correlation matrix that is positive semi-definite. The function iteratively adjust the correlation matrix by clipping the eigenvalues of a difference matrix. The diagonal elements are set to one. Parameters ---------- corr : ndarray, (k, k) initial correlation matrix threshold : float clipping threshold for smallest eigenvalue, see Notes n_fact : int or float factor to determine the maximum number of iterations. The maximum number of iterations is the integer part of the number of columns in the correlation matrix times n_fact. Returns ------- corr_new : ndarray, (optional) corrected correlation matrix Notes ----- The smallest eigenvalue of the corrected correlation matrix is approximately equal to the ``threshold``. If the threshold=0, then the smallest eigenvalue of the correlation matrix might be negative, but zero within a numerical error, for example in the range of -1e-16. Assumes input correlation matrix is symmetric. Stops after the first step if correlation matrix is already positive semi-definite or positive definite, so that smallest eigenvalue is above threshold. In this case, the returned array is not the original, but is equal to it within numerical precision. See Also -------- corr_clipped cov_nearest rzmatrix is not squarer)shape ValueErrorr zeroscopyarangerangeintlenrwarningswarnrr) corr thresholdn_factk_varsdiffrdiag_idxiix_adjx_psdrs r corr_nearestr0sXZZ]F A/00 88DJJ D IIKEyy HCD F*+, B #E;wE  L u} $%h ! B  )+@A Lrct||\}}|s|Stjtj|}||z |dddfz }|S)a Find a near correlation matrix that is positive semi-definite This function clips the eigenvalues, replacing eigenvalues smaller than the threshold by the threshold. The new matrix is normalized, so that the diagonal elements are one. Compared to corr_nearest, the distance between the original correlation matrix and the positive definite correlation matrix is larger, however, it is much faster since it only computes eigenvalues once. Parameters ---------- corr : ndarray, (k, k) initial correlation matrix threshold : float clipping threshold for smallest eigenvalue, see Notes Returns ------- corr_new : ndarray, (optional) corrected correlation matrix Notes ----- The smallest eigenvalue of the corrected correlation matrix is approximately equal to the ``threshold``. In examples, the smallest eigenvalue can be by a factor of 10 smaller than the threshold, e.g. threshold 1e-8 can result in smallest eigenvalue in the range between 1e-9 and 1e-8. If the threshold=0, then the smallest eigenvalue of the correlation matrix might be negative, but zero within a numerical error, for example in the range of -1e-16. Assumes input correlation matrix is symmetric. The diagonal elements of returned correlation matrix is set to ones. If the correlation matrix is already positive semi-definite given the threshold, then the original correlation matrix is returned. ``cov_clipped`` is 40 or more times faster than ``cov_nearest`` in simple example, but has a slightly larger approximation error. See Also -------- corr_nearest cov_nearest rN)rr sqrtdiag)r'r(rrx_stds r corr_clippedr5^sPd I6NE7   GGBGGEN #E EME!T'N *E Lrcddlm}m}||d\}}|dk(rt||} nt |||} || |}|r|| |fS|S)aw Find the nearest covariance matrix that is positive (semi-) definite This leaves the diagonal, i.e. the variance, unchanged Parameters ---------- cov : ndarray, (k,k) initial covariance matrix method : str if "clipped", then the faster but less accurate ``corr_clipped`` is used.if "nearest", then ``corr_nearest`` is used threshold : float clipping threshold for smallest eigen value, see Notes n_fact : int or float factor to determine the maximum number of iterations in ``corr_nearest``. See its doc string return_all : bool if False (default), then only the covariance matrix is returned. If True, then correlation matrix and standard deviation are additionally returned. Returns ------- cov_ : ndarray corrected covariance matrix corr_ : ndarray, (optional) corrected correlation matrix std_ : ndarray, (optional) standard deviation Notes ----- This converts the covariance matrix to a correlation matrix. Then, finds the nearest correlation matrix that is positive semidefinite and converts it back to a covariance matrix using the initial standard deviation. The smallest eigenvalue of the intermediate correlation matrix is approximately equal to the ``threshold``. If the threshold=0, then the smallest eigenvalue of the correlation matrix might be negative, but zero within a numerical error, for example in the range of -1e-16. Assumes input covariance matrix is symmetric. See Also -------- corr_nearest corr_clipped r)cov2corrcorr2covT) return_stdr)r()r(r)) statsmodels.stats.moment_helpersr7r8r5r0) covmethodr(r) return_allr7r8cov_std_corr_s r cov_nearestrAs\lD#$/JD$ TY7TYvF E4 DUD   rc <d} ||} t|| d} t| D]u} ||| |zz}||}||zj}|| || z|zzkr| || |zz||fcSd| dzz|z|| z | |zz z }||kr ||| zkr|} n| dz} |} wy)a  Implements the non-monotone line search of Grippo et al. (1986), as described in Birgin, Martinez and Raydan (2013). Parameters ---------- obj : real-valued function The objective function, to be minimized grad : vector-valued function The gradient of the objective function x : array_like The starting point for the line search d : array_like The search direction obj_hist : array_like Objective function history (must contain at least one value) M : positive int Number of previous function points to consider (see references for details). sig1 : real Tuning parameter, see references for details. sig2 : real Tuning parameter, see references for details. gam : real Tuning parameter, see references for details. maxiter : int The maximum number of iterations; returns Nones if convergence does not occur by this point Returns ------- alpha : real The step value x : Array_like The function argument at the final step obval : Real The function value at the final step g : Array_like The gradient at the final step Notes ----- The basic idea is to take a big step in the direction of the gradient, even if the function value is not decreased (but there is a maximum allowed increase in terms of the recent history of the iterates). References ---------- Grippo L, Lampariello F, Lucidi S (1986). A Nonmonotone Line Search Technique for Newton's Method. SIAM Journal on Numerical Analysis, 23, 707-716. E. Birgin, J.M. Martinez, and M. Raydan. Spectral projected gradient methods: Review and perspectives. Journal of Statistical Software (preprint). ?Ngg@)NNNN)maxr"sum)objgradrdobj_histMsig1sig2gammaxiteralpha last_obvalobj_maxiterobvalggtda1s r_nmono_linesearchrXsx EQJ(A23- GgAaK  G1ukkm Gc%im+ +!eAg+ua/ / %(]3 %*"4uSy"@ A BJR4:-E RKE !$ "rc td|z| } |j}||}||g}tt|D]}||z }||||z}t j t j ||krtd id||ddcS|| |zz }||||z}t||||||| | | | \}}}}|td id||ddcS|j|||z }||z }||zj}|dkr| } n,||zj}t |t||z | } |}|}td id||d dS) a0 Implements the spectral projected gradient method for minimizing a differentiable function on a convex domain. Parameters ---------- func : real valued function The objective function to be minimized. grad : real array-valued function The gradient of the objective function start : array_like The starting point project : function In-place projection of the argument to the domain of func. ... See notes regarding additional arguments Returns ------- rslt : Bunch rslt.params is the final iterate, other fields describe convergence status. Notes ----- This can be an effective heuristic algorithm for problems where no guaranteed algorithm for computing a global minimizer is known. There are a number of tuning parameters, but these generally should not be changed except for `maxiter` (positive integer) and `ctol` (small positive real). See the Birgin et al reference for more information about the tuning parameters. Reference --------- E. Birgin, J.M. Martinez, and M. Raydan. Spectral projected gradient methods: Review and perspectives. Journal of Statistical Software (preprint). Available at: http://www.ime.usp.br/~egbirgin/publications/bmr5.pdf TzConverged successfully) Convergedparamsobjective_valuesMessage)rKrLrMrNrOFzFailed in nmono_linesearchrzspg_optim did not converge) minr r"r#r rEabsrrXappendrF)funcrHstartprojectrOrKctol maxiter_nmlslam_minlam_maxrLrMrNlamr\gvalrJitrdfrIrPparams1fvalgval1sysysss r _spg_optimru4sX bj' "C ZZ\F &(0;= >>rctj||zjd}tj|dkD}t |dkDr||ddfxx||dddfzcc<yy)z Project a matrix into the domain of matrices whose row-wise sums of squares are less than or equal to 1. The input matrix is modified in-place. rrN)r r2rF flatnonzeror$)Xnmr-s r_project_correlation_factorsrzs^ !A#1 B Q B 2w{ "a%BrF1d7O#rc.eZdZdZdZdZdZdZdZy)FactoredPSDMatrixa Representation of a positive semidefinite matrix in factored form. The representation is constructed based on a vector `diag` and rectangular matrix `root`, such that the PSD matrix represented by the class instance is Diag + root * root', where Diag is the square diagonal matrix with `diag` on its main diagonal. Parameters ---------- diag : 1d array_like See above root : 2d array_like See above Notes ----- The matrix is represented internally in the form Diag^{1/2}(I + factor * scales * factor')Diag^{1/2}, where `Diag` and `scales` are diagonal matrices, and `factor` is an orthogonal matrix. c||_||_|tj|dddfz }tjj |d\}}}||_|dz|_y)NrrD)r3rootr r2r svdfactorscales)selfr3r~urqvts r__init__zFactoredPSDMatrix.__init__sW  bggdmAtG,,99==q)1b d rctj|jtj|j|jjzS)zh Returns the PSD matrix represented by this instance as a full (square) matrix. )r r3rr~rrs r to_matrixzFactoredPSDMatrix.to_matrixs3 wwtyy!BFF499diikk$BBBrc^ddtjd|jzz z}|tj|jdddfz }tj|j j |}||dddfz}tj|j |}||z }|S)aE Decorrelate the columns of `rhs`. Parameters ---------- rhs : array_like A 2 dimensional array with the same number of rows as the PSD matrix represented by the class instance. Returns ------- C^{-1/2} * rhs, where C is the covariance matrix represented by this class instance. Notes ----- The returned matrix has the identity matrix as its row-wise population covariance matrix. This function exploits the factor structure for efficiency. rN)r r2rr3rrr)rrhsqvalrhs1s r decorrelatezFactoredPSDMatrix.decorrelates4ADKK000BGGDII&q$w//vvdkkmmS) QW vvdkk4( t  rcX|j d|jzz }tj|j}||dddfz }|dddftj|j j |z}|tj|j |z}||dddfz S)a Solve a linear system of equations with factor-structured coefficients. Parameters ---------- rhs : array_like A 2 dimensional array with the same number of rows as the PSD matrix represented by the class instance. Returns ------- C^{-1} * rhs, where C is the covariance matrix represented by this class instance. Notes ----- This function exploits the factor structure for efficiency. rN)rr r2r3rrr)rrrdrmats rsolvezFactoredPSDMatrix.solves* |q4;;/ WWTYY Bq$wK1d7mbffT[[]]C88BFF4;;,,R4[  rcJtjtj|j}|tjtj|jz }|tjtjdd|jz zz }|S)za Returns the logarithm of the determinant of a factor-structured matrix. r)r rFlogr3r)rlogdets rrzFactoredPSDMatrix.logdetsm tyy)*"&& ,--"&&A O 3455 rN) __name__ __module__ __qualname____doc__rrrrrr_rrr|r|s",C#J!8 rr|c D|j\}}t||\}} } |tj| z} tj| dzj d} tj | dkD} | | ddfxx| | dddfzcc<|j ttjurtjdnrtjrRjtjjdjjn t!dfd}fd}t#||| t$|||| }|j&}d|dzj dz }t)||}||_|`|S) a Find the nearest correlation matrix with factor structure to a given square matrix. Parameters ---------- corr : square array The target matrix (to which the nearest correlation matrix is sought). Must be square, but need not be positive semidefinite. rank : int The rank of the factor structure of the solution, i.e., the number of linearly independent columns of X. ctol : positive real Convergence criterion. lam_min : float Tuning parameter for spectral projected gradient optimization (smallest allowed step in the search direction). lam_max : float Tuning parameter for spectral projected gradient optimization (largest allowed step in the search direction). maxiter : int Maximum number of iterations in spectral projected gradient optimization. Returns ------- rslt : Bunch rslt.corr is a FactoredPSDMatrix defining the estimated correlation structure. Other fields of `rslt` contain returned values from spg_optim. Notes ----- A correlation matrix has factor structure if it can be written in the form I + XX' - diag(XX'), where X is n x k with linearly independent columns, and with each row having sum of squares at most equal to 1. The approximation is made in terms of the Frobenius norm. This routine is useful when one has an approximate correlation matrix that is not positive semidefinite, and there is need to estimate the inverse, square root, or inverse square root of the population correlation matrix. The factor structure allows these tasks to be done without constructing any n x n matrices. This is a non-convex problem with no known guaranteed globally convergent algorithm for computing the solution. Borsdof, Higham and Raydan (2010) compared several methods for this problem and found the spectral projected gradient (SPG) method (used here) to perform best. The input matrix `corr` can be a dense numpy array or any scipy sparse matrix. The latter is useful if the input matrix is obtained by thresholding a very large sample correlation matrix. If `corr` is sparse, the calculations are optimized to save memory, so no working matrix with more than 10^6 elements is constructed. References ---------- .. [*] R Borsdof, N Higham, M Raydan (2010). Computing a nearest correlation matrix with factor structure. SIAM J Matrix Anal Appl, 31:5, 2603-2622. http://eprints.ma.man.ac.uk/1523/01/covered/MIMS_ep2009_87.pdf Examples -------- Hard thresholding a correlation matrix may result in a matrix that is not positive semidefinite. We can approximate a hard thresholded correlation matrix with a PSD matrix as follows, where `corr` is the input correlation matrix. >>> import numpy as np >>> from statsmodels.stats.correlation_tools import corr_nearest_factor >>> np.random.seed(1234) >>> b = 1.5 - np.random.rand(10, 1) >>> x = np.random.randn(100,1).dot(b.T) + np.random.randn(100,10) >>> corr = np.corrcoef(x.T) >>> corr = corr * (np.abs(corr) >= 0.3) >>> rslt = corr_nearest_factor(corr, 3) rDrh㈵>NrzMatrix type not supportedcJtj|tj|j|}ttjur|tj|z}n|j|z}|||zj ddddf|zz}d|zS)Nr)r rrtypendarrayrF)rxgrcorr1s rrHz!corr_nearest_factor..grads VVArvvacc1~ & ;"** $ "&&" "B %))A, B qsiil1d7#a''t rct tjurPtj||j}tj |d| z}||zj }|Sd}d}t||jdz }d}||jdkrt||z|jd}tj|||ddf|j}tj|jd}d||||zf<|tj ||ddfjz}|||zj z }||z }||jdkr|S)Nrg.A) rr rrr fill_diagonalrFr#rr`r!asarraytodense) rxrKromax_wsbsirir2rr-rs rrcz!corr_nearest_factor..funcsE ;"** $q!##A   Q " JAaC99;DKDFVaggaj()BBqwwqz/"R%,FF1RVQY<-YYqwwqz* "be) RZZbfai 0 8 8 :;;1 #bqwwqz/Kr)rfrhrirO)rrr r2rFrwr rrrsparseissparsesetdiagreliminate_zeros sort_indicesrrurzr\r|r')r'rankrfrhrirOp_rrqrrxryr-rHrcrsltr~r3solnrs @rcorr_nearest_factorrsbj ::DAqD$HAq" BGGAJA !Q$A B T "Bb!eH2q$wH IIKE E{bjj  "   bhhu{{1~./  455. dD!%A%w ID ;;D a}}Q D T4 (DDI Krc |j\ }t||\} }tj|r{t j |j |j | |jj |j |jjnrt j |j t j || t j| t jt j || fd}t|dd} |z }|t j|z}|t j tjz} t| |S)a' Approximate an arbitrary square matrix with a factor-structured matrix of the form k*I + XX'. Parameters ---------- cov : array_like The input array, must be square but need not be positive semidefinite rank : int The rank of the fitted factor structure Returns ------- A FactoredPSDMatrix instance containing the fitted matrix Notes ----- This routine is useful if one has an estimated covariance matrix that is not SPD, and the ultimate goal is to estimate the inverse, square root, or inverse square root of the true covariance matrix. The factor structure allows these tasks to be performed without constructing any n x n matrices. The calculations use the fact that if k is known, then X can be determined from the eigen-decomposition of cov - k*I, which can in turn be easily obtained form the eigen-decomposition of `cov`. Thus the problem can be reduced to a 1-dimensional search for k that does not require repeated eigen-decompositions. If the input matrix is sparse, then cov - k*I is also sparse, so the eigen-decomposition can be done efficiently using sparse routines. The one-dimensional search for the optimal value of k is not convex, so a local minimum could be obtained. Examples -------- Hard thresholding a covariance matrix may result in a matrix that is not positive semidefinite. We can approximate a hard thresholded covariance matrix with a PSD matrix as follows: >>> import numpy as np >>> np.random.seed(1234) >>> b = 1.5 - np.random.rand(10, 1) >>> x = np.random.randn(100,1).dot(b.T) + np.random.randn(100,10) >>> cov = np.cov(x) >>> cov = cov * (np.abs(cov) >= 0.3) >>> rslt = cov_nearest_factor_homog(cov, 3) c|z }|dzzztj|dzzd|zzz }|d|ztj|zdtjtj|zzz z }|S)NrD)r rFr3)kLambda_tvLambdaQSQmtstsss rfunz%cov_nearest_factor_homog..funs}A: !QT(NRVVHaK0 01Q3r6 9 QqS! !AbffRWWS\H-D&E$E EErrgj@dtype)rrrrr rrdiagonalrFtracerr2onesfloat64r|)r;rnQrrk_opt Lambda_optfac_optr3rrrrrs @@@@@rcov_nearest_factor_homogrsj 99DAqT?LAvq sffQSS#''!*% \\^   !ggcl##%))+ffQSS"&&a.) XXc]hhrvvc3'( c1c "E%J"''*%%G 2771BJJ/ /D T7 ++rc|j\}}|dt|z }|j}||jddddfz}|j dd}t j |dkD}||ddfxx||dddfzcc<t j |dk}d||ddf<tt j||z }ggg} } }d} | |krt|jd| |z} t j|| | ddf|j|dz z } t j| }t j||k\\}}|j|| z| j|| j| ||f| |z } | |krt j|}t j| }t j| } t!j"| ||ff||f}|S)ap Construct a sparse matrix containing the thresholded row-wise correlation matrix from a data array. Parameters ---------- data : array_like The data from which the row-wise thresholded correlation matrix is to be computed. minabs : non-negative real The threshold value; correlation coefficients smaller in magnitude than minabs are set to zero. If None, defaults to 1 / sqrt(n), see Notes for more information. Returns ------- cormat : sparse.coo_matrix The thresholded correlation matrix, in COO format. Notes ----- This is an alternative to C = np.corrcoef(data); C \*= (np.abs(C) >= absmin), suitable for very tall data matrices. If the data are jointly Gaussian, the marginal sampling distributions of the elements of the sample correlation matrix are approximately Gaussian with standard deviation 1 / sqrt(n). The default value of ``minabs`` is thus equal to 1 standard error, which will set to zero approximately 68% of the estimated correlation coefficients for which the population value is zero. No intermediate matrix with more than ``max_elt`` values will be constructed. However memory use could still be high if a large number of correlation values exceed `minabs` in magnitude. The thresholded matrix is returned in COO format, which can easily be converted to other sparse formats. Examples -------- Here X is a tall data matrix (e.g. with 100,000 rows and 50 columns). The row-wise correlation matrix of X is calculated and stored in sparse form, with all entries smaller than 0.3 treated as 0. >>> import numpy as np >>> np.random.seed(1234) >>> b = 1.5 - np.random.rand(10, 1) >>> x = np.random.randn(100,1).dot(b.T) + np.random.randn(100,10) >>> cmat = corr_thresholded(x, 0.3) NrCr)ddofrr)rfloatr meanstdr rwr#floorr`rrranonzerorb concatenater coo_matrix)dataminabsmax_eltnrowncolsdr-ripos_alljpos_all cor_valuesrrcmcmaiposjposcmats rcorr_thresholdedrsjJD$ ~eDk! 99;DDIIaLD !!D !! B T "BQK2b6!T'?"K d #BDQK RXXgn % &B%'R hH B t)$**Q-b) VVDCOTVV ,q 9ffRjZZv . dr ""T4Z.) b t) >>( #D >>( #D +J   j4,7$ FD Krc*eZdZdZdZdZdZddZy)MultivariateKernelz Base class for multivariate kernels. An instance of MultivariateKernel implements a `call` method having signature `call(x, loc)`, returning the kernel weights comparing `x` (a 1d ndarray) to each row of `loc` (a 2d ndarray). ctr )NotImplementedErrorrrlocs rcallzMultivariateKernel.callis!!rc2||_|jy)z Set the bandwidth to the given vector. Parameters ---------- bw : array_like A vector of non-negative bandwidth values. N)bw_setup)rrs r set_bandwidthz MultivariateKernel.set_bandwidthls rctj|j|_|j|jz|_yr )r prodrbwkbw2rs rrzMultivariateKernel._setupys,77477#77TWW$rNcV|jd}tj|ddgd\}}||z dz }tj||k||}|d|jddzz z}|||z}tj |tj |_|jy) aP Set default bandwiths based on domain values. Parameters ---------- loc : array_like Values from the domain to which the kernel will be applied. bwm : scalar, optional A non-negative scalar that is used to multiply the default bandwidth. rK)axisg/$??g?Nr) rr percentilewhererrrrr)rrbwmrq25q75iqrrs rset_default_bwz!MultivariateKernel.set_default_bwsWWQZ==r2hQ7SSyE! XXcBhR ( cCIIaLC''' ? #IB**Rrzz2 rr )rrrrrrrrr_rrrr`s" % rrceZdZdZdZy)GaussianMultivariateKernelzA The Gaussian (squared exponential) multivariate kernel. ctj||z dz d|jzz jd|jz S)NrDr)r exprrFrrs rrzGaussianMultivariateKernel.calls<vvC!|mq488|4599!jkg|=zgEffective sample size is 0. The bandwidth may be too small, or you are outside the range of your data.) ritemsr$r indicesflateinsumrFr%r&nan ones_like)rrrkxkyrcwrUr-rj1j2wmsgexogixkernelrs rr;zkernel_covariance..covs [[C  [[C BXXZ EArBAZZA'FBBGGBBGGB2BA "))M4A;RU QG GB !%%'MB  :GC MM# 66BLL,, ,Bwr)r rndimrr$r`rEr enumeraterbkeyssortrrisscalarr) rrgroupsrrrrirUr;rs `` ` @rkernel_covariancersJ\ ::d D **S/C ZZ F xx1}!T'l A ! c&k2A 1vQGo B&!1 B;BqE 1 QWWY11~+- zc" Rcr*R 8 Jr)r)V瞯)Fr rr!)gư>r#r$i)NgcA)NN)rnumpyr scipy.sparserscipy.sparse.linalgrscipy.optimizerr%statsmodels.tools.toolsrstatsmodels.tools.sm_exceptionsrrrr0r5rArXrurzr|rrrrrrr_rrr+s$$)0@F9x@C BJ=@25R"j;=4959e>P $qqh8=.2SlN,bYx99xH!3Hir