!gqSdZddlmZddlZddlmZddlmZddl m Z dZ d:dZ d ZeZd Zd Zd Zd ZdZiZeeeeeeeded<dZdddded<deied<GddZd;dZdd ZGd!d"Zed#k(rdd$lmZej@jCd%d&Z!e"d'e"e e!d(ee!d(Z#e"e#jIgd)Z%e%D]Z&e"e&e#jIe&de"d*dd+l'm(Z(e(e)Z*dZ+e,dD]WZ-ej\j_e+Z!ee!d(Z#e%D],Z&e*e&jae#jIe&ddd,.Yejbe%Dcgc]}e*| c}Z2e"d-d.jge%e"d/e2d0kjid,e"d1e2d2kjid,e"d3e2d4kjid,ed5d(d6dZ+dZ5eed7e+e5d8Z6ejne5ejbgd9zjqe9Z:e"e6e:yycc}w)?atMore Goodness of fit tests contains GOF : 1 sample gof tests based on Stephens 1970, plus AD A^2 bootstrap : vectorized bootstrap p-values for gof test with fitted parameters Created : 2011-05-21 Author : Josef Perktold parts based on ks_2samp and kstest from scipy.stats (license: Scipy BSD, but were completely rewritten by Josef Perktold) References ---------- )lmapN) distributions)cache_readonly) kolmogorovcttj||f\}}|jd}|jd}t |}t |}tj |}tj |}tj ||g}tj||dd|zz }tj||dd|zz }tjtj||z }tj||zt||zz } t|dzd|z z|z} || fS#d} Y|| fSxYw)aA Computes the Kolmogorov-Smirnof statistic on 2 samples. This is a two-sided test for the null hypothesis that 2 independent samples are drawn from the same continuous distribution. Parameters ---------- a, b : sequence of 1-D ndarrays two arrays of sample observations assumed to be drawn from a continuous distribution, sample sizes can be different Returns ------- D : float KS statistic p-value : float two-tailed p-value Notes ----- This tests whether 2 samples are drawn from the same distribution. Note that, like in the case of the one-sample K-S test, the distribution is assumed to be continuous. This is the two-sided test, one-sided tests are not implemented. The test uses the two-sided asymptotic Kolmogorov-Smirnov distribution. If the K-S statistic is small or the p-value is high, then we cannot reject the hypothesis that the distributions of the two samples are the same. Examples -------- >>> from scipy import stats >>> import numpy as np >>> from scipy.stats import ks_2samp >>> #fix random seed to get the same result >>> np.random.seed(12345678) >>> n1 = 200 # size of first sample >>> n2 = 300 # size of second sample different distribution we can reject the null hypothesis since the pvalue is below 1% >>> rvs1 = stats.norm.rvs(size=n1,loc=0.,scale=1) >>> rvs2 = stats.norm.rvs(size=n2,loc=0.5,scale=1.5) >>> ks_2samp(rvs1,rvs2) (0.20833333333333337, 4.6674975515806989e-005) slightly different distribution we cannot reject the null hypothesis at a 10% or lower alpha since the pvalue at 0.144 is higher than 10% >>> rvs3 = stats.norm.rvs(size=n2,loc=0.01,scale=1.0) >>> ks_2samp(rvs1,rvs3) (0.10333333333333333, 0.14498781825751686) identical distribution we cannot reject the null hypothesis since the pvalue is high, 41% >>> rvs4 = stats.norm.rvs(size=n2,loc=0.0,scale=1.0) >>> ks_2samp(rvs1,rvs4) (0.07999999999999996, 0.41126949729859719) rright)side?Q?)\(?) rnpasarrayshapelensort concatenate searchsortedmaxabsolutesqrtfloatksprob) data1data2n1n2data_allcdf1cdf2denprobs f/var/www/dash_apps/app1/venv/lib/python3.12/site-packages/statsmodels/sandbox/distributions/gof_new.pyks_2sampr$s%P UEN3LE5 QB QB UB UB GGENE GGENE~~uUm,H ??5w 7R @D OOE( 83r6 BD r{{49%&A BuRU|# $Br$wtBw)* d7N d7Ns $D??Ec t|trG|r||k(r5tt|j}tt|j }n t dt|trtt|j}t|r d|i}tj||i|}n tj|}t|}||g|}|dvrTtjd|dz|z |z j} |dk(r"| tjj| |fS|dvrQ|tjd||z z j} |d k(r"| tjj| |fS|d k(rtj  g} |d k(r7| tjj| tj |zfS|d k(rtjj| tj |z} |d kDs| d|dzdz z kDr7| tjj| tj |zfS| tjj| |dzfSyy)a Perform the Kolmogorov-Smirnov test for goodness of fit This performs a test of the distribution G(x) of an observed random variable against a given distribution F(x). Under the null hypothesis the two distributions are identical, G(x)=F(x). The alternative hypothesis can be either 'two_sided' (default), 'less' or 'greater'. The KS test is only valid for continuous distributions. Parameters ---------- rvs : str or array or callable string: name of a distribution in scipy.stats array: 1-D observations of random variables callable: function to generate random variables, requires keyword argument `size` cdf : str or callable string: name of a distribution in scipy.stats, if rvs is a string then cdf can evaluate to `False` or be the same as rvs callable: function to evaluate cdf args : tuple, sequence distribution parameters, used if rvs or cdf are strings N : int sample size if rvs is string or callable alternative : 'two_sided' (default), 'less' or 'greater' defines the alternative hypothesis (see explanation) mode : 'approx' (default) or 'asymp' defines the distribution used for calculating p-value 'approx' : use approximation to exact distribution of test statistic 'asymp' : use asymptotic distribution of test statistic Returns ------- D : float KS test statistic, either D, D+ or D- p-value : float one-tailed or two-tailed p-value Notes ----- In the one-sided test, the alternative is that the empirical cumulative distribution function of the random variable is "less" or "greater" than the cumulative distribution function F(x) of the hypothesis, G(x)<=F(x), resp. G(x)>=F(x). Examples -------- >>> from scipy import stats >>> import numpy as np >>> from scipy.stats import kstest >>> x = np.linspace(-15,15,9) >>> kstest(x,'norm') (0.44435602715924361, 0.038850142705171065) >>> np.random.seed(987654321) # set random seed to get the same result >>> kstest('norm','',N=100) (0.058352892479417884, 0.88531190944151261) is equivalent to this >>> np.random.seed(987654321) >>> kstest(stats.norm.rvs(size=100),'norm') (0.058352892479417884, 0.88531190944151261) Test against one-sided alternative hypothesis: >>> np.random.seed(987654321) Shift distribution to larger values, so that cdf_dgp(x)< norm.cdf(x): >>> x = stats.norm.rvs(loc=0.2, size=100) >>> kstest(x,'norm', alternative = 'less') (0.12464329735846891, 0.040989164077641749) Reject equal distribution against alternative hypothesis: less >>> kstest(x,'norm', alternative = 'greater') (0.0072115233216311081, 0.98531158590396395) Do not reject equal distribution against alternative hypothesis: greater >>> kstest(x,'norm', mode='asymp') (0.12464329735846891, 0.08944488871182088) Testing t distributed random variables against normal distribution: With 100 degrees of freedom the t distribution looks close to the normal distribution, and the kstest does not reject the hypothesis that the sample came from the normal distribution >>> np.random.seed(987654321) >>> stats.kstest(stats.t.rvs(100,size=100),'norm') (0.072018929165471257, 0.67630062862479168) With 3 degrees of freedom the t distribution looks sufficiently different from the normal distribution, that we can reject the hypothesis that the sample came from the normal distribution at a alpha=10% level >>> np.random.seed(987654321) >>> stats.kstest(stats.t.rvs(3,size=100),'norm') (0.131016895759829, 0.058826222555312224) 5if rvs is string, cdf has to be the same distributionsize) two_sidedgreaterr r))r(lessr+r(asympapproxj 皙?333333?@@N) isinstancestrgetattrrcdfrvsAttributeErrorcallabler rrarangerksonesf kstwobignr) r8r7argsN alternativemodekwdsvalscdfvalsDplusDminDpval_twos r#kstestrJ}s>f#s--11C--11C !XY Y#smS)--}qzwwsD(4()wws| I$G..3!$Q&0557 ) #---00q99 9++"))C+A--224 & ,,//Q77 7k! FFE$<  7?m--002771:>> > 8 $..11!BGGAJ,?H4x8dQsU6\&99-1144Qrwwqz\BBB---0015a777  "ctj|dzdtj|z z}||z}tjd|dzz}tj|tjgdkD}|||fS)Nr r r3)= ףp=?rNr r rexpsumarraystatnobs mod_factor stat_modifiedpvaldigitss r#dplus_st70_upprZsr%rwwt}(< >D VVD288$788 9F $ &&rK)d_plusd_minusr vwsquusqua stephens70uppctjj|tj|z}|dkDs|d|dzdz z kDrF|tjj|tj|ztj fS|tj j||dztj fS)Nr/r0r1r2r3)rr>r=r rrfr<)rHr@rIs r#pval_kstest_approxr|^s&&))!BGGAJ,7H4x8dQsU6\11-)),,Qrwwqz\:BFFBB-%%((1-a/77rKcd|tjj||tjfSNrr<r=r rf)rFr@s r#rfs$ (;(;(>(>ua(H"&& QrKcd|tjj||tjfSr~r)rGr@s r#rrgs$}':':'='=d1'Ervv NrKc|tjj|tj|ztj fSr~)rr>r=r rrf)rHr@s r#rrhs.=2255a lCRVVLrK)rtrur scipyr scipy_approxceZdZdZd dZedZedZedZedZ edZ edZ ed Z ed Z dd Zy )GOFaPOne Sample Goodness of Fit tests includes Kolmogorov-Smirnov D, D+, D-, Kuiper V, Cramer-von Mises W^2, U^2 and Anderson-Darling A, A^2. The p-values for all tests except for A^2 are based on the approximatiom given in Stephens 1970. A^2 has currently no p-values. For the Kolmogorov-Smirnov test the tests as given in scipy.stats are also available as options. design: I might want to retest with different distributions, to calculate data summary statistics only once, or add separate class that holds summary statistics and data (sounds good). ct|trG|r||k(r5tt|j}tt|j }n t dt|trtt|j}t|r d|i}tj||i|}n tj|}t|}||g|}||_ ||_ ||_ y)Nr&r')r4r5r6rr7r8r9r:r rrrU vals_sortedrE)selfr8r7r?r@rCrDrEs r#__init__z GOF.__init__s c3 SCZmS155mS155$%\]] c3 --11C C=1:D773,t,-D773C#7!";;; rKc|j}|j}d}td|D]0}|||d|z }|dkD}d||z ||<||jz }2|dz d|z |zz }|S)Nrr*rg@rr)rUrErangerQ)rrUrEmsumjmjmaskrys r#ryzGOF.asyy,,q Agbqk)BHD2d8|BtH BFFH D   2IT D( (rKc |j}|j}dtjd|dzzdz tj|tjd|dddz zzj |z |z }|S)z4Stephens 1974, does not have p-value formula for A^2rrr r*N)rUrEr r;logrQ)rrUrEasqus r#rzGOF.asqusyy,,ryyT!V,,q0266!GDbDM/#::=>AceDDHIKOP rKct||}|dk(rt||||j|fSt||||jS)z rz)r6 gof_pvalsrU)rtestidpvalsrTs r#get_testz GOF.get_testsS tV$ O #U#F+D$)).;sBNN1a$8rKc.tj|dSrrrs r#rzbootstrap..Dsq! 4rK) ValueErrorintr ceilrrr8fit_vecrrr7rrQr)rr?rUnrepvalue batch_sizen_batchcountirepr8paramsrErT stat_sorteds r# bootstraprst, =>? ?bggd5#4456'N +D%))D@VZ,>$?@C]]3Q]/F8&AFggeiiV41=G7+D dem((* *E  +uWz1222eii6t 56s+4f=''%))C0q9wQ' =''$-K EM'') )rKcd}t|D]b}|j|fid|i}|j|}tj|j ||} t | d} || |k\z }d|dz|z S)zMonte Carlo (or parametric bootstrap) p-values for gof currently hardcoded for A^2 only non vectorized, loops over all parametric bootstrap replications and calculates and returns specific p-value, rename function to less generic rr'rr )rr8rr rr7r) rrr?rUrrrr8rrErTs r# bootstrap2rOs$ Ed !eii. .s#''%))C01wQ' $%-  ! 2: rKc$eZdZdZddZdZdZy)NewNormz-just a holder for modified distributions cF|j||j|fSr~)rstd)rrrs r#rzNewNorm.fit_vecpsvvd|QUU4[((rKcRtjj||d|dS)Nrr*)locscale)rnormr7)rrr?s r#r7z NewNorm.cdfss(!!%%aT!WDG%DDrKcb|d}|d}||tjj|zzS)Nrr*r')rrr8)rr?r'rrs r#r8z NewNorm.rvsvs8 G1gU]//333>>>>rKNr)rrrrrr7r8rrKr#rrls)E?rKr__main__)statsrz scipy kstestr)r rtrurvrwrxryz Is it correctly sized?rr*rrrrrrdrrkcDtjjd|S)Nrr)rtr8rUs r#rrs AD 1rKr)rr*)r?rUrr)gGz?gffffff?rq)rrr(r.)rr)rrrNN)rrr);rstatsmodels.compat.pythonrnumpyr scipy.statsrstatsmodels.tools.decoratorsr scipy.specialrrr$rJrZdminus_st70_uppr]rbrirorsrr|rrrrrrrrrrr8rrrrrrrrrrUrrrandomrandnrrRrrrrbtfloorastyper quantindex)rs0r#rs&+%7.Z~Y8|'!'''''        /8RN L '  .~=~=N." ..*d: ? ?& z ''++ac+ "C . &f  sF D $--/CH6 b$--O456 $%'$G D 3ZIiiood#3 IB BK  t}}RA!DQG H II RXXX6rwr{6 7F +x}}X./ *v}**1-. *v}**1-. *v|))!,- 16D D D 795t$d KB$*;!<<=DDSIJ "Z.Q07s I1