!g@ dZddlmZmZddlmZddlZddlm Z ddl m Z dCdZ dZdDd Zd Zd Zd ZdEdZdEdZdFdZdEdZdEdZdEdZdGdZdHdZedk(rededgdZgdZeD]ZeeeeD]ZeeeeeeeeegdZ e jBde jDde jFdejHddd Z%e jLe%ee%e jBde jDd!e jFdejHddd"Z%e jLe%eeee%de%z ejNgd#gd$gd%gZ(e(jSdZ*e(jSdZ+ee*Z,ee+Z-ee(Z.ee*e+e(Z/ee+e*e(Z0ed ejbejde*e+zZ3ed ejbejde+e*zZ4ee*e+e(Z5ed&ee,e-e.e/e0e3e4e5ed'ejNgd(ZeeZ6eed)ed*ed+ed,ed-ed.e*zed/de*d z Z7ed0e7zee7de7z gZ8ed1e8zed2e8e7ejrd zze/fzed3ed4e7d5d6e,dz ejrd7z d5ed8ed9ejNgd:gd;gd<gZ:ee:ed=ed>ejvdee:dee:d gdz ejrd7z zed?ejNgd@gdAgdBgZ>!#44T9: ~q BFF1I%u%^^A& d88D>Aw"aq +AAw%$*a!Aw! $*a!  + OrcXtj|tj|z S)z There is a one-to-one transformation of the entropy value from a log base b to a log base a : H_{b}(X)=log_{b}(a)[H_{a}(X)] Returns ------- log_{b}(a) )r r)rbs r logbasechanger;s 66!9RVVAY rc<ttjd|zS)z$ Converts from nats to bits r;r ers r natstobitsr@s q !A %%rc<tdtj|zS)z$ Converts from bits to nats r=r>rs r bitstonatsrBs BDD !A %%rr=cLtj|}tj|dkrtj|dk\s tdtjtj |tj |z }|dk7rtd||zS|S)aC This is Shannon's entropy Parameters ---------- logbase, int or np.e The base of the log px : 1d or 2d array_like Can be a discrete probability distribution, a 2d joint distribution, or a sequence of probabilities. Returns ----- For log base 2 (bits) given a discrete distribution H(p) = sum(px * log2(1/px) = -sum(pk*log2(px)) = E[log2(1/p(X))] For log base 2 (bits) given a joint distribution H(px,py) = -sum_{k,j}*w_{kj}log2(w_{kj}) Notes ----- shannonentropy(0) is defined as 0 r r&px does not define proper distributionr=)r r r ValueErrorr nan_to_numlog2r;)pxlogbaseentropys rshannonentropyrKs~2 BB 66"'?"&&q/ABBvvbmmBrwwr{N344G!|Qw''11rctj|}tj|dkrtj|dk\s td|dk7r#t d| tj |zStj | S)z Shannon's information Parameters ---------- px : float or array_like `px` is a discrete probability distribution Returns ------- For logbase = 2 np.log2(px) r rrDr=)r r rrEr;rG)rHrIs r shannoninforMsn BB 66"'?"&&q/ABB!|q))BGGBK77}rc Rt|r t|s td|t|s td|tj||}tj|tj tj ||z z}|dk(r|Std||zS)a Return the conditional entropy of X given Y. Parameters ---------- px : array_like py : array_like pxpy : array_like, optional If pxpy is None, the distributions are assumed to be independent and conendtropy(px,py) = shannonentropy(px) logbase : int or np.e Returns ------- sum_{kj}log(q_{j}/w_{kj} where q_{j} = Y[j] and w_kj = X[k,j] 1px or py is not a proper probability distribution&pxpy is not a proper joint distribtionr=)r!rEr outerrrFrGr;)rHpypxpyrIcondents r condentropyrUs(  M"$5LMM  d 3ABB |xx2ffTBMM"''"T'*:;;  M"$5LMM  d 3ABB |xx2 BD' 2b"dG4 56rr ct|s tdt|}|dk(r!t|}|dk7rt d||zS|Sdt |j vs|tjk(r)tjtj| S||z}tj|j}|dk(r dd|z z |zSdd|z z t d|z|zS)as Renyi's generalized entropy Parameters ---------- px : array_like Discrete probability distribution of random variable X. Note that px is assumed to be a proper probability distribution. logbase : int or np.e, optional Default is 2 (bits) alpha : float or inf The order of the entropy. The default is 1, which in the limit is just Shannon's entropy. 2 is Renyi (Collision) entropy. If the string "inf" or numpy.inf is specified the min-entropy is returned. measure : str, optional The type of entropy measure desired. 'R' returns Renyi entropy measure. 'T' returns the Tsallis entropy measure. Returns ------- 1/(1-alpha)*log(sum(px**alpha)) In the limit as alpha -> 1, Shannon's entropy is returned. In the limit as alpha -> inf, min-entropy is returned. z+px is not a proper probability distributionr r=inf) r!rEfloatrKr;strlowerr r_rrr)rHalpharImeasuregenents r renyientropyrfks:  FGG %LE z# a< G,v5 5 #e*""$ $rvvbz""" UB VVBFFH F!|!E'{V##!E'{]1g66??rcy)a Generalized cross-entropy measures. Parameters ---------- px : array_like Discrete probability distribution of random variable X py : array_like Discrete probability distribution of random variable Y pxpy : 2d array_like, optional Joint probability distribution of X and Y. If pxpy is None, X and Y are assumed to be independent. logbase : int or np.e, optional Default is 2 (bits) measure : str, optional The measure is the type of generalized cross-entropy desired. 'T' is the cross-entropy version of the Tsallis measure. 'CR' is Cressie-Read measure. N)rHrRrSrcrIrds rgencrossentropyrisr__main__zQFrom Golan (2008) "Information and Entropy Econometrics -- A Review and Synthesisz Table 3.1)皙?rkrkrkrk)S㥛?g;On?g'1Z?gK?gMbp?) gh㈵>g-C6?gMbP?g{Gz?g?g333333?rkg?g333333?gffffff?g?g??o Information ProbabilityiEntropye)rrUUUUUU?)qq?rtrt)gqq?rtUUUUUU?z Table 3.3zdiscretize functions)2g3333335@g@F@g?@g3@gLD@gYC@g333333&@g/@gfffff?@g9@g3333334@gffffff,@g8@g5@g&@g2@L0@g3333336@g333333@g;@rvǧA@-@g1@g333333<@gffffff0@g0@gG@g#@g2@g @@g:@g0@g333333@gffffff5@g4@gL=@rwg @g6@g)@gfffff:@g9@gfffff6@gffffff&@g333334@g333333:@g"@g%@g333333/@z0Example in section 3.6 of Golan, using table 3.3z'Bounding errors using Fano's inequalityz"H(P_{e}) + P_{e}log(K-1) >= H(X|Y)zor, a weaker inequalityzP_{e} >= [H(X|Y) - 1]/log(K)z P(x) = %sz?X = 3 has the highest probability, so this is the estimate Xhatz1The probability of error Pe is 1 - p(X=3) = %0.4gzH(Pe) = %0.4g and K=3z-H(Pe) + Pe*log(K-1) = %0.4g >= H(X|Y) = %0.4gzor using the weaker inequalityzPe = z0.4gz >= [H(X) - 1]/log(K) = z>Consider now, table 3.5, where there is additional informationz.The conditional probabilities of P(X|Y=y) are )ryg?)rsrsrs)rursrmz2The probability of error given this information iszPe = [H(X|Y) -1]/log(K) = %0.4gz+such that more information lowers the error)gV-?gV-?gw/?)g(\?g+?g%C?)gzG?rlgPn?)N)r#N)r=)Nr=)r r=R)r r=T)=__doc__statsmodels.compat.pythonrrscipyrnumpyr matplotlibrplt scipy.specialrr r!r8r;r@rBrKrMrUrYr[r]rfri__name__printr Yr7psubplotylabelxlabellinspacexplotarraywrrHrRH_XH_YH_XY H_XgivenY H_YgivenXr?rJD_YXD_XYI_XYdiscXpeH_perGw2mean markovchainrhrrrsA D1$3 !H#P & & F,3@<&P(6\.@f, z   + A"A  k!n  k!n .  . ;ACKKCJJ}CJJ} AaA CHHQ ACKKCJJyCJJ} AaA CHHQ^T!AaC[12 *-.?@AA qB qB  C  C ! DBr!$IBr!$I 244 r2!6 6D 244 r2!6 6D bA D + #c4ItT4@ !bhh:;A qME  <= 34 ./ #$ () +  KL RUB = BC 2ad) $D !D () 9 "WRWWQZ-  + ,- *+ E"T2C!GWRWWQZ3G2M NO JK :; :./?@ AB "I >? +>"Q%;PQ_`bcd`eQf8g0hij0jlslnlslstulv/v wx 78"((,-=>NOPKir